Search, discovery and marketing


"However vast any person’s basic reading may be, there still remain an enormous number of fundamental works that he has not read"
Italo Calvino

Erasmus is said to have been the last person in Europe to have read everything. He lived just at the point that printing took off, and so it was in his lifetime that it ceased actually to be possible to have read everything. Some time later there was presumably also a last person to have heard of every book, or every book worth reading - in the 18th century, perhaps. Then, sometime in the past few decades, it also became impossible to have heard every good song. It is still possible, just, to see every good film, but the so-called 'golden age of television' means it's harder and harder to watch all the great shows. 

This is not a problem of search or availability, but a problem of too much availability, and of course the internet magnifies this a thousand-fold. Paradoxically, the internet makes it possible to get anything you've ever heard of but also makes it definitively impossible to have heard of everything. It allows anyone to be heard, but how do people hear of you?

The first attempt online was Yahoo's directory - an editorial home-page and a directory of every single website, organized by category. This sounds like a joke now - "there was a website that listed every single website that there was". Yahoo actually worked pretty well when there were 20,000 websites, just as a book shop with 20,000 titles works pretty well. But the Yahoo directory peaked at 3.2m sites and at that that point it definitely didn't work - you can't possibly scroll past that many entries (though it lingered on in a half-life until Marissa Mayer shut it down this year). And in the meantime, Google invented PageRank, coming at the problem from an entirely different direction. Suddenly, search worked, and that seemed like the answer. 

Today, app stores look a lot like the Yahoo of 20 years ago, and they don't work for the same reasons - you can browse 20,000 apps but not a million. Hierarchical directories don't scale. And so while it's easy to make a list of things that Apple and Google should fix on their app stores, that misses the point - it's like making a list of ways that the Yahoo home page should have been better. You might have been right but the answer was still Google. (I suspect that the same applies, just a little, to the current moves towards app search and deep linking, incidentally. PageRank uses the signal of links between pages - the ability to link of itself is only half the picture.) This is one reason mobile messaging apps are so hot - because they might become acquisition and discovery channels. 

However, I think our preoccupation with the problems of apps and app stores and with the ways that they broke Google masks a deeper issue - that Google didn't really solve the problem either. Or rather, it moved the problem. Google is very good at giving you what you're looking for, but no good at all at telling you what you want to find, let alone things you didn't know you wanted. Like Amazon, it's essentially a passive product (which is why Now is so interesting). It relies on waiting for you to find out what you want somewhere else, in some other way, and then it gives it to you. No-one complains that ‘I put my book on Amazon and no-one can discover it there’, but that’s really no different to saying ‘I put my app in the app store and no-one can discover it there’, or indeed 'I made a web page and no-one came'. 

So Google moved the problem from 'I want to find this and can't' to 'yes, but what do I want to find?' That is, 'What should read next?', 'what lamp would I like?' or 'where should we go on holiday next weekend?' are not valid search queries. And that's just for problems you know you have - the most interesting businesses are often things you'd never given any thought to. What search would you do that would tell you about Lyft, Instacart, Pinterest, AirBnB or Evernote if you had no idea that they existed?  (This prompts the question, indeed, of what didn't or couldn't work before 2007 because search was the dominant model.)

One of the clearest places to see this problem of ‘too much’ is Yelp. I’ve been fascinated by how many companies are effectively trying to unbundle Yelp, despite that fact that (unlike Craigslist) it’s a modern technology company that does most of the things one would expect it to. But where people unbundling Craigslist generally try to peel off a category and deliver a modern experience, the people going after restaurant listings are often doing so with constraint. That is, instead of giving you every single restaurant that’s within 2 miles and that lots of people liked, they give you 10 restaurants. YPlan gives you one, and just one, thing to do tonight. People are attacking crowdsourced universal scale with constraint, curation and personal preference. 

Looking at these companies, it strikes me that actually, saying that ‘Yahoo’s directory didn’t scale’ misses the point. What we’re really seeing is a trade-off between two problems. You can have a list, solving discovery and recommendation, but once the domain gets big then your list is either unusably long or partial and incomplete (and perhaps uneconomic to maintain). Or you can have a searchable index of everything but you’re on your own working what’s good and finding things you didn't know to search for. Time Out is an interesting attempt to sit in the middle of that scale - enough coverage to be quasi-universal, and to promise something good nearby wherever you are, but also enough curation that you don’t just get 5,000 listings all with five stars.  ProductHunt is an attempt to use community to surface quality at scale, as is Pinterest (both are a16z investments). In contrast, Canopy uses hand-curated selections on Amazon. The question for all of these: do you filter crowdsourcing down enough to get quality, or scale up editorial to get coverage, or you give up on coverage and do a purely curated product? 

One answer is that the machine will scale to solve the problem - you aggregate the opinions of the many (Yelp), or data about your purchases (Amazon) or perhaps you yourself (Google Now). Amazon’s failure to do this reliably makes me hesitate (one suggestion it gave me is above - I collect these), but more deeply, there are some questions, again, that just do not make good text search queries. I can ask Amazon for Owen Hatherley’s new book on Communist architecture and it’ll find it, and Google will give me reviews. But if I ask them “what should I read next?” then you quickly fall into the uncanny valley between data mining and the 'real', HAL 9000 AI that we don't actually have yet. That is, a machine can learn that I like architecture and history books, but that’s not the same as knowing that I will buy Owen Hatherley’s book but never Jacqueline Yallop’s book on Victorian utopian model villages, and we're not quite there yet. 

You can also see this challenge right now in both books and fashion. Amazon, after 20 years of ruthless execution, still only has under a third of the entire print books market. Most people buy most of their books in physical retail, because book shops are not just relatively inefficient end-points to a physical logistics network, but also filters and recommendation platforms. They’re high-latency but also high-bandwidth. Fashion, meanwhile, is going online very fast, but not through Amazon. Rather, dozens of companies are circling around the right models or recommendation, curation and discovery.

So, perhaps, a split might be:

  1. There is giving you what you already know you want (Amazon, Google),
  2. There is working out what you want (Amazon and Google's aspiration),
  3. And then there is suggesting what you might want (Heywood Hill). 

Perhaps this is just the next step in retail. Amazon let people in one-bar towns buy products that could only previously be had in big cities, but it doesn't let you shop the way people can shop in big cities - once you understand that physical logistics is a very small part of what shopping means (this can be hard to spot if you've only ever lived in the suburbs of the South Bay). Buying and shopping are not the same thing. That's what the new generation of internet retailers are trying to do - to scale curation instead of catalogues. 

This is also, very obviously, what Apple News and Apple Music are trying to do. Each of them approaches a data set in which the default answer is a million options and a search box. Each asks the question: "how do we take a commodity in a database (web pages, music tracks) and layer curation and recommendation in ways that are more usable and friendly than just giving people a search box and pushing them out of the door?" It's not as though this is a solved problem anywhere else. RSS (which actually powers Apple news) failed as a consumer technology  and following a magazine or musician on Facebook means you won't see more than one in ten of their posts (and I don't choose my friends for their taste in music, or book reviews). And you can't Google for 'what do I read next? I hesitate to declare that this can't work. 

Another strand here, which really does take us back to the beginning, is that I always thought the most useful part of Flipboard (now the last man standing of all the iPad news aggregators that followed the iPad launch, and superficially similar to Apple News) is not any of the formatting or design but the built-in directory of sites. If you want to see 15 sites about basketball, or typography, or hats or makeup, where would you find such a list? Yahoo did that once, and so did link rolls and web rings or, in another way, (It's funny how many people keep trying to rebuild Yahoo - it didn't work out that well for Yahoo itself). Tumblr today provides one route into this, if you invest the time in surfing the topics, as does Pinterest. But you don't get that from Facebook or Google. 

Of course, once you give up on a universal search and a universal store - that is, Google and Amazon - as the only answer, then you've moved the problem again. There's an old Soviet joke that a man walks into a shop and asks “You don’t have any fish, do you?" And the shopkeeper says “No, we’re a butcher - we don’t have any meat. The shop next door is a fishmonger - they don’t have any fish”. So: where do you want to be hard to find? Do you want to be one of a million listings in Google or the app stores, or do you want to be one of ten or 100 listings in a carefully curated selection - but where that selection is one of a million listings in Google or the app store? 

All of this takes us to marketing. I sometimes tease my Xoogler colleagues by suggesting that if PageRank Really Worked, SEM wouldn't exist - if the links were always the right answer then no-one would click on search advertising. (Larry Page is fond of asking challenging questions, but that might be one step too far.)  Until then, though some companies can make it entirely through organic search or Facebook virality, most cannot. (Indeed, very often the mere fact that you've made these channels work for acquisition means they stop working, since your link advantage gets arbitraged away by imitators or Facebook decides you're taking just a little too much of the newsfeed.) For the rest of us, that means marketing. In effect, by removing all other constraints, the internet makes advertising more important than ever.