Tuesday, October 9, 2007

Search vs. Discovery

Recently, Silicon Valley startup Powerset has made waves by promising to use natural language and sentence parsing to better search the web. They claim that by better understanding the intention of a user's question and the meaning of a document's content, the search results can be better matched to a user's need. So, they are betting that searchers will be willing to put (arguably) more effort into composing their search, to retrieve documents keyed exactly to their well formed English question.

First, this is not a new concept. Companies like Inquira and Kaidara have been applying language processing to the customer service problem for a number of years. They’ve developed technology allowing users (like you and me) who go on line needing help technical support to write in natural English to get to the answer from a natural language search of the repository of solutions they have.

While this is working somewhat well for customer service there are some hurdles to adoption for the consumer search market.

First, it is unclear whether we (the consumers) are willing to put more effort into searching to receive incrementally better results.

Second, how often are web users *really* trying to answer a question? We’re often not looking for a single answer, but we are instead trying to get a general sense of some topic or idea.

And third, if we are seeking something specific, the "phrase your query in the form of a question" restriction seems annoying. I just want a really simple way to indicate what I am looking for.

This is where web content discovery tools (vs. search tools) like ours and others really deliver. In many cases, users of advanced systems don't even know what they want. People have a (seemingly reasonable) expectation for technology to develop to get me more of what I want with less effort. Even better, we want to get what we didn't yet know we wanted with *no* effort.

This is what a good movie, or a book recommendation gives you - the huge benefit of tools like Amazon's recommendations and Netflix personalized ratings. In both cases users are delighted by a system which rewards very little effort with a personalized and serendipitous experience.

How often do you choose a Netflix movie based on search vs. how often do you go with their recommendations? I know their recommendations make me think and remind me of movies I’d forgotten I wanted to see – much like we prompt users to ask questions around their investments which they had not thought of, and generate new ideas for investments.

Companies like Powerset fill a need, and I hope they'll succeed at their stated goal, but they'll be solving a minor problem, leaving the grander, game changing discoveries still to come.


Anonymous said...

Powerset doesn't require that queries be phrased as questions, Jeopardy-style. It will also deal with single words or phrases, such as "politicans who died in office":


Mark Johnson said...

The power behind Powerset isn't just in our ability to analyze natural language queries that users type in; but rather our index analyzes every single sentence that we read and extracts some meaning out of that sentence. Having an index like that makes queries possible that were previously impossible or extremely difficult and leaves open the possibility to many new kinds of highlighting, results presentation, etc. It's ALL about the index.

There was an error in this gadget