The History and Evolution of Search

We’ve been thinking about different ways in which technology has made our lives easier, and luckily we were able to come up with a lot of examples. The next question we considered was “What are the next big infrastructure steps we need to take in order for technology to make or lives even simpler”?

It is clear that one of the key challenges we need to overcome is the personalization challenge. There are many recommendation engines out there today, but unfortunately they do a very poor job of knowing a person’s preferences. They are targeted to be recommendation engines and no more. It is time we started looking at these engines in a new light and say that a 3% – 20% hit rate is not good enough. This actually means that recommendation engines have a miss rate of 80% – 97%.  Ouch! With all of the big data technology that is out there today and all of the information that exists about people and their preferences,  it is time we improve these hit rates and started striving towards what I would, for lack of a better term, call Action Engines. Engines that actually know who we are, what our preferences are, and don’t give us random recommendations that have very little applicability to who we actually are. We’ll post more about our thoughts regarding action engines in a later post, but before we do we would like to examine how we got here. We’re not providing a comprehensive history of search online but only touching on the major developments in the past 20 years. For a more complete history of search on the internet (and before) you can follow this link: .


Indexing Engines – the Early Web Years

The first few hundred web sites began in 1993 and most of them were at colleges, but long before most of them existed came Archie, the first search engine. Archie was in 1990 by Alan Emtage, a student at McGill University in Montreal.

It was 20 years ago that the web’s first bot came. In June 1993 Matthew Gray introduced the World Wide Web Wanderer. He initially wanted to measure the growth of the web and created this bot to count active web servers. In October of 1993 Martijn Koster created Archie-Like Indexing of the Web, or ALIWEB in response to the Wanderer. ALIWEB crawled meta-information and allowed users to submit their pages they wanted indexed with their own page description. By December of 1993, three full fledged bot fed search engines had surfaced on the web: JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider.

The next evolutionary step came with Excite which came out of project Architext, which was started in February, 1993 by Stanford undergrad students and introduced statistical analysis of word relationships to make searching more efficient. In April 1994 David Filo and Jerry Yang created the Yahoo! Directory as a collection of their favorite web pages. These were followed by a number of sites including Lycos, Webcrawler, Infoseek, AltaVista and Hotbot, the meta-search engine (my favorite at the time).

Moving to “Intelligent” Search

It took about 5 years and in April 1997 Ask Jeeves was launched as a natural language search engine. Ask Jeeves was the first engine to include additional smarts on top of statistical word indexing. At about the same time a number of vertical search engines popped up on the scene. In 1998 Overture, the pioneer in paid-search, was launched by Bill Gross under the name GoTo.

And then in 1998 Google came on the scene. Google launched Google AdWords in 2000 and on November 15, 2003 Google began to heavily introduce many more semantic elements into its search product. And 10 years after it started searched changed forever (or at least for the foreseeable future).

Discovery & Recommendation Engines

The next step in the evolution started in 2005 when Yahoo! purchased Flickr. &, and introduced social relevance to the broader market. Yahoo! also made a strong push to promote Yahoo! Answers, a popular free community driven question answering service. In 2006 and 2007 numerous social bookmarking and decentralized news sites like became popular.

In 2008, 15 years after our story begins recommendation engines started becoming prevalent when eBay, Amazon and others started driving recommendations following purchases.

Pure recommendation apps have become very popular in the last few years with the evolution of the iPhone, iPad, Android and apps on the Apple Store and Google Play.

Action engines

It is now 5 years after recommendation engines came on the scene as a significant driver in search. Based on the evolution over the past 20 years, we are due for the next big thing. The Action Engine revolution!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s