Net Buzz with Richard Wiggins | 12

Inktomi: You Never Heard of This Search Engine, But You'll Use It

How to Build a Search Service Using Cheap Boxes

From the beginning, Inktomi has claimed a unique architecture among search services. Their model calls for a tightly integrated network of commodity computers. Inktomi's architecture uses a large number of nodes connected over a fiber network with shared memory space. The building blocks can include cheap "commodity" boxes, and a given box can remain in service for years, doing its share of work alongside newer, more powerful boxes added as time goes on. Brewer claims significant advantages for this "Network of Workstations" architecture:

• Scalability:
Brewer claims the Inktomi architecture can scale to match the exponential growth of the Web.
• Fault-tolerance:
if a server in the Inktomi farm fails – partially or totally – its partners can pick up the work.
• Performance:
Inktomi strives for response time of ¼ second or less. Brewer claims his architecture optimizes for the bottleneck in large databases – the time it takes to position disk heads under the data you need – by spreading I/O requests among a large collection of servers and RAID disks.
• Cost-effectiveness:
instead of replacing a handful of high-end servers with a new set of expensive high-end servers, Inktomi can keep a commodity computer on the engine's work detail for a useful life of years.

Brewer said that rivals such as AltaVista use server clusters that reflect the Symmetric Multiprocessor (SMP) model. To Brewer, employing SMP is analogous to using the expensive, centralized mainframe services of the past. Brewer claimed that as the Web continues its exponential growth, the cost picture will only grow worse for AltaVista and other competitors whose services are based on more traditional architectures. In essence, he argues that their architecture demands a relatively small group of high-powered servers, and that the whole farm of servers must be periodically upgraded to state-of-the-art.

By contrast, Brewer claims he doesn't need to keep a small stable of state-of-the-art servers at work. He can start with relatively cheap boxes, and add new ones as time goes on, taking full advantage of the 50 to 80% improvement in price/performance each year. He presented an analysis of AltaVista's cost structure, based on data they published with their never-completed Initial Public Offering in 1996, showing that Inktomi's architecture could deliver a per-search cost several times lower than AltaVista's – assuming DEC values its own hardware at list price.

Currently, Brewer said that Inktomi handles some 30 million queries per day. That number is likely to grow dramatically over the next year as new partners come online, but Brewer expressed total confidence that his architecture can accommodate any foreseeable growth. Brewer noted that the number of queries handled per day is already growing faster than the number of pages on the Web.

And now we come to the reason why Inktomi can expect phenomenal growth in search transactions: the deal with Microsoft.

More...