Big Data/Analytics Zone is brought to you in partnership with:

Fascinated by the "craft" of software development, Eric Pugh has been healthily involved in the open source world as a developer, comitter, and user for the past five years. He is a member of the Apache Software Foundation, and lately has been mulling over how we move from read/write web to the read/write/share web. In biotech, financial services and defence IT, he has helped European and American companies develop coherent strategies for embracing open source software. As a speaker has has advocated the advantages of Agile practices in software development. Eric became involved in Solr when he submitted the patch SOLR-284 for Parsing Rich Document types such as PDF and MS Office formats that became the single most popular patch as measured by votes! The patch was subsequently cleaned up and enhanced by three other individuals, demonstrating the power of the Free/Open Source Model to build great code collaboratively. SOLR-284 was eventually refactored into Solr Cell as part of Solr version 1.4 Eric co-authored "Solr 1.4 Enterprise Search Server", the first book on Solr. he blogs at Eric is a DZone MVB and is not an employee of DZone and has posted 8 posts at DZone. You can read more from them at their website. View Full User Profile

Forget Search, Let's Talk Discovery: Enterprise Search Summit Recap

  • submit to reddit

The middle of May saw me road tripping to not just one, but two Enterprise Search Summits to tell some war stories in doing search at scale. I attended my first ESS back in New York in 2010, and I was curious to see how the conversation had changed in the intervening three years. Would we still be talking about the same things three years later?

So first the bad news…

The Search industry as we’ve known it for the past 15 years is dead. There used to be many large pure search platform vendors like Autonomy, Endeca, FAST, and they’ve all been driven into the arms of larger companies. These deep pocketed companies, and their deep pocketed customers, made Enterprise Search a niche cozy business to be in for years. There was a lot of conversation around “What Next” for the market. And some mourning of the commoditization of search, as evinced by the rise of both open source search engines like Solr and ElasticSearch, as well as the various small companies (BasisTechSmartLogicRaytion) building products that extend search engines, but aren’t search engines of their own.

Martin White

Martin White

Additionally, as Martin White puts it, Enterprise Search is only (only!) a billion dollar business, compared to Business Intelligence at $14 billion, or the ERP market at $45 billion or so. And since Sharepoint 2013, with Microsoft’s acquisition of FAST, has turned itself into the dominant enterprise search engine/CMS solution, this redirects much of that billion dollars of revenue to Microsoft. So this sounds like a stagnating small market right where we’d all better learn Sharepoint right?

Stephen Arnold

Stephen Arnold

Not so fast! Stephen Arnold had something different to say in his very memorable talk on Big Data vs Search. The most interesting slide, which I wish I could find a original of to share, was of the word “Search” with the word “Big Data” splatted all over it. Conveying the idea that there was this massive buzzword coming that was going to steam roll everything in its way, and we all better get ready for its arrival because it’s going to change everything. He did write about some of his observations of how the search market is going to have to adapt to this.

The impact of Big Data on Enterprise Search

]11 The impact of Big Data on Enterprise Search

However, I think he has the impact of the “splat” backwards. I think that Big Data is finally going to let the search community deliver what it has been promising for so many years. I like to talk about the impact of Machine Learning and Natural Language Processing in building smart search experiences. The search engine can move beyond being a statistical token matching tool to an actual discovery interface. One of the attendees commented that “That sounds like an Autonomy pitch from 2004″, and yet that pitch is now becoming a reality, at a cost that any enterprise can afford, not just the deep pocketed ones. Big Data is going to let us build the search engines that we’ve promised our users so many years, and it’s re-invigorating the search market.

So here’s the good news for the search market:

Once you move beyond the definition of our market as the narrowly defined “Enterprise Search” to the more general “Discovery for Enterprises”, our market size explodes. Gartner in their 2012 Magic Quadrant for Business Intelligence and Analytics Platforms report led off with the quote:

The dominant theme of the market in 2012 was that data discovery became a mainstream BI and analytic architecture. The market also saw increased activity in real time, content and predictive analytics.

And search engines are the dominant platform for delievering discovery applications.

As more evidence of the value of search, have you noticed that all the major Big Data vendors have been integrating search into their offerings? Cloudera came out this week with Cloudera Search that puts search on Hadoop.LucidWorksDataStax, and Elasticsearch all have offerings that marry Big Data and Search together.

So let’s all get beyond Search meaning a query box and 10 blue links on a web page, and let’s embrace the new world of Discovery that Big Data enables! And next year, instead of meeting up at the Enterprise Search Summit, let’s attend the Discovery Summit for Enterprises.

Published at DZone with permission of Eric Pugh, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)