Member Article

Search, the Future, and the Big Data World

The final article in this series on enterprise search will touch on current and future issues.

Over the past few years, the industry has witnessed a series of takeovers. Notably, HP bought Mike Lynch’s Autonomy, Oracle acquired Endeca, and Vivisimo became a part of Big Blue. Big Data strategy has been a key factor behind these acquisitions. Other established enterprise search players such as Exalead and Attivio have also been making the news in marketing on the Big Data theme. Why is this?

‘Big Data’ is about the holistic analysis of data sets in the hope of finding insights to drive business decisions. This is not new. Large retailers, for example, have been studying the effects of subtle price changes on sales volumes for years, through the analysis of very large structured data sets.

Unstructured Data Analysis

What is new is the notion of gaining insight from the 80 per cent of data that is not structured. For anumber of reasons, this is not as simple as it sounds. Structured data is largely computer-generated and therefore reliable, precise and easily amenable to accurate analysis. Unstructured content on the other hand is largely produced by humans and therefore extremely variable in quality and format. Before analysis of unstructured content can bemade, it is necessary to pre-process the data to add structure and normalise it.

The technologies and skills needed to do this are well-known to the enterprise search industry, which has been doing this for years - creating metadata (structure) to drive important search functions such as search navigation and results ordering. As a result, the current popularity of BigData is causing acquisition activity.

Leading search products also have index capabilities that are able to deal with hybrid environments in which some of the data is originally from the structured world, and some unstructured. To paraphrase a leading vendor, “the structured element might tell you what is happening, the unstructured element can tell you why.”

Enterprise Search – Where Next?

Despite these acquisitions, enterprise search continues to be of growing importance in its own right. The process of finding information becomes more difficult as data sizes scale. At the same time, in the information economy, finding information - whether it is to check a fact, retrieve a known document, or conduct new research into a subject - remains a critical part of the process of doing business.

The enterprise search market is maturing. In terms of market share, as with the database market, a few very large players dominate. Microsoft SharePoint and FAST, the Google Search Appliance and Apache Solr, the leading open source alternative, command the majority of the market.

As with databases:

• Technology decisions are starting to be based on policy and price rather than technical differentiation
• Implementation services and applications are becoming more important than the underlying search engine technology

In conclusion, a well-implemented enterprise search system remains a key component for driving business productivity.

This was posted in Bdaily's Members' News section by Iain Fletcher .