Imagine walking into a dark, abandoned library, armed only with a flashlight. It’s your job to find an important piece of information somewhere in this building. With the narrow beam of the flashlight, you can read the titles on the spines of the books, but little else.
Now imagine you find a light switch and the card catalog. Suddenly you have an index of every book, searchable by subject, author or title. Now you can find the most obscure piece of information, no matter where it is in the building.
In many ways, standard search methodologies such as Google and other text-based search engines are like that flashlight — a narrow band of results from a single perspective. In fact, the basic function of a search engine has remained the same since the early 1980’s; a user enters a search term and gets back links to pages that include the word or words found in that search string.
The limitation of traditional keyword searching is especially painful for organizations that rely on Electronic Content Management (ECM) systems to manage their internal document storage. A dirty secret of document management is that an estimated 80–85% of aggregated information in ECMs is hidden, unsearchable Dark Data.
Digging in the Dark
So how do you ditch the narrow search results of traditional keywords and move to a world where you have all of the information you need at your disposal? As I mentioned in a recent article, we at the Gordon Flesch Company have come to believe that the answer to this shortfall is cognitive computing.
Specifically, we have partnered with IBM Watson, the most mature and advanced cognitive system. Watson is a system that is not simply programmed; it is trained to learn based on interactions and outcomes. It’s a cognitive system that rivals a human’s ability to answer questions posed in natural language with speed, accuracy and confidence. It’s the most successful example of a system to navigate the complexities of human language and to analyze massive amounts of data exceptionally quickly (over 200 million pages in three seconds when it beat Ken Jennings on Jeopardy).
To understand how a cognitive system can get past the limitations of traditional search methods, let’s consider a couple of hypotheticals. First, assume you’re investigating an insurance issue and you need documents pertaining to insurance claims for young people. If you search for terms like “young people,” most Electronic Content Management (ECM) systems will return documents with that term. Of course, those writing about young people rarely use the words, “young people.” However, a cognitive system like Watson can find documents with related words like mother, father, dad and words for activities correlated with children like baseball and football.
Here’s another example. We have been working with a large government agency that wants to improve results from public queries of documents on its website. The agency has thousands of documents related to vegetable-related research alone, and finding the right one is an arcane and opaque process. The application has been learning about the corpus, so that if someone searches for a type of study about vegetables, the search will recognize documents on research that may be relevant, but perhaps related to a different vegetable. It will recognize whether research related to Brussel sprouts is applicable to your research on carrots.
In another example, we saw department employees use the term “FV” when they really meant “fruits and vegetables.” However, in financial studies, FV is shorthand for “Future Value,” so Watson has learned to identify the context of a search. The system is trained to learn over time exactly how researchers think, talk and use information. It’s also the solution to the “Paris Hilton problem” I mentioned in my last post. That is, the ability, for example, to determine whether a query is being made by someone who is trying to reserve a hotel in France or is simply passing time surfing the internet in search of has-been celebutants.
Everyone is an Expert
To overcome the limitations of ECM search technology, large corporations often employ data scientists, data miners and Business Intelligence architects to design more intelligent ways to mine data for information. Of course, few organizations have the luxury of hiring these types of experts.
However, unlike most data analytics projects, our Watson-based solution does not demand significant intervention by business intelligence experts with a PhD or in-house Subject Matter Experts (SMEs) to continually maintain and update a knowledge base. We use automated ingestion and training routines to handle new documents and to assure that the first potential results from a new publication are available immediately.
Within organizations, workers need tools that can help them generate insights, make better decisions and develop expertise faster. Cognitive computing meets this need by sorting through vast quantities of structured and unstructured data to provide specific, personalized recommendations that are backed by solid evidence. And the system continues to learn and get better over time.
Keeping up with the latest technology is a challenge. Knowing what systems and programs are best for your unique business situation can help you provide advanced services and products and, ultimately, outpace competitors. At the Gordon Flesch Company, we’re on the forefront of cutting-edge technology and can provide solutions for all your computing needs. Contact us today for a free consultation. Also, look here for exciting information from the Gordon Flesch Company about how our partnership with IBM will help your team find exactly the information it needs in any database.