How To Evaluate AI Software

Buying off-the-shelf AI (Artificial Intelligence) software is a good first step for those companies that are new to the technology. There should be little need to make investments in technical infrastructure or to hire expensive data sciences. There will also be the benefit of getting a solution that has been tested by other customers. For the most part, there should be confidence in the accuracy levels as the algorithms will probably be implemented properly.

But there is a nagging issue: there are many AI applications on the market and it is extremely difficult to determine which is the best option. After all, it seems that most tech vendors are extolling their AI capabilities as a way to stand out from the crowd.

Then what are some factors to consider when evaluating a new solution? Let’s take a look at the following:

Data Connectors: AI is useless without data. It is the fuel for the insights. 

But when it comes to a new AI solution, it can be tough to find the right sources of data, wrangle it and integrate it. Thus, when evaluating an application, you need to make sure that there are ways to handle this process. 

“The most complex task in an AI solution is not to implement the machine learning algorithm anymore—this is usually available as a set of functions in every tool—but to collect the data,” said Rosaria Silipo, a Ph.D. and a principal data scientist at KNIME. “That is, to connect to a variety of data sources, on premise, on the web, or on the cloud, and extract the data of interest.”

Flexibility: AI does not have general scope. Instead, it is focused on particular use cases. This is known a “weak AI.”

This is why it is important to see if the application is built to handle your particular vertical or situation. 

“Take search, for instance, where AI can be used to re-rank results and improve relevance,” said Ciro Greco, who is the Vice President of Artificial Intelligence at  Coveo. “When applied to ecommerce, search is searching on semi-structured records, such as products with little text available and we can count on reasonable amounts of behavioral data produced by users who browse the website. A strategy based on user behavior can be very effective, because we can count on having enough data to learn from.”

Yet AI-search for customer service use cases is often much different. It’s often about finding technical documents. “There is plenty of unstructured text, such as Knowledge Articles, and fewer behavioral data points, because customer service websites usually are less visited than e-commerce platforms,” said Greco. “So in that case, a strategy based on NLP for topic modeling will probably be more effective, because we need to maximize the gain from the information we have, in this case free text.”

Ease-of-use: This is absolutely critical. The user of AI is often a non-technical person. If the application is complex, there could easily be little adoption.

Ethical AI: Even if the application is accurate, there could be risks. The data may have inherent biases, which could skew the results. This is why you should get an explanation of the data and how it is used. 

“What many forget when evaluating an AI solution is the potential damage or risk it could pose to your organization,” said Michael Mazur, who is the founder and CEO of AI Clearing. “What if your organization is sued for deploying this solution?”

Costs: “If you’re the first customer in a specific industry for an AI vendor, then you’re a very valuable customer to have and you can use that as negotiating leverage for a beneficial contract,” said Brian Jackson, who is an analyst and research director at Info-Tech Research Group.

Tom (@ttaulli) is an advisor/board member to startups and the author of Artificial Intelligence Basics: A Non-Technical Introduction, The Robotic Process Automation Handbook: A Guide to Implementing RPA Systems and Implementing AI Systems: Transform Your Business in 6 Steps. He also has developed various online courses, such as for the COBOL and Python programming languages.