AI needs automated testing, monitoring

Find the Latest SoC Solutions
for... Automotive... IoT... Security... Audio... Video

sdtimes.com, Nov. 04, 2022 – 

In the 1990's, when software started to become ubiquitous in the business world, quality was still a big issue. It was common for new software and upgrades to be buggy and unreliable, and rollouts were difficult.

Software testing was mostly a manual process, and the people developing the software typically also tested it. Seeing a need in the market, consultancies started offering outsourced software testing. While it was still primarily manual, it was more thorough. Eventually, automated testing companies emerged, performing high-volume, accurate feature and load testing. Soon after, automated software monitoring tools emerged, to help ensure software quality in production. Eventually, automated testing and monitoring became the standard, and software quality soared, which of course helped accelerate software adoption.

AI model development is at a similar inflection point. AI and machine learning technologies are being adopted at a rapid pace, but quality varies. Often, the data scientists developing the models are also the ones manually testing them, and that can lead to blind spots. Testing is manual and slow. Monitoring is nascent and ad hoc. And AI model quality is suffering, becoming a gating factor for the successful adoption of AI. In fact, Gartner estimates that 85 percent of AI projects fail.

The stakes are getting higher. While AI was first primarily used for low-stakes decisions such as movie recommendations and delivery ETAs, more and more often, AI is now the basis for models that can have a big impact on people's lives and on businesses. Consider credit scoring models that can impact a person's ability to get a mortgage, and the Zillow home-buying model debacle that led to the closure of the company's multi-billion dollar line of business buying and

flipping homes. Many organizations learned too late that COVID-19 broke their models – changing market conditions left models with outdated variables that no longer made sense (for instance, basing credit decisions for a travel-related credit card on volume of travel, at a time when all non-essential travel had halted).

Not to mention, regulators are watching. Enterprises must do a better job with AI model testing if they want to gain stakeholder buy-in and achieve a return on their AI investments. And history tells us that automated testing and monitoring is how we do it.

click here to read more...

 Back