Accurate and reliable

Aspect 1 of 10. The question behind it is simple to ask and hard to answer: does this system do what it claims, consistently, for the work your asset protection team does?

What you will be able to do

• Tell the difference between accuracy and reliability, and explain why you need both.
• Ask a video analytics or facial recognition vendor for evidence instead of accepting a marketing number.
• Set a performance baseline before a system goes live in your stores.
• Recognize the three failure patterns that show up most often in deployment.

Accuracy and reliability are not the same thing

Accuracy is how often the system is right. A facial recognition tool that correctly matches 94 percent of true person-of-interest cases is making an accuracy claim.

Reliability is whether it stays that good over time, across stores, camera angles, lighting, seasons, and edge cases. A tool that is 94 percent accurate in a well-lit flagship store and 61 percent accurate in a dim stockroom camera feed is accurate but not reliable for your setting.

Where vendor numbers come from

Most published accuracy figures for video analytics, POS exception scoring, or ORC group detection come from a benchmark dataset chosen by the people selling the system. That is fair as far as it goes. It is still a lab result and says little about your stores.

• Ask what data the number was measured on.
• Ask how similar that data is to your store environments and camera hardware.
• Ask what the number looks like for the groups or cases you care about most, such as self-checkout lanes or high-shrink categories, in addition to the average.
• Ask when it was last measured.

Set a baseline before you go live

A baseline is a small, honest measurement of how the system performs on your own cases before anyone depends on it.

• Pull a sample of real incidents your investigators have already confirmed, such as verified ORC cases or sweethearting transactions.
• Run them through the system without changing anything.
• Compare the results to what your investigators found.
• Write the number down, with the date, and who ran it.

You now own a defensible starting point. Every later question about drift, disputed stops, or contract renewal traces back to this one page.

Three failure patterns to watch for

1. Drift. The world changes and the model does not. A shrink analytics model trained before a store remodel keeps predicting the old traffic pattern.
2. Confident wrong answers. Facial recognition and person-of-interest matches can look confident on screen whether or not the underlying match is real. A clean-looking alert is not evidence.
3. Silent degradation. Nobody notices a self-checkout loss detection model slipping because nobody is measuring it after launch. This is the most common of the three.

A scenario

A regional retailer adopts an AI-driven POS exception reporting tool that flags likely sweethearting so investigators can review footage. In the pilot it is 88 percent accurate at flagging confirmed cases.

Six months later, a new loyalty promotion changes how cashiers process discounts. Legitimate discount transactions spike, but the system keeps flagging them as high risk. Investigators pull footage on employees who did nothing wrong, morale drops, and complaints concentrate in the stores running the promotion hardest.

Questions to bring to the table

• What exactly does this system claim to do, in one sentence?
• What evidence do we have that it does that on our stores and our cameras?
• Who measured it, when, and how do we re-measure?
• What accuracy level is good enough for this decision, such as a stop or a case escalation, and who decided that?
• What happens when it is wrong, and who finds out?

Recap

• Accuracy is being right. Reliability is staying right in your stores.
• Treat vendor benchmarks as a starting point for questions.
• A baseline measured on your own confirmed cases is the cheapest governance artifact you will ever produce.
• Silent degradation is the failure you should expect, because nothing announces it.