Ongoing monitoring

Aspect 9 of 10. A video analytics or shrink detection system that was appropriate on launch day can drift into something you would never have approved, and nothing announces it.

What you will be able to do

• Explain why AP AI performance changes after deployment.
• Choose a small set of signals worth watching, from override rates to match accuracy.
• Set thresholds and name who acts when one is crossed.
• Build a review rhythm that survives busy quarters and staff turnover.

Why things change

Traditional CCTV behaves the same way until someone edits the settings. AI-driven AP systems sit in a moving environment, so the same system produces different results over time.

• The world moves. ORC tactics, self-checkout theft methods, and store layouts change, and the system was trained on how things used to be.
• The users move. Bad actors learn what defeats a system, and shoplifting patterns shift away from what the model was tuned for.
• The vendor moves. Facial recognition or analytics models get updated, retired, or quietly retuned.
• The data moves. Your store footprint, product mix, and case management categories evolve.

Watch a few things well

A dashboard nobody reads is not monitoring. Pick a short list you will look at.

• Quality. A sample of alerts and matches reviewed by an investigator against a written standard.
• Outcomes by group. Stop, detainment, and escalation rates broken out by the demographics of your customer and employee base.
• Override rate. How often investigators dismiss an AI alert as a false positive, and in which direction.
• Volume and mix. Where the system is generating alerts, including store types or regions you did not plan for.
• Complaints and appeals. The clearest signal that something is wrong, including customer complaints about wrongful stops.

Thresholds turn signals into action

A number without a threshold is trivia. For each signal, write down what normal looks like, what triggers a review, and what triggers a stop of the technology itself.

• Quality sample of alerts falls below the agreed standard for two consecutive reviews.
• Override or false-positive rate moves more than a set amount from baseline.
• Stop or detainment rate gap between demographic groups widens beyond the level you agreed to tolerate.
• Any complaint alleging a wrongful stop, discrimination, or harm.

A scenario

A retailer deploys an AI system that flags self-checkout transactions for likely loss. For six months it performs well. The retailer then launches a new bagging and weight-verification process, changing normal scanning behavior. The system keeps applying the old pattern confidently, and store investigators approve the flags because they always have.

Automation complacency

The most common monitoring failure is human. When a system is usually right, investigators stop reviewing.

• Track how long investigators spend before approving a stop as well as whether they approved it.
• Rotate a blind sample of matches or alerts past a second reviewer.
• Give investigators a way to flag doubt about a facial recognition match that does not slow down operations.
• Treat a falling override rate as a question to investigate.

Build the rhythm

1. Continuous. Automated alerts on alert volume, system errors, and hard thresholds.
2. Monthly. Sample review of alerts and matches, and a look at customer complaints.
3. Quarterly. Stop and detainment outcomes by group, override trends, and drift against the original test results.
4. Annually or on change. Full re-assessment when the model, vendor, store process, or ORC pattern changes.

Write the schedule down and give each item an owner and a date. A rhythm with no calendar entry does not exist.

What to do in your own deployment

1. For each AP AI system, list three to five signals you will monitor.
2. Record a baseline now, while things are working.
3. Set review and stop thresholds in writing, including who can pause facial recognition matching.
4. Name the person who receives alerts and the person who can pause the system.
5. Put the monthly and quarterly reviews on the calendar.
6. Keep the review records. They are your evidence that oversight was real if a stop is ever challenged.

Recap

• AP AI systems drift because ORC tactics, store operations, the vendor, and the data all keep moving.
• Monitor a short list of signals you will genuinely review, from override rates to demographic outcomes.
• Pair every signal with a threshold, an owner, and an authorized action.
• Watch your investigators as closely as the model, because complacency is the quiet failure.