Safe and ethical
Aspect 4 of 10. Every AI system in your AP stack can fail. Safety work is deciding, ahead of time, what happens when it does, and who absorbs the damage, including the shopper on the other end of a detention.
What you will be able to do
- Separate safety (physical and financial harm) from ethics (harm we choose to allow or prevent).
- Run a simple pre-launch harm walkthrough for an AP tool you did not build.
- Define limits the system is not allowed to cross, and enforce them outside the algorithm.
- Build a kill switch for facial recognition or automated detention alerts that you have tested.
Safety is measured by consequences
A tool built to speed up self-checkout loss detection can trigger a physical confrontation between an associate and a customer over a false alert. A facial recognition system built to flag known offenders can send a plainclothes investigator to approach the wrong person in a parking lot.
Nobody intended either outcome. A deployer has to answer whether harm was foreseeable and what was put in place to catch it.
The deployer test
If this system gives its worst plausible answer on its worst day, for example a wrong-person detention, who gets hurt and how badly?
A harm walkthrough you can run in an hour
- Write one sentence describing what the AP system does and who acts on its output.
- List the people affected, including customers who never see the alert, only its consequences.
- For each group, name the worst realistic outcome of a wrong result, such as a wrongful detention or a false civil recovery demand.
- Rate each one by how bad it is and how easily it is reversed.
- For anything severe or hard to reverse, add a control before launch, such as requiring a second reviewer before any physical stop.
Reversibility matters more than probability. A rare misidentification you can catch before contact is a different problem from a common alert that leads an associate to confront the wrong customer.
Set limits outside the model
Instructing a system to behave is not a control. Controls live in the workflow around it.
- Scope limits. Named uses the system is approved for, such as internal case management, and uses it is not, such as automatic detention without human review.
- Thresholds. Match confidence scores, dollar amounts, or exception-report rankings above which a person must sign off before any customer contact.
- Refusal paths. What the video analytics or facial recognition system does when confidence is too low to act.
- Rate and blast radius. How many stores or how many customers a single bad model update or a corrupted watchlist can reach before someone notices.
Common gap
Pilot stores have tight review requirements. Those limits often do not survive the rollout to hundreds of locations, because nobody rewrote the field procedures.
Ethics is the part nobody assigns
Some uses are legal, technically sound, and still not something your organization should do. Running facial recognition against every customer who enters instead of known persons of interest only. Scoring associates' honesty from break-room camera behavior. Using license plate recognition to build profiles unrelated to any case.
These decisions rarely arrive as ethics questions. They arrive as shrink-reduction projects. The practical fix is to require one written paragraph, before purchase, answering: if this use were described accurately in a local news story, would we defend it?
A scenario
A big-box retailer deploys an AI video analytics tool that flags customers for potential self-checkout theft based on scanning behavior. It works. Detected shrink incidents rise sharply.
Six months in, AP investigators report that store associates are approaching and detaining flagged customers on the spot, without pulling the case file or checking the confidence score, because the alert pops up as urgent on a handheld device. Nobody told them to skip the review. The workflow simply made the fast path the only path.
What went wrong
The system was safe in the lab. The incentive it created on the floor was not. Safety review stopped at the algorithm and never looked at the behavior the alert design rewarded.
Have a way to stop
- Name the person who can suspend a facial recognition or automated detention alert without asking permission.
- Know how long a shutdown takes, and what the manual case review fallback is.
- Test the fallback once, on a normal day, before you need it during an incident.
- Define the trigger in advance: what pattern of wrongful stops or complaints means stop immediately.
- Keep enough logging, video, and case notes to reconstruct what the system did during the failure window.
Until someone has pulled the kill switch, it is only an assumption.
Recap
- Foreseeable harm to a customer or associate is your responsibility whether or not you built the system.
- Rate harms by severity and reversibility, then control the severe and irreversible ones first, starting with anything that leads to physical contact.
- Real limits live in the workflow: scope, thresholds, refusal paths, blast radius.
- Ethical problems usually arrive disguised as shrink-reduction projects.
- Test the stop procedure before the day you need it.
