Section 1
The four doors bias walks through
Labels. Your training examples are past human decisions, complete with whatever shaped them. Nothing in the pipeline flags that. Proxies. You removed the protected attribute, so you assume you are safe. Postcode, school name, employment gap, device type, and name spelling all carry information about protected characteristics. Removing the field does not remove the signal. Sampling. The data describes the customers you already serve well. Anyone outside that group is underrepresented, so the system performs worse on them and you have no easy way to notice. Feedback. The model recommends, the recommendation shapes the outcome, the outcome is recorded, and the next version trains on it. A small initial skew compounds each cycle. This is the failure mode that survives a clean audit at launch. It is also the reason a system that looked fair in month one can be unfair by month nine. Related pressure appears as you grow: [Scaling AI Automation as Your Startup Grows](/blog/scaling-ai-automation-as-your-startup-grows).
Section 2
The three automations that carry real risk
Most automations in a small company are low stakes. Routing a message, summarising a call, drafting a first pass. Discrimination risk concentrates in a short list. Anything that filters people: applicant screening, tenant checks, freelancer or partner vetting. Anything that sets a price or a limit for an individual: dynamic pricing, credit terms, deposit requirements, fraud thresholds. Anything that decides who gets service: which enquiries get a human, which complaints get escalated, which accounts get restricted. Spend your review budget there. A summariser that occasionally writes a bland sentence is not what ends up in a regulator's inbox. The workforce version of the same filter is in [AI Automation and Job Displacement: What Founders Should Know](/blog/ai-automation-and-job-displacement-what-founders-should-know).
Section 3
Testing for disparity without a research team
You cannot audit what you refuse to record, and that is the uncomfortable part. Testing whether outcomes differ across groups requires knowing group membership for at least a sample. The workable answer is a limited, consented, access-controlled test set held separately from the production pipeline, used only for measurement. With that in hand, the test itself is simple arithmetic. Slice outcomes by group. Compare approval rates, escalation rates, and error rates. Look at the false negatives specifically, because those are the people who were wrongly excluded and who never appear in your success metrics. Then check the reviewers. If a human is signing off on every output but changes almost nothing, you do not have oversight. You have a rubber stamp with a name attached.
Section 4
Design choices that reduce harm
Prefer rules where you need reasons. If you must be able to explain a decision to the person affected, an explicit rule set will serve you better than a model that produces a plausible narrative after the fact. Set the system to abstain. A confident answer on a thin case is worse than a referral to a person. Build a route for that referral before you build the model. Keep the decision reversible. The lower the cost of correcting an output, the less damage a systematic error does before you catch it. Ranking beats rejecting. Flagging beats blocking. Recommending beats acting.
Section 5
Governance that fits a small company
NIST frames AI risk management around trustworthiness, design, evaluation, and use, and the useful part for a small team is that it treats fairness as an operating requirement rather than a values statement. Ethics that lives in a policy PDF does not change an output. Ethics that lives in the release checklist does. Keep a register of every automation that affects a person, with an owner and a review date. Log the inputs and the decision, because you cannot investigate a complaint you have no record of. Disclose when a decision involved an automated system. Keep the sensitive fields out of prompts that do not need them. And name in advance the decisions the system will never make alone.
Section 6
What to measure in operation
Track outcome rates by group, reviewer override rate, abstention rate, complaint volume and the themes inside it, and time to reverse a bad decision. The override rate is the most informative single number: too high and the automation is not working, near zero and the review is not working. Re-run the disparity test on a schedule rather than at launch only, because the inputs drift even when the model does not. And read the complaints directly rather than through a dashboard. The first signal of a discriminatory pattern usually arrives as an individual complaint that sounds unreasonable until you find four more like it. How you talk about all of this publicly matters as well: [Storytelling in the Age of AI and Automation](/blog/storytelling-in-the-age-of-ai-and-automation) covers the disclosure side.