Some product problems become easier when you add automation. Fraud prevention becomes more complicated. The system is operating against people who adapt, it rarely has complete information, and every action creates consequences for both the business and legitimate customers.
I designed and implemented this class of production system while building Telzio, alongside AI work in customer service, voice generation, communications, and back-office operations. I am deliberately leaving out proprietary signals, thresholds, and countermeasures here. The useful lesson is not a particular fraud rule. It is how to design the product around uncertain evidence and asymmetric costs.
The wrong starting point is: “Can a model detect fraud?” The better question is: “Which decision are we trying to improve, what evidence is available at that moment, and what should happen when the evidence is incomplete?”
The output is an action, not a score
A risk score is only an intermediate artifact. The customer experiences an action: a transaction proceeds, an account is asked for more information, a capability is temporarily limited, a case is reviewed, or access is denied. Product quality lives in that translation from evidence to action.
That translation is difficult because the costs are not symmetrical. A missed abuse case can create direct loss and wider operational damage. An aggressive block can punish a legitimate customer at the exact moment they need the product. A slow review can be nearly as harmful as a wrong answer. The system therefore cannot be optimized for a single abstract accuracy number.
I start by mapping the decisions and the cost of being wrong in each direction. Which actions are reversible? Which can wait for review? Which need to happen in milliseconds? Which require the system to show its evidence to a person? Those questions determine the architecture more reliably than choosing a model first.
Separate the four layers
I find it useful to separate a risk product into four layers, even when they eventually run in one platform:
- Signals: what the system can observe about the event, account, device, behavior, history, and surrounding context.
- Interpretation: rules and models that turn those observations into evidence about a specific risk.
- Decision: policy that weighs confidence, severity, customer impact, and the cost of delay.
- Action and learning: what the product does, how a person can review it, and how the outcome returns to the system.
This separation prevents several common mistakes. A signal is not proof. A model prediction is not policy. Policy is not the user experience. And an action without a recorded outcome teaches the system nothing.
It also creates clearer ownership. Data and engineering teams can improve signal quality. Model work can improve interpretation. Product, risk, support, and legal stakeholders can agree on decision policy. Design can make the resulting action understandable and recoverable. Everybody is working on the same system without pretending their layer is the whole product.
Rules and models should cooperate
There is a recurring temptation to treat learned models as a replacement for rules. In production, the stronger design is usually a combination.
Deterministic rules are valuable when the condition is known, the consequence is clear, and consistency matters. They are easy to audit and can stop a known pattern immediately. Models are useful when risk is spread across many weak signals, when behavior varies, or when the pattern is too complex to express as a stable set of conditions.
The model can prioritize cases, recognize relationships, or estimate the likelihood of a scenario. The rules can enforce hard boundaries, preserve policy, and provide safe fallback behavior. Human review handles the cases where context or consequence exceeds what either mechanism should decide alone.
This is not less sophisticated than end-to-end automation. It is more mature. The goal is not to maximize the amount of work performed by a model; it is to create the most dependable decision at an acceptable cost.
Design evidence, not a mystery number
A reviewer cannot make a responsible decision from “risk score: 87.” The interface should organize the evidence that moved the case away from normal behavior, show what the system does and does not know, and distinguish a verified fact from an inferred pattern.
That does not mean revealing every countermeasure to every user or exposing sensitive internals. It means designing an appropriate explanation for each audience. An internal specialist may need an event timeline and related signals. A support agent may need a concise status and the next permitted action. A customer may need a clear request for information and an honest description of what happens next.
Explanations also improve the system itself. If a reviewer cannot understand why a case was escalated, the team cannot tell whether the result came from a useful signal, noisy data, a brittle rule, or an unintended interaction among them.
Confidence and severity are different
Teams often compress two questions into one score: “How likely is this?” and “How much would it matter?” They should be separated.
A highly confident prediction about low-impact behavior may justify quiet monitoring. A lower-confidence signal attached to a potentially severe outcome may justify temporary friction or faster review. The decision depends on likelihood, consequence, reversibility, and time—not confidence alone.
Separating those dimensions produces a better action ladder. The system can allow, observe, challenge, limit, queue for review, or block. Each level can have a clear threshold, owner, and recovery path. This is both a risk strategy and an experience design problem.
Reversibility is a product feature
False positives are not an edge case; they are part of the operating reality. A trustworthy product assumes they will happen and designs the recovery before launch.
Can a legitimate customer provide the missing evidence? Can a reviewer see the original state and undo the action? Does the system preserve an audit trail without forcing the customer to repeat the entire story? Is there a time limit on temporary restrictions? Who owns a case that does not fit the normal playbook?
The easier a decision is to reverse safely, the more room the system has to act early. Irreversible actions deserve substantially stronger evidence and authority. That relationship between confidence and reversibility should be explicit in the product requirements.
Evaluate the consequence, not just the prediction
A useful evaluation set includes normal activity, confirmed abuse, ambiguous cases, new customers, unusual but legitimate behavior, and adversarial attempts to manipulate the system. It should also preserve important failures the team has already seen. Otherwise a new version can quietly reintroduce an old problem.
Model precision and recall matter, but they do not describe the whole product. I also want to know:
- How much legitimate activity was delayed or interrupted?
- How quickly were high-consequence cases identified?
- How many reviews produced a changed decision?
- Which signals created work without improving outcomes?
- How long did recovery take after a false positive?
- What did each decision cost in infrastructure and human attention?
These measures reveal whether the system is improving the business decision or simply producing more confident-looking output. They also keep product, model, and operations teams accountable to the same result.
Build the feedback loop before broad automation
A risk model trained on yesterday’s behavior is working in an environment that changes because of the model itself. Abusive users probe controls and adapt. Legitimate behavior shifts with new products, markets, and customer types. Reviewers develop knowledge that is never captured if their work ends with a button click.
Every consequential case should create a useful outcome: confirmed, cleared, inconclusive, or sent for a different kind of investigation. Reviewer changes should be recorded with enough structure to expose patterns. Appeals and customer-support contacts should feed the same picture rather than living in a separate queue.
This is where the back-office platform becomes a product advantage. When the decision, evidence, review, support context, and outcome are connected, learning can travel through the system. When they are split across dashboards and spreadsheets, the people may learn while the product stays ignorant.
The interface affects the quality of human judgment
“Human in the loop” is not a safety strategy by itself. Reviewers can be overloaded, anchored by the model’s confidence, or pushed into approving whatever clears the queue fastest. A poorly designed review tool turns human judgment into theater.
The interface should help the reviewer form an independent view. Show the relevant chronology. Group related evidence without hiding contradictions. Make uncertainty visible. Avoid false precision. Require a reason when the reviewer overrides a high-consequence decision, but do not add ceremony to routine corrections.
Queue design matters too. Priority should reflect consequence and time sensitivity, not simply arrival order or model confidence. Staffing, escalation, and service levels are part of the product system because a correct prediction that waits two days for the wrong person is not a successful outcome.
Privacy and security are architectural constraints
Risk systems naturally attract more data. That does not mean every available signal should be collected or retained. More data increases exposure, operational complexity, and the possibility that irrelevant correlations begin influencing decisions.
I prefer to define the purpose of each signal, who can access it, how long it remains useful, and what happens when it is missing or wrong. Models and tools should operate with the least authority they need. External language and retrieved content should be treated as untrusted input. Sensitive actions should remain behind explicit permissions and durable audit trails.
These controls are not obstacles to AI product velocity. They make iteration safer because the boundaries are known. A team can move faster inside an architecture that already understands access, isolation, validation, and recovery.
The same pattern applies beyond fraud
Customer service, voice generation, and operational AI look different at the interface, but the product pattern is similar. A support system gathers signals, interprets intent and policy, recommends or takes an action, and learns from the resolution. A voice system turns text, pronunciation, context, and brand constraints into an output that may need review or fallback.
In each case, the model is one participant in a decision system. The product has to supply context, define authority, expose uncertainty, measure consequences, and create a feedback loop. That is why I think of this work as product architecture rather than “adding AI.”
What this work taught me about product leadership
Complex products fail at the seams between disciplines. The model team optimizes a benchmark. Operations optimizes queue volume. Support optimizes handle time. Security minimizes exposure. Design removes friction. Every goal can be reasonable in isolation and still produce a bad system.
The product leader’s job is to keep the complete decision visible: what the user is trying to do, what the business must protect, what the system actually knows, what authority it has, and how everyone will learn from the outcome.
That requires enough technical depth to understand the mechanism, enough design judgment to make it legible, and enough operational experience to know that the exception path is part of the product. It also requires restraint. The strongest AI decision is sometimes a rule, a request for more information, or a clear handoff to a person.
Production AI earns trust the same way a good product does: it behaves predictably where it can, admits uncertainty where it cannot, and gives people a credible way forward when something goes wrong.
Continue reading
For the broader system principles, read AI that survives contact with production. For my approach to capable, understandable workflows, read Building tools for experts.