AI Trust Patterns

Trust and Transparency

Separated Confidence Scores

When the agent checks several things, show each result on its own. Never blend them into one number.

Phase
Design · Build
Applies to
End users, developers
Oversight
Varies with stakes
Key factor
Decision consequence
OverviewWhen to useQuestions to askExampleAnti-patternAnti-patterns to pay attention toReferencesEvidenceRelated

Overview

When an agent checks several independent things (completeness, compliance, duplication, identity), a single combined score lets strong results on two checks hide a failed third. The user sees 0.82, approves, and the problem that was averaged away goes through. Separated Confidence Scores treats each check as its own result, shown individually, with uncertain or failed checks first.

Attention

This applies to match confidence too. When the agent is uncertain between two candidates, showing only the top result is a silent guess. Both candidates should be visible, with the reason for the uncertainty stated.

When to use

Use when

  • The agent checks multiple independent dimensions in one flow.
  • Low confidence on one dimension has meaningful consequences, even if the others are high.
  • The agent resolves a match from multiple candidates and confidence is below threshold.

Not needed when

  • The agent is genuinely judging one thing. Splitting a single judgment into artificial sub-scores creates noise, not clarity.

Questions to ask

Ask these at kickoff, when design work starts. Tick them off as your team answers.

Example

An agent pre-checks an expense claim before a reviewer approves it.

Expense claim #4471 · Pre-checkDo
Policy categoryUnsure
The receipt lists a meal and a taxi; policy treats them differently.
Amount matches receiptMatches
Duplicate claimNone found

The uncertain check appears first, so the reviewer knows immediately where to look.

Match confidence: an agent tries to match an inbound support request to an existing case. Confidence is 0.58, below the threshold for automatic assignment. Instead of silently assigning, the interface shows both candidates with the reason for the uncertainty.

Support request · Match reviewDo
Two open cases match this topic and the requester's name. They were opened 3 days apart.
Case A
#4471 · opened 12 Oct
Case B
#4489 · opened 15 Oct
Check the reference number in the request to confirm which one.
Assign to Case AAssign to Case BCreate new case

Anti-pattern: The blended score

One overall confidence number across several checks: 0.82. The reviewer approves. What the number didn't show: the policy category check failed, and the other two were strong enough to pull the average up.

Expense claim #4471 · Pre-checkDon't
Overall confidence0.82
ApproveReject

Anti-patterns to pay attention to

  • A single score when the agent checked several independent dimensions.
  • Uncertain or failed checks appear after passing ones, or not at all.
  • The reviewer can't tell which check caused a low score.
  • A low-confidence match shows one resolved answer, with no alternatives visible.

References

Current guidelines · EU AI Act2 references

For orientation only, not legal advice. EU AI Act links open a readable version of the official text published on EUR-Lex; ISO/IEC 42001 is a paid standard.

  • EU AI Act Art. 14(4)(b) People overseeing the system must stay aware of automation bias. A blended high score is a strong interface-level trigger for it; separated scores interrupt that reflex.
  • EU AI Act Art. 14(4)(c) People overseeing the system must be able to interpret output correctly. A single number across several checks doesn't support that.

Evidence

Survey & researchSurvey insights, respondent quotes

Survey insightDemand for confidence signals rises with stakes. Respondents valued them most for high-stakes decisions and least for generating ideas.

“Aside confidence score, being able to see attached sources from which the AI referenced helps in both verification and trust of the answer provided.”

Survey respondent, open-ended answer

The survey is still open. Add your perspective ↗