KYC Vendor Evaluation: What the accuracy number doesn’t tell you

In November 2025, Shufti’s Chief Commercial Officer Roger Redfearn-Tyrzyk opened a public webinar with a blunt claim. It was that most identity verification systems that advertise 99 percent accuracy are hiding the number that actually matters. Fraudsters do not need to break a verification system. They only need it to fail once, in their favour, and that single failure sits inside the 1 percent every vendor leaves off the spec sheet.

For a buyer running a KYC vendor evaluation, that missing 1 percent is the entire decision, not a footnote to it. A blended accuracy figure says nothing about whether a vendor’s errors let fraud through or block genuine customers, and nothing about the population it was measured against. What follows is the metrics and the checklist that separate a real evaluation from a spec-sheet comparison.

Is a headline accuracy number a good way to evaluate a KYC vendor?

No, not on its own. A single accuracy percentage blends two very different kinds of error into one number, hides the test population it was measured against, and is almost always self-reported. Two vendors could claim the same 99 percent and mean completely different things by it, tested on different document mixes, different fraud attempt rates, and different definitions of what counts as a pass.

In this guide, we’ll help you understand what these numbers mean and how you should approach vendor evaluations for your business.

Why 99 percent accuracy hides the number that matters

Any verification system can make two kinds of mistakes. 1) It can accept someone it should have rejected, and 2) reject someone it should have accepted (two terms also known as FAR and FRR that we’ll discuss below). A blended accuracy number averages both of these numbers, so a vendor can claim they’re strong at one, but they won’t publish their performance on the other.

Why self-reported numbers are hard to compare vendor to vendor

Every vendor tests against its own document set, its own synthetic fraud attempts, and its own definition of a valid pass, and all of these are typically tested on clean, lab data. None of that is disclosed on a typical spec sheet, and none of it is independently audited unless the vendor holds a specific conformance standard. That’s the reason it’s difficult to compare vendors based on only what they say in their marketing.

Compliance risk stays with you, not the vendor

It’s important to understand that no matter which vendor you use, the obligation to comply with the regulations is still on you. The 2023 Interagency Guidance on Third-Party Relationships from the Federal Reserve, FDIC, and OCC states plainly that a bank’s use of third parties does not diminish or remove its responsibility to operate safely and in compliance with applicable law. If a vendor’s false acceptance rate lets a sanctioned individual through, the finding lands on the buyer’s compliance file, not the vendor’s.

What is a false acceptance rate vs a false rejection rate?

A false acceptance rate (FAR) measures how often a system wrongly confirms someone who should have failed the check, and a false rejection rate (FRR) measures how often it wrongly turns away someone who should have passed.

Both definitions are derived from the same standard used across biometric and identity testing, ISO/IEC 19795-1, as documented in the NIST glossary.

The two metrics move in opposite directions, because a system that’s tuned to significantly cut FAR will almost always raise the FRR, and a system loosened to cut FRR raises FAR in turn. FAR is sometimes described in plain terms as the false positive rate in identity verification, and FRR as the false negative rate, though the ISO/IEC terminology is the more precise version to use.

High FAR is a compliance problem

A high false acceptance rate (FAR) means fraudulent identities, synthetic profiles, and sanctioned individuals get through the check. That could surface later, in a regulatory audit, a suspicious-activity report, or a fraud loss, and there’s nothing that could be done to fix it then.

A high FRR is a revenue problem

A high false rejection rate means genuine customers get blocked at onboarding. What will happen is that every wrongly rejected applicant will either abandon the signup or call support, and the cost shows up in conversion metrics rather than a compliance report, which is why it gets underweighted in vendor evaluations.

What is a good KYC pass rate?

There is no single good pass rate, because pass rate is meaningless without the population and document mix it was measured from. It depends on two things:

1. who applied
2. what kind of documents they submitted

A vendor tested on clean documents from countries with reliable ID systems will show a higher pass rate than the same vendor tested on a harder mix of documents from less standardised markets. That does not mean the latter is not a reliable vendor; it just means that it was operating in a different market with different challenges.

A real KYC pass rate benchmark, with its definitions attached

Shufti’s case study with Misterb&b, a travel platform, reports an 84.5 percent lifetime approval rate across the full verification volume since 2019, an 86 percent first-time pass rate, and an average verification time of 16.8 seconds.

Those are three different, clearly defined measurements, not one blended figure, and they belong to a single client’s deployment rather than a platform-wide average. That is what a usable benchmark looks like: a named metric, a stated scope, and a source you can check.

What criteria actually separate a real evaluation from a feature checklist?

A feature checklist asks whether a vendor offers document verification, biometric matching, and AML screening, and almost every vendor answers yes. A real evaluation asks how each of those claims is tested, disclosed, and proven on data that looks like yours. The table below works as a KYC vendor comparison checklist you can take into any procurement conversation.

Why the last row matters most

Every other row in that table can be answered from a document. The last row cannot. A vendor’s stated FAR, FRR, and conformance letter tell you what happened on someone else’s traffic. A proof of concept on your own applicant mix is the only step that tells you what happens on yours.

One thing worth measuring in that proof of concept is what happens after a match fires. In Shufti’s Voice of Customer research across more than 600 organisations, 45% named the manual gathering of context to resolve a match as their main difficulty in sanctions screening, well ahead of the 18% who raised list coverage. Accuracy decides how many alerts reach the queue. What each alert carries with it decides how long it sits there.

When is a headline accuracy number still useful?

A stated accuracy figure is still a reasonable first filter. It tells you whether a vendor is even in the conversation, and it is faster to compare ten vendors on one number than to run ten proofs of concept. The number just cannot be the last step. That is how to choose a KYC provider without over-relying on a single figure. Use the accuracy claim to build a shortlist of three or four vendors, then test the shortlist on the criteria above before any of them see a signature.

Shufti’s approach to KYC vendor evaluation

Most buyers cannot audit a vendor’s accuracy claim from a spec sheet, because the test data, the methodology, and the population are all controlled by the vendor making the claim. That gap is exactly what an evaluation process has to close before signature, not after.

Shufti’s KYC solution applies that same standard across its verification stack, publishing conformance artefacts that buyers can check independently. Shufti’s face verification solution layer holds iBeta Level 3 conformance under ISO/IEC 30107-3, tested by an independent accredited lab rather than self-reported, currently the highest published tier for liveness and presentation-attack detection. The conformance letter and the testing methodology behind it are available to review directly, which is the artefact a real evaluation is looking for in the first place, a claim you can check rather than one you have to take on trust.