Why Liveness Vendors Should Be Tested on the Phones Real People Use

The way most identity verification systems get tested does not match the way they get used.

In the lab, the cameras are clean. The lighting is controlled. The phones are usually the latest models, often the same generation as the engineers who built the software. The numbers come back impressive. 99% accuracy and zero attacks accepted. This is the kind of result that looks great on a marketing page.

In production, when the same system has to verify a freelancer in Manila, a delivery driver in Lagos, or a customer signing up to a remittance app on a four-year-old Android, the numbers can look very different.

This gap between lab performance and real-user performance is the variable most vendor comparisons quietly skip. It is also the variable that most decides whether your onboarding flow actually works.

Where Liveness Lab Numbers Come From

When a vendor publishes a liveness result, it usually comes from one of two places.

The first is internal testing. The company runs its own evaluation on its own setup, picks favourable conditions, and publishes the result on a marketing page. There is no third party watching.

The second is independent lab certification. Organisations like iBeta Quality Assurance run formal tests against international standards such as ISO/IEC 30107-3. These tests are taken seriously by procurement teams because they are audited, repeatable, and evidence-based.

Both produce numbers. But the most important question, what hardware the test was actually run on, often gets buried in the fine print or skipped entirely.

The Hardware Most Vendors Choose

Look at the published lab results in the liveness space over the last two years and a pattern shows up. The hardware is almost always premium-tier and recently released. The newest iPhone Pro, latest Google Pixel, or a flagship Samsung from the current generation.

These are excellent phones. High-resolution sensors, powerful chips, the best camera tuning the manufacturer offers. They make liveness systems look good.

But they do not represent the device a typical user is holding.

Across many of the markets where digital onboarding is growing fastest, including Southeast Asia, the Gulf, Africa, and Latin America, mid-range and older phones make up the majority of active devices. The customer signing up to a fintech in Kenya or a wallet in Indonesia is unlikely to be on the latest flagship. They are more likely on a four-year-old Android, a budget Samsung from 2021, or a Pixel from 2019 still being passed through a household.

If a vendor’s lab result was produced on hardware that customer does not have, the result is a poor predictor of what happens in production.

Why This Gap Matters in Procurement

Identity teams writing requests for proposal (RFPs) for liveness usually focus on three things: lab certification level, accuracy numbers, and price. The hardware question rarely makes it onto the comparison sheet.

This is a problem. The same vendor’s system can produce different results on different devices. Camera sensors, processing power, and color rendering can make big difference. A passive liveness model that was trained and validated only on premium hardware will sometimes misfire on a mid-range chipset because the input signal is genuinely different.

For an onboarding team operating in a market with broad device diversity, a vendor that only proves itself on flagship phones is taking a quiet risk on the team’s behalf. The risk shows up as drop-off, false rejections, and complaints from users who did everything right but still failed verification.

The Test That Actually Predicts Production

What buyers should look for is independent lab testing on devices that match deployment reality. Not just any phone. The phones their users actually have.

In practical terms, this means asking a vendor for the device list from their most recent lab evaluation. If the answer is two flagship handsets released in the last 18 months, that is a yellow flag. If the list includes mid-range or older devices, the result is more credible.

It also means asking about demographic coverage. Lab testing under ISO/IEC 30107-3 typically samples across age, gender, and skin tone. Not all certifications make that data public, but the ones that do are the ones worth weighting more heavily. A liveness system that was tested only on a narrow demographic will perform unevenly in production.

How Shufti Approached that Question with Independant Evaluation

In Shufti’s most recent independent lab evaluation, the test devices were a Google Pixel 4, released in October 2019, and an Apple iPhone 12 Pro, released in October 2020. At the time of testing, those devices were five and six years old.

The result was a 0% attack acceptance rate and a 0% rejection rate for genuine users across 900 presentation attacks under iBeta PAD Level 3, the highest tier of the ISO/IEC 30107-3 standard.

The test crew was deliberately diverse: 40% female, 30% non-Caucasian, with age groups represented.

Both choices were made on purpose. Shufti operates across 240+ countries, and the platform has to work in markets where mid-range phones make up most of the active install base. A test that did not reflect that reality would not have told us, or our customers, anything useful about how the system actually performs.