WearableQA put 14 big language models through 4,084 health questions built from real wearable data. And most of them finished below 60% accuracy (papers.cool). WearableQA is a new benchmark for health reasoning over real-world wearable data, listed under Computation and Language on arXiv. Every question carries 10 answer options, which puts the blind-guess baseline at 10%. And the evaluated models spread from 19.6% to 72.9%. The authors’ own summary is that the benchmark “remains far from solved.” If you build anything where an LLM reads sensor streams, health records, or any messy operational data, that spread is your planning assumption.
What WearableQA Actually Tests
The benchmark comprises 4,084 multiple-choice questions, each with 10 answer options, constructed from wearable time series, blood biomarkers. And the demographics of 200 real users (papers.cool).
Each user contributes up to 500 days of daily measurements.
That is longitudinal data measured in seasons, not snapshots, and the questions sit on top of the full history.
The design choice that matters most is what the authors left in. WearableQA preserves authentic wearable distributions, including device noise and inter-individual variability, instead of scrubbing the signals into something tidy. I run a small AI automation agency. And I can tell you the demo always runs on the clean CSV.
Production data looks like the noisy version, which is exactly why this benchmark deserves your attention.
The questions come in 16 types organized along two axes. One axis separates data reasoning from health reasoning, meaning computation over longitudinal measurements versus physiological interpretation of what those numbers mean. The other separates single-signal questions from cross-signal ones, where the model has to combine heart rate, activity. And biomarkers in one reasoning chain (papers.cool).
That two-axis split is the diagnostic payload.
Because it tells you whether a model fails at the math or fails at the meaning.
The Scores: Best Model At 72.9%, Most Below 60%
Fourteen models, proprietary and open source, sat for the test.
Accuracy ranged from 19.6% to 72.9% against the 10% chance baseline (papers.cool). The authors state that “most models achieve accuracies below 60%” and call the whole exercise “far from solved.” They position it as a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data. And that spread is the point.
Read those numbers the way you would read them in a client report.
The floor model scored barely double random guessing on a 10-option test. The ceiling, 72.9%, still means the strongest system is wrong on more than a quarter of the questions. The 10-option format deserves credit here, as it strips out the inflation you get with four-option tests where partial understanding already lands near 50%.
My take: benchmark scores circulate as marketing, and this one should circulate as a reality check. Any vendor quoting high accuracy on wearable or health data without naming the evaluation is selling you the clean CSV. The whole value of WearableQA is that it refuses to hand one over.
Why Agent Frameworks Beat Raw Models Here
The same research wave contains the fix, and it is not a bigger model. WEQA, a query-adaptive agent framework (arXiv 2606.18147), uses an LLM controller that synthesizes execution plans and dynamically routes each query to a combination of sensor analysis modules and pretrained models, then performs grounded response auditing with external knowledge. In the paper’s experiments that architecture is 24% more accurate than LLM and agentic baselines. A blinded study with 12 medical experts and 8 users found substantial gains in usefulness and clinical soundness.
Its benchmark spans four open datasets, including TILES-2018, COVID-19 Sounds.
And PPG-BP, with over 1,100 question-answer pairs across four task families (arXiv HTML).
The same shape appears in PHIA, described as the first open-ended wearable question-answering system powered by LLM-based agents, running an iterative ReAct-style loop with code generation and web search (arXiv 2406.06464).
PHIA ships with two benchmark datasets totaling more than 4,000 questions: one set of objective numerical queries for automatic evaluation.
And one set of open-ended queries for human expert evaluation. In a 650-hour human expert evaluation, PHIA hit 84% exact match accuracy on objective numerical questions against 74% for a strong code-generation baseline. Its synthetic-user data was sampled from 30,000 users with heart-rate-enabled Google Fitbit and Google Pixel Watch devices.
A fourth effort, HEARTS, pitches itself as a unified benchmark for evaluating hierarchical reasoning capabilities of LLMs over general health time series (arXiv 2603.06638), widening the lens past any single wearable platform. Side by side, the field looks like this:
| Benchmark | Scale | Built on | Headline result |
| — | — | — | — |
| WearableQA | 4,084 questions, 10 options each | 200 real users, up to 500 days of daily measurements | 14 models scored 19.6% to 72.9%; most below 60% |
| WEQA | 1,100+ question-answer pairs, four task families | Four open datasets including TILES-2018, COVID-19 Sounds, PPG-BP | Agent 24% more accurate than LLM and agentic baselines |
| PHIA | 4,000+ questions across two datasets | Data sampled from 30,000 Google Fitbit and Pixel Watch users | 84% exact match vs 74% code-generation baseline |
| HEARTS | Unified general health time series benchmark | General health time series, beyond one platform | Evaluates hierarchical reasoning across LLMs |
The pattern holds across all of them.
The win comes from routing, tooling, and the audit layer, not from raw model intelligence.
Computation goes to code, interpretation stays with the LLM. And outputs get checked against external knowledge before anyone sees them. That is also the cheapest architecture a small team can build, which is the part nobody says out loud.
What This Means If You Ship AI For A Small Operation
Your data is the wearable stream.
Noisy, longitudinal, multi-signal, and collected under real conditions rather than lab ones.
The lessons from this benchmark wave transfer directly, whether you are wiring an LLM into CRM history, sensor logs, or inventory records.
– Build the eval before the integration, and split it the way WearableQA does: computation versus interpretation, single source versus cross-source. You learn which half breaks first instead of guessing.
– Treat benchmark claims as opening offers. Ask how a model scores on your distribution, with your noise, before it touches anything client-facing.
– Copy the agent shape. A controller that plans and routes to code and specialized tools beat straight model calls by 24% in WEQA’s experiments, and you do not need research funding to build that pattern.
– Keep a human reviewer for anything health-adjacent. WEQA validated with 12 medical experts in a blinded study, and that was for a research-grade system, not a weekend build.
In my own client work, the evaluation is the line item nobody budgets for and the thing that decides whether the automation survives contact with real data. Teams that skip it ship a demo that quietly rots.
Teams that build it catch the 19.6%-shaped failure in week one, before the client ever notices.
The wave building on real wearable data, from WearableQA through WEQA, PHIA.
And HEARTS, converges on one honest conclusion: raw model accuracy on messy real-world data is unimpressive. And routed, audited agents close a big piece of the gap.
Write your messy-data eval before the integration and budget for the routing layer, since that is where the accuracy actually lives.
If you want a second pair of eyes on an AI build like that, that is what I do at Mediascout. Get in touch.
