Two runners. Three wearables with wrist-based heart rate monitoring. Twenty four graphs. Wareable reports from the front line of running wearables and fitness trackers to investigate whether heart rate tracking watches are accurate enough to be trusted.
It's not surprising that watches with built-in continuous heart rate monitoring have set pulses racing this year. No more stinky, sweaty chest straps wrapped around your torso.
Instead, LEDs on the bottom of the watch peer through your skin and monitor the flow of blood. All very clever and convenient. Over time, you'll also get the added motivation of seeing your resting heart rate decline as your fitness improves.
But are these new heart rate monitors accurate? If not, it's little use for training.
Why we need accurate sensors
Heart rate training relies on exercising in different heart rate zones - each of which is a percentage of your maximum heart rate and stimulates different metabolic pathways and has different effects on the enzymes in the muscles.
The aerobic zone, for example, is generally recommended for overall cardiovascular fitness and is considered to be 70-80% of your maximum. You can estimate your maximum heart rate with a formula based on your age - though, be warned, their accuracy varies. For better accuracy you can take a professional test, which involves a particularly unpleasant workout.
Once you know your zones you can schedule workouts to make the greatest fitness gains without overworking your body, that is if you are measuring the zones correctly.
Chest strap heart monitors have been in use for a while. They detect the electrical activity that is transmitted through the heart muscle to make it contract and are deemed accurate enough, by fitness specialists such as Garmin, Polar and Suunto, to use effectively in training with heart rate zones.
The accuracy of wrist-based heart rate monitoring is still up for debate.
How manufacturers test HRM wearables
While wearable tech companies, including HRM darling Mio, cite studies that demonstrate the precision of wrist-based HR monitoring, we found these studies, often tested the sensors alone, not completed watches. These sensors were then secured under a sweatband, protected from light and movement. That's important, as we'll see shortly.
Often these studies were conducted at running speeds that are slower than most serious runners would be doing in the real world.
Worse, when it came to reporting findings, the studies took an average of walking, jogging and running results. For runners only the third reading is relevant and, from what we saw, it's often the least accurate.
The real world Wareable test
We wanted to know how good the latest generation of sport watches with built-in HR monitors are. We test heart rate tracking in individual Wareable reviews but wanted to focus in on this metric with a dedicated investigation. How accurate are they in real world use? We devised a test to find out.We took two runners, and had each of them run while wearing a wrist-based heart rate monitor and a chest-based one that they had verified as accurate after a period of regular use.
We used two runners because we wanted to get at least some sense of whether different heart profiles would be read differently. Runner one, your author, runs 15-20 miles per week and, while he goes slowly, his heart doesn't. On an economy run, an average HR of 150 - in the upper end of the aerobic zone - is not unusual. He compared the three wrist-based monitors with a Suunto Ambit3 Run and chest strap.
Runner two, Paul Radford, runs ultramarathons and has five wins and four course records to his name. He hasn't finished outside the top 10 since 2011. He runs about 80 miles a week - averaging a bpm count about 114 - and used a Polar V800 with chest strap as his comparison device.
Studying our results would be Sean Radford, Paul's brother, who is a doctor, runner and long-time coach. Sean is the founder of TrainAsOne, currently in beta, which delivers customised training programmes for runners based on data collected from their wearables. He has plenty of experience in assessing training methods, wading through medical reports and making sense of medical assessments of fitness gadgets.
Just to show our working, we've published the graphs we were able to produce from the heart rate data. They show the recorded heart rates, the differences between the wrist based devices and the chest strap and also the overall discrepancy.
A proper scientific survey studying these heart rate monitors would include many runners, strapped to medical-grade equipment and running - not walking or jogging - on treadmills. Our test doesn't provide this but it does at least replicate the experience of real runners. If you buy one of these devices and put it on for a run, how reliable can you expect the data to be?
Adidas miCoach Smart Run
First up was the Adidas miCoach Smart Run, a chunky device with a broad strap that, in white at least, is fairly ugly and bulky. It has a clear display, though the the user interface could be simpler.
Here's why it matters. The Smart Run, and Adidas' latest wearable the Fit Smart both use Mio's continuous optical heart rate monitoring sensor which Mio CEO Liz Dickinson says is more accurate than the tech in rival trackers such as Fitbit devices. It's also the tech Garmin has chosen for its first ever running watch with wrist-based heart rate monitoring tech, the Forerunner 225. High praise indeed.
For your author, averaging 150 bpm, the Smart Run did a good job. It was accurate 90% of the time and was never out by more than five beats per minute. That's a level of accuracy which could definitely replace your chest strap.
For Paul, who averages 114 bpm on his runs, the Smart Run was slightly less reliable. It was in the correct zone 86% of the time and very occasionally - about 1%t of the time - it was out by two HR zones. That's probably on the cusp of viability as a running device. A casual or mid-level runner might accept Mio's 86-90% reliability as a trade-off for ditching the chest strap. A more serious runner, or someone who demands maximum accuracy, might not.
The next device was the Basis Peak, which will track runs as well as monitoring all-day activity and sleep. While it's true that looks are subjective, we have to say this watch is no looker, either. It feels a little cheap. Also, perhaps because of its role as an all-day activity tracker, the Peak doesn't offer the kind of detailed information a runner wants.
Basis says it combines an optical sensor with special algorithms to continuously measure heart rate 24/7. It has done internal testing and is in the process of getting third party validation.
Is the Peak's HRM accurate? Not brilliantly. Your correspondent's readings were in the correct heart rate zone just 70% of the time and for a small portion of the run - two per cent - they were two heart rate zones out. The readings varied more, too, and were often 10 beats per minute out.
The Basis worked a little better for Paul. It took some time to pick up his run and was under-reading by about 18 beats at first; you don't start a run on the Basis, it notices the activity by itself. Once it did pick up the run, however, it stayed within five beats and was in the right zone 75% of the time. Even so, neither one of us would be happy to replace chest strap data with that.
Finally, we turned to the Fitbit Surge.
This was the best looking of the three devices, though its small screen doesn't fit much information - not ideal when you are on the move. It was simple to set up but, once again, accuracy was suspect.
As for the heart rate tracking, the Surge uses Fitbit's PurePulse tech which can also be found in Wareable's current best overall fitness tracker the Fitbit Charge HR. The Charge HR was our first choice for this in-depth test but as we weren't able to export the full data (just daily totals), we took the Surge out running instead.
So how did it do? The Surge seemed to take about five minutes to properly pick up the heart rate for your correspondent. Once it did, it was in the right heart rate zone 77% of the time.
Paul's test with the Fitbit saw it perform marginally better. It took eight minutes to settle down and start reading accurately but once it did, it was in the right heart rate zone 82% of the time.
Back to the chest strap?
As we've said, this experiment was not a scientific test but if it was, it would be one of those that concludes more research is needed. These three devices do not appear to match the accuracy of a chest strap, with the best performance we saw being Mio's HRM tech at 10% out.
Why aren't these devices as accurate as a chest strap? One clue is that they seem to be more accurate when not running and, in the studies we've seen, they are more accurate for slower runners.
One educated guess is that the movement of the arm is to blame. Generally the faster you run, the faster your arms move and this may either cause the device to shift on the wrist, hampering accuracy, or perhaps stop the sensor from getting a clear view of blood flow.
It's also possible that small discrepancies in the fit are magnified when you are in motion. Though we followed the manufacturer's recommendations in putting the devices on, perhaps a slightly too tight or too lose watch struggles when you are running.
More worrying is why the two of us saw different results with the same device. Perhaps the difference in our heart profiles accounts for that - two of the three devices, the Fitbit and Basis, were more accurate for Paul's lower bpm, whereas the Adidas was vice versa. Differences in wrist shape could also be partly to blame and as we said, particularly with Mio's tech fit is important to get the correct readings.
As for Wareable's verdict, if you don't mind sacrificing 10-15% accuracy versus a chest strap, the Adidas miCoach Smart Run is clearly the best of the bunch, and the just-tested Garmin Forerunner 225 which uses the same sensor is worth considering.
If you are in the market for a running watch and heart rate monitoring is of secondary importance then these devices are fine; you will get a reasonable estimate of your average heart rate for a run.
However, if you are doing training based on heart rate zones or you want the most accurate data possible then we cannot recommend them. Our doctor/runner/coach Sean Radford's verdict is that having already been skeptical about these wrist-based heart rate monitoring devices, he still isn't convinced that they are ready for serious runners.
In other words, you'll be wearing that stinky, stretchy chest strap close to your heart for a while yet.