Research has suggested that wearable fitness trackers can be inaccurate. To test how well they measure calories burned, Healthista editor Anna Magee did some unique experiments for a feature in the Daily Mail – including one with sports scientists at the University of Bath. Don’t shell out your cash before you read it
Look around any high street and you’ll spot little rubber bracelets in various colours on the wrists of commuters as they go about their daily lives.
They’re wearing tracking devices that register calories burned, steps taken, distance travelled and even sleep quality.
Right now as many of us try to get fit for 2015, we’ll be sporting a shiny new one after receiving such gadgets for Christmas.
But a growing body of research is now suggesting that fitness trackers may be inaccurate and might not be the weight loss panacea they’re often made out to be.
Whether it’s a Fitbit, a Jawbone, a Nike+ Fuelband or one of the countless others being launched daily, the popularity of these fitness trackers is soaring.
Some 22 million were sold last year and UK company Juniper Research estimates that figure will reach a staggering 130 million by 2018.
‘The fashion accessory of the A-list’
They’ve even become a style statement. Vogue magazine called the Nike+ Fuelband – which measures calories and steps in an LED display on your wrist – the fashion accessory of the A-List and fitness trackers have been spotted on the wrists of Gwyneth Paltrow, Katy Perry and Jessica Alba.
But a raft of dieters on weight loss forums across the net are now joining threads called ‘My fitbit is making me fat!’
One, Tedens 711 says: Only third day with FitBit but I’m getting more calories and I’m eating them! Scared I actually bought a device that may slow or even undo my progression.’
Another, Hhorncastle states: ‘I’ve only been using my FitBit for five days but I’ve noticed something unsettling, I’ve gained three pounds’.
I have been drinking the fitness tracker Kool-Aid for the last six months.
I edit healthista.com and we get sent new health products to test everyday.
After a stint testing the Nike+ Fuelband, with its sleek, bracelet-like design I had become a fully paid-up member of the ‘datarati’, obsessively tracking my calories and steps.
After a stint testing the Nike+ Fuelband, with its sleek, bracelet-like design I had become a fully paid-up member of the ‘datarati’, obsessively tracking my calories and steps.
I gushed to anyone who would listen how having this information at my fingertips (literally, as you pressed a button and an LED display showed calories burned, steps, time and ‘fuel’ Nike’s measure of activity) was addictive and had me competing with myself to increase my daily activity.
But more recently, I was sent another new tracker, the Misfit Shine, launched last year. This swish looking device has a slim rubber band with what looks like a small platinum watch that you tap and flashing white dots signify not only the time but also how close you are to your daily activity goal. It seamlessly sends data to your phone via blue tooth so you can get an update on calories burned, steps and distance travelled anytime you have a net connection.
One day without thinking, I wore both the Nike+ Fuelband and the Misfit Shine, did my regular morning workout – a 30-minute circuit comprising light weight training, some cardio such as jogging on the spot and jumping plus a series of resistance moves such as planks and push-ups – and walked to work for 25 minutes.
After doing these activities, I noticed some 250 calories’ difference in the number the Nike FuelBand said I burned and that recorded on the Misfit Shine.
While the Nike Fuelband claims to only measure ‘active burn’ that done while being active and the Misfit Shine measures all calories burned, I still had no idea which one to believe.
I asked both companies – Nike and Misfit Shine – if they could send me studies conducted by the companies proving the accuracy of their trackers. Neither supplied them.
‘There’s no published data on the validity of these fitness monitors,’ says Gregory Welk, professor of kinesiology at Iowa State University in the US, who conducted a recent trial on fitness monitors.
‘These companies release a product and people buy it but they don’t supply anything by way of evidence showing they work.’
Indeed, in June last year, Professor Welk’s team tested eight different activity monitors to determine their accuracy and found some got calories burned wrong by up to 23 per cent.
‘If 2000 calories is a typical woman’s energy expenditure, 23 per cent of that is over 400 calories,’ he explains.
These companies release a product and people buy it but they don’t supply anything by way of evidence showing they work
That’s significant if you’re trying to trim down as to lose weight we’re often told to eat a calorie deficit of 4-500 calories each day because it takes around 3500 calories burned to lose a pound.
A tracker getting it wrong by over 400 calories could add up to the difference between my diet working and not. It was alarming.
But not prepared to give up on my fitness tracker addiction, I decided to conduct my own experiment by testing out other popular trackers including Jawbone UP, Fitbit Flex, Garmin Vivofit
, Misfit Shine and the Ki Fit Monitor.
My experiment to test accuracy – wear them all at once
I’d wear them all at once for a few days to see how big a difference – or not – they would show in calories burned, the main thing most of us are worried about when it comes to tracking activity.
The first day, a Friday, I did a workout in the morning and walked to and from work for 25 minutes each way then sat at my desk typing for the rest of the time.
The results ranged from 2196 calories burned for the day (the Misfit Shine) to 1591 (the Jawbone Up).
The next day, a Saturday, I pottered about the house doing light housework, did some grocery shopping at Lidl, lingered on Twitter and watched telly.
There was a 2-500 calorie range between them with the Misfit Shine saying I had burned 1841 calories and the Garmin VivoFit stating I’d burned 1393 calories.
I normally watch every morsel that passes my lips as I only have to sniff food to gain weight. But the next day I did what any food-loving woman worth her salt would do and chose to believe the higher calorie-burn read outs. Some had shown I had burned in excess of 2000 calories on previous days so I decided to treat myself to a huge fry-up while shopping in Ikea on Sunday morning! I haven’t eaten a fry-up since 2007.
That afternoon, I also treated myself to a piece of carrot cake over coffee with a friend in Costa. I’d burned all those calories so why the hell not?
Normally, I only eat cake on my birthday or at Christmas but in the back of my mind I had burned so many calories I felt I could eat more.
On the Monday, I stepped on the scales and lo and behold I had, like Hhorncastle, gained three pounds after only a few days wearing my fitness trackers.
So what was going on?
Firstly, my tracking devices are probably wrong and could be overestimating the number of calories I had burned, says trainer Jon Denoris (jondenoris.com).
Denoris, author of The Pop-up Gym
, has tested countless tracking devices in his London studio on clients or by wearing a number of them simultaneously.
‘We found a definite over-estimating of calories burned during our testing and saw variations of around 200 calories-burned over one 90 minute session,’ says Denoris.
We found a definite over-estimating of calories burned during our testing and saw variations of around 200 calories-burned over one 90 minute session.
‘Over time that could add up to quite a lot of misinterpretation of how much energy is being expended with the potential danger that the wearer thinks they’re burning more calories than they have done, and thinking that allows them to go off and eat more. That could ruin their weight loss efforts,’ he says.
Denoris adds that most fitness trackers use something called an accelerometer to measure movement of the body. These, he says ‘have been shown to be wildly inaccurate.’
How do they work, anyway?
Accelerometers use electronics to determine bodily movement and intensity; they sense movement and feed data into the device where equations calculated using averages based on your height and weight determine how many calories you burn.
‘Where you place them is important too,’ he explains, adding that when people are cycling and their arms aren’t moving, an accelerometer placed on their wrists often thinks they’re sitting down.
Still not prepared to give up my dream of finding a tracking device that accurately recorded every calorie burned and step taken, I called each of the companies whose trackers I had tested and asked to see evidence for trials they had undertaken showing their accuracy.
Only one supplied it, the Ki Fit Monitor, which was shown in a study published in the British Journal of Sports Medicine to have an error rating of less than ten per cent.
I called each of the companies whose trackers I had tested and asked to see evidence for trials they had undertaken showing their accuracy. Only one supplied it
In Professor Welk’s research too, the Ki Fit Monitor was found to be the most accurate, but at nearly £270, it’s expensive and out of most people’s price range.
Plus, it’s like a thick bandage worn on the top of your arm and errs towards the wrong end of the style spectrum.
It was time to find a more scientific way to measure the accuracy of these tracking devices.
The scientific experiment to test accuracy
When movement scientists want to track the calorie expenditure of different activities, they use a machine called Indirect Calorimetry (IC).
Users wear a huge mask that they breathe into while doing activities and it analyses the amount of carbon dioxide production and oxygen consumption to estimate calories burned.
A version of IC was used in the James Bond flick Skyfall in which Bond, played by Daniel Craig was tested to his physical limits with an IC machine.
‘Indirect calorimetry has been used for 100 years and is the gold standard technique for the estimation of calories burned,’ says Dylan Thompson, Professor of Sport, Health and Exercise Science at the University of Bath.
Wearing all my devices, I visited Professor Thompson’s lab in Bath to see how they would perform against IC, which he explained is the measure most universally accepted as the most accurate for calorie burn.
My experiment involved doing four activities, walking on a treadmill, cycling on a stationary bike, typing and packing groceries and comparing the calories burned on the IC machine to the different devices I was wearing and seeing how they stacked up.
I was hooked up to the IC machine, a mask placed over my face that I breathed into attached to a long pipe that linked to a large plastic bag.
From there my gas exchange was inputed into a computer that interpreted the data and calculated the number of calories I’d burned.
Most surprisingly, during the typing, the Misfit Shine thought I’d burned 88 calories when in reality, the IC machine showed I had burned just nine.
during the typing, the Misfit Shine thought I’d burned 88 calories when in reality, the IC machine showed I had actually burned just nine
‘A tracking device worn on the wrist while typing might over-estimate the number of calories you burn because your wrists are moving, so they think your body is moving,’ says Professor Thompson.
On the other hand, when I cycled, IC showed I had burned 69 calories while the Nike+ Fuelband thought I had burned just 3! The most accurate during cycling was the Ki Fit Monitor, which registered 54 calories burned.
when I cycled, IC showed I had burned 69 calories while the Nike+ Fuelband thought I had burned just 3!
Dr Oliver Peacock, a research officer in Professor Thompson’s lab who helped us with our experiment said most tracking devices would have a hard time registering calories burned through cycling because of where the trackers are placed.
‘During cycling, most of the movement is done with your lower body,’ he explains. ‘When we have tested activity monitors in the past, we have found they often fail during cycling tests because the wrists don’t move therefore the monitors think the rest of the body isn’t moving.’
After I completed and recorded the number of calories burned during the four activities, I worked with Professor Thompson and Dr Oliver to see how the different devices measured up against IC, the gold standard measure of calories burned.
So how accurate were the monitors?
The results were not impressive. While the FitBit Flex over-estimated calories burned by 23 percent, the Misfit Shine overestimated them by a staggering 102 per cent!
‘Every device will have some room for error,’ says Professor Thompson, and generally speaking 10-20 per cent over or under the number of calories burned is fairly acceptable as none will be perfect.
FitBit Flex over-estimated calories burned by 23 percent, the Misfit Shine overestimated them by a staggering 102 per cent!
But the experiment shows how vastly different the outputs of the various devices may be, he explained, adding that his department can’t take responsibility for the accuracy of my experiment as it ‘wasn’t as controlled as it would be if it were a research experiment.’
So, what to make of all this?
Fitness trackers are not a great weight loss tool as they’re only designed to track activity, not food, says Cara Sloss, state-registered dietitian and spokesperson for the British Dietetic Association.
‘These fitness trackers are not usually targeted at losing weight and because most of them are calculated using what the average person of your height and weight is going to burn, it’s probable that they will not only be wrong, but might also over-estimate the number of calories burned,’ she explains.
‘This means that someone might look at yesterday’s calories and think ‘Oh I have burned 2000 calories’ and then think they can eat an awful lot more than they would have done, had they not been wearing a tracking device,’ says Sloss.
In fact, a 2010 study published in the US Journal of Clinical Nutrition found that those on a 24-week tracked exercise programme compensated by eating more food – exactly what had happened to me.
‘If you really want to lose weight, you need to focus on your diet as well as your exercise as it’s difficult to focus on just one of those and lose weight,’ she says, adding that studies have found those who also track their calories consumed through an online calorie counting tool such as myfitnesspal.com or food diary tend to lose more weight that those that only track their activity.
Indeed, recent US research by Endeavour Partners, earlier this year found fitness trackers often go the way of the gym membership, with a third of people abandoning theirs after just six months.
After my experiment, I’ve decided to join them. I am going back to that simple equation of moving more and eating less. It’s low-tech but it works.
My verdict if you’re looking to buy a body monitor
If you love data and detail as much as I do, it’s hard to resist a little gadget that spouts out your daily activity in coloured graphs and numbers. But don’t take it as gospel and don’t eat more to make up for calories burned as I did. Given the evidence, it’s better to assume that the burn you’re seeing recorded by your device is about 20 per cent over what you’re actually burning.
Given the fact that – other than the Nike Fuelband which I actually got last Christmas as a present from my husband – all the other devices were given to me as press loans, people often ask me which I would actually shell out my own cash for. After this experiment, I need to be honest, I wouldn’t buy the Nike Fuelband again. I don’t like how it only measures ‘active calories’ (as though calories spent doing everything other than actually working out don’t count, I want to know every calorie I burn thanks!), Plus, in some of my experiments with the scientists it was way off accurate.
Having looked so closely at all of them, the Ki Fit was the one that provided clinical evidence of its effectiveness and accuracy. However that doesn’t mean I would actually buy it – I wouldn’t. At nearly £270 (which only buys you a year’s subscription) it’s way too expensive on my journalist’s (and most average people’s) wages – though it’s data and detail is second to none (you can see mind-numbingly detailed graphs that measure your burn, activity and sleep quality by the minute).
The Garmin Vivofit came first or second in all our tests, with the exception of cycling — where it wildly underestimated the number of calories I’d expended. Although I’d always far prefer that something underestimated than overestimated calories burned, lulling you into a false sense of security and giving you licence to over-eat. In fact, the Garmin only thought I’d burned more calories than I actually had during one test, the typing test — if I was typing, as I often do, for eight hours, I would think I’d burned 960 calories when I’d only really have burned 600. Still, I don’t like the look of it.
‘Every device will have some room for error,’ says Professor Thompson, although he points out that our trial wasn’t as controlled as a proper research experiment. Still, I wouldn’t invest in the Misfit Shine, which overestimated my calorie burn in almost every task — by more than 700 per cent in one case.
That all said, one I would buy is the Jawbone UP (£75.99 from Amazon)
. Despite its shortcomings in accuracy, I am a stickler for simplicity, style and convenience. It not only looks stylish, it delivers the data (also on burn, activity, steps and sleep quality) to your phone via an app that updates wirelessly. The battery lasts ages and it’s super-easy to charge. Plus, it’s got the simplest interface for delivering the data, colourful and clear. Just don’t take it too seriously.
A version of this feature first appeared in the Daily Mail
Like this article? Sign up to our newsletter to get more articles like this delivered straight to your inbox.
