Accuracy vs. Useful Feedback: Can Fitness Biosensors Revolutionize Prevention?





An interview with Matt Buman, Assistant Professor in the School of Nutrition and Health Promotion, Arizona State University

By Claire Topal, Senior Research Consultant, Center for Sustainable Health

What motivated you to conduct this study?

Matthew Buman, MS, PhD, is an Assistant Professor in the School of Nutrition and Health Promotion at Arizona State University. His research focuses on measuring and intervening upon the interplay of sleep, sedentary, and physical activity behaviors and how to harness these behaviors to prevent chronic disease.

Matthew Buman, MS, PhD, is an Assistant Professor in the School of Nutrition and Health Promotion at Arizona State University. His research focuses on measuring and intervening upon the interplay of sleep, sedentary, and physical activity behaviors and how to harness these behaviors to prevent chronic disease.

I’m very interested in behaviors that make up the full 24-hour spectrum. And those, to me, would be sleep, sedentary activities, and then more physically active behaviors, whether that is exercise or just walking to your car. I became curious about how we balance those behaviors.
We all know we need to be more physically active, sleep more, and have more restful sleep, and we’re starting to learn that we want to reduce the amount of time we spend being sedentary, independent of how much time we’re exercising. But we don’t really know the optimal combination of these behaviors and how we fit them together for different individuals. And that’s what brought me to be interested in wearable sensors and other commercial and research-grade devices that can measure both physical activity and sleep over time.

How much of the study took place in a lab, and how much in a free-living environment?

runningstairsThe first phase of the project, which ended in May 2014, was the lab-based validation portion, where we did very controlled activities with 288 people between the ages of 18 and 101. These included a range of mostly ambulatory activities, like walking at different speeds on a treadmill and up and down stairs in a very controlled environment. This allowed us to get a precise sense of how well these various sensors were counting those steps under different walking conditions.
Phase two of the project was what we call the ‘free-living validation portion.’ It’s one thing to know these devices are good under controlled settings, but it’s another thing to know how good they are under normal activities. In this phase, we sent 35 people aged 18-64 home for four nights and three days with a subset of the consumer devices to see whether their natural activities were being accurately counted. We also had participants wear another research-grade device called the StepWatch. While a very accurate step counter, the StepWatch is also very expensive and meant for research; it provides us with the gold standard against which to compare consumer devices.
In Phase Two, participants also took home sensors that measured their sleep. The gold standard there was the ActiWatch, a wrist-worn device that has a lot of established validity to measure key sleep metrics, such as sleep onset, total duration, etc.

Of all the measures of physical activity, why did you choose steps?

Steps are a very quantifiable and an intuitive measure of physical activity that people understand. Steps are also sensitive to change across a range of activities—not just traditional exercise, but also behavioral activities like spending more time walking by parking further away from the grocery store, taking stairs instead of the elevator, or other similar things.

Steps seem like a straightforward measurement, but sleep much more complex. How do you meaningfully measure sleep?


Sleep is very understudied. It’s surprising how little we actually know about it, even though it provides a relatively important barometer of our health. Measuring sleep is both fascinating and difficult.
Of course, you can ask someone about their sleep, and their answers give you a general sense of their sleep patterns. But oftentimes, their perception of sleep does not match up very well—objectively—with when we monitor someone’s sleep. That’s something that we’re grappling with: how do we reconcile our perception of sleep versus our actual sleep?
One way is through polysomnography – the gold standard of sleep measurement—where you go into a lab, typically in a hospital setting, and sleep there for 1-3 nights, hooked up to sensors that measure movement, respiration, and brain activity.

That approach offers a very precise quantification of objective sleep, but it isn’t feasible or cost-effective for the general public or for long-term monitoring. How do you measure sleep outside the lab?


Exactly our motivation for the sleep component of this study. For more personal in-home sleep monitoring that’s objective, we typically use accelerometers, like the ones in the UP band, Fitbit, and more research-grade devices in our study. In this case we’re not looking at brain activity but instead at movement as a proxy of sleep. The underlying assumption is that if you’re not moving (or moving less), then you’re probably sleeping as opposed to in bed awake. Obviously, there are problems with that approach, too.
For somebody who is generally a good sleeper a tool like a wrist-worn accelerometer can do a pretty good job of quantifying their sleep patterns. However, if you have a sleep disorder or general trouble sleeping at night, these devices tend to overestimate how much sleep you’re actually getting, and for these contexts they tend to be less accurate tools.

Why is a better understanding of individual sleep patterns valuable?

I think the broader picture is that there are lots of people out there who don’t necessarily have a sleep disorder but who could benefit from better or more sleep. The feedback that could come from these sensors could be very useful for them.
More easily-acquired feedback about an individual’s sleep could also provide helpful data points to a clinician who understands that sleep is an important risk factor for whatever condition they are looking at. For example, sleep is an important issue for heart failure patients. For those with diabetes, we know there is a link between sleep and pre-metabolic risk factors. As a result, understanding someone’s sleep generally from a lifestyle health behavior perspective, as opposed to a more clinical sleep disorder perspective, is very useful for many clinicians.

Stepping back from the nuts and bolts of the study, what does ‘validation’ actually mean in the context of fitness tracker-type wearable biosensors, and why is it important?


In this context, the term validation refers to whether or not a given device or sensor measures what it’s intending to measure. Most of these devices claim to measure steps, activity level, and/or sleep. Validity is really just confirming whether the device accurately measures those things, something else of meaning, or if it is just inaccurate.
The importance of validation has to do with what your goal really is. For example, if the goal is for a clinical diagnosis (very, very precise quantification) then there is a certain accuracy bar that you need to reach.
There are debates in our field as to what level of accuracy is needed for that sort of goal. For example, if a clinician wants to know with a high level of precision a quantification of activity, blood pressure, or weight status there is a certain bar of accuracy that needs to be reached. Many consumer-based tools might not be useful here.
Alternatively, if the device is primarily used as a feedback tool for the individual patient or even an ongoing monitoring tool to look at an individual’s trajectories over time for a clinician, then there’s quite a bit of utility for existing consumer wearable tools in clinical settings. To be useful in clinical settings in this case, you need reliability over time, rather than high levels of precision.
There are many smartphone apps already being used in clinical practice. The precision of that information is not as important as getting trajectories and the continuous monitoring of this information over time. That’s where, I think, these tools offer significant promise.

Why is continuous monitoring so important and what tool does it give us to improve healthcare?

handwithfitbitI can explain this best through an example. Let’s just say we are looking at weight in a heart failure patient. If the patient experiences rapid weight gain, that could be a sign of some trouble, right? And if you’re taking weight measurements once a week, once a month, you might miss that, or by the time you catch it, it might be too late to intervene effectively. And so having constant weighing through a system that allows you to do that and to communicate that information to a provider can be very, very useful. There is similar value for continuous monitoring of physical activity.
The future of this field lies in getting people to wear these sensors for very long periods of time. This way you get trajectories and patterns over time, not just snapshots of a week or two weeks. The potential is to create individual signatures of data over long periods of time or even a lifetime.

In addition to validating the accuracy of these devices, does your study also explore how and why individuals use the devices?

Yes, in Phase Three—the continuous monitoring (and current) portion of the study—we are interested in the usability rather than validity of the data. In this phase, participants consented for us to track their use of the Jawbone UP band and the corresponding smartphone app over the course of a year. Our key questions are: what are some of the characteristics of individuals who continue using the app? Who drops out? What are some of the factors that predict whether someone succeeds at changing their behavior over time?
We are essentially trying to understand who this [wearable biosensor device] system works for, and how we can improve the system for individuals it doesn’t work as well for. This phase will run through spring of 2015.

That information will clearly be useful for the companies that make wearable biosensors; who else can benefit?

BumanQ4I think the information from phase three of our study will offer important insights in terms of how we might design and better leverage devices and smartphone apps for healthcare. There are many smartphone apps and activity tracking devices on the market right now; typically, people using them are motivated and fitness-minded. The beauty of our study is that it follows people who, for the most part, are not as active as they could be. We hope the information we uncover can help the building of apps that will encourage people who are not already motivated to introduce healthy changes into their behavior.

What is the relationship between the validity and usability of a wearable biosensor?
There certainly needs to be a balance between them. While research-grade gold standard devices are the most accurate, they are also the least usable, comfortable, and the most expensive. By contrast, one of the appealing things about many of the sensors coming onto the consumer market is that the designers and manufacturers think carefully about long-term usability.
A lot of the consumer-based devices like the Jawbone UP and Fitbit are focused on making the device aesthetically pleasing, too. If you want people to wear it, they need to enjoy wearing it. If they’re not willing to wear it, then it’s not accurate at all.

Are you finding that these devices are catalyzing positive behavior change?
handwithwatchWhat interests me here is the distinction between measuring something and then giving feedback to someone to help them change their behavior. We know that monitoring or tracking alone (what these devices do) is not enough to really get someone to change behavior over time. You need monitoring plus some feedback and motivation.
I was involved in a project at Stanford where we built three smartphone apps focused on different motivational frames for increasing physical activity and decreasing sedentary time in midlife and older adults.2 The first app was focused on numbers and quantification, the second was emotion-focused, and the third was more of a social app where users were part of a team.
We found that all three of the apps were effective at changing behaviors, but there was an indication that what we call the ‘social app’ tended to do better. People tend to be more motivated by helping others and working collectively towards a goal as opposed to working towards a more individually-focused goal.

One last question: are you a good sleeper?
I’ve always been a good sleeper. Except when my kids were very little. But that had nothing to do with my physiology.

HoneyBee Engagement
You are on the Project HoneyBee Waggle Committee—named for the “waggle” dance honeybees do to instruct the rest of the hive where to find a food source. Please share a little more about your engagement in the project.
I’m involved in a couple of different ways. One is serving as a member of the Waggle Committee. I’m also working on individual projects linked to Project HoneyBee. One, clearly, is the validation study on which this interview is focused. Another is a project starting in the autumn, which will look at wearable sensors in clinical studies, examining early behavioral patterns both during and post-hospitalization. Specifically, we hope to understand what behavioral patterns might be predictive of hospital readmissions, with an overarching goal of being able to give more intensive care and counseling to these individuals in to prevent readmission.
How does industry fit into Project HoneyBee—beyond the fact that many private companies are creating and selling wearable biosensors?
Hospitals, clinicians, researchers, academics, and commercial entities are all engaged in finding better health outcomes at lower costs through the employment of biosensors. I think one important component of Project HoneyBee is that it explicitly asks and explores responses to the question of how all those stakeholders can partner to create the best systems for addressing key clinical problems
I’ve found the strength of commercial entities is in their ability to move much faster than academic institutions. They can build products more quickly and innovate more readily. There’s great value in that. Project HoneyBee is about partnership, understanding who has different strengths, and building multidisciplinary teams to solve the problems that matter most in clinical settings as opposed to working in very siloed, duplicative ways. Project HoneyBee is a very innovative pattern and system that is being developed to engage all those partners from the beginning.

*Consumer devices in the study include the Jawbone UP band,
Nike FuelBand, some Fitbit monitors, and the Omron pedometer.
**See King AC, Hekler EB, Grieco LA, Winter SJ, Sheats JL, et al. (2013) “Harnessing Different Motivational Frames via Mobile Phones to Promote Daily Physical Activity and Reduce Sedentary Behavior in Aging Adults.” PLoS ONE 8(4): e62613. doi:10.1371/journal.pone.0062613. Available at:

(480) 727-7577

Written by Claire Topal