When it comes to the development and discovery of novel digital biomarkers in clinical trials, one size does not fit all. Patrick Hankey, our head of strategic business development, sums up the problem very well when he explains, “If you have seen one clinical trial, you have seen one clinical trial.” In this article, we describe what we mean by “one size does not fit all” within the context of digital biomarker discovery and provide examples of how this principle is particularly relevant to the validation of wearable sensor solutions.
Validation Within a Defined Context is Essential
The collection of real-world data through wearable sensors is a key component of digital biomarker development. The Digital Medicine Society and other experts in the field have collaborated to devise a three-phase validation process for determining if a wearable sensor is fit-for-purpose. Fundamental to this process is to define the purpose or context, in which the device will be used. While this concept is not new in the world of drug development, the novelty and nuances inherent to wearable sensors have muddied the waters when it comes to the proper processes used to establish their validity.
FDA Clearance or Approval is Not the Holy Grail
Some wearable sensors have successfully undergone the FDA premarket evaluation program; others have not. This distinction is part of an essential risk-based regulatory framework put forth by the FDA. Still, it is not a comprehensive certification covering all use cases of the particular device. FDA clearance/approval evaluates the safety and efficacy of the device and its outputs within a clear and sometimes narrowly defined context.
The FDA has clearly stated that for wearable sensors used in clinical trials, 510(k) clearance is not compulsory (nor does it assure biomarker acceptance). Just because a device has FDA clearance doesn’t mean it is the one for your purpose. A process to gather the data to determine if the device is for your study is required. Even so, there is a need for a strong scientific rationale and robust, context-specific validity data.
Device Suitability Depends on the Context of Use
You may be interested in using a particular wearable solution in which strong validation data exist. However, it is essential to evaluate whether the existing validity data are within the scope of your use case or not. In this determination, it is important to remember that factors that impact either the “raw” signal or the metric to be measured can influence the validity of the wearable solution.
Validation of a wearable solution spans layers of hardware, software, and analytical procedures. For these reasons, they are best validated within a well-defined use case. When evaluating the suitability of a solution for your study, or when planning a validation study, it can be helpful to start with the end in mind. For example, who is your study population, how will they wear/use the device, and in what setting will they be studied. Three important considerations for wearable solutions for clinical trials are:
1. Hardware Specifications and Form Factor Impact the “Raw” Signal
Each device has its own unique set of specification factors, such as sensitivity and filtering ranges, data compression, and sampling frequency. For example, many devices contain tri-axial accelerometers to measure characteristics of gait passively, but they are not created equally.
Example 1: For example, band-pass filter settings impact the range of movements to which the accelerometer is sensitive. Filter settings that do not attenuate the signal at the low end of the spectrum are more sensitive to small movements, such as characteristics of shuffled or impaired stepping.
Example 2: Multiple form factors, such as different wear locations, also exist. The same device, with the same specification settings, wore on varying body locations (e.g., wrist, hip, ankle), produce unique acceleration signals related to how that body part moves during the given activity. Because hardware specifications and form factors have implications for how the device measures the “raw” acceleration signal, validity testing must be performed with these factors in mind.
2. Unique Clinical Characteristics Influence the Metric Measured
The wearable solution used to measure the digital biomarker must be valid within the unique clinical characteristics of the disease state being studied.
For example, the image below shows the stepping pattern characteristic of healthy individuals (a) and those of patients of different neurologic disorders (b-e). If the measured metric (e.g., steps) is performed differently from one clinical group to another, it should not be expected that the validity of a wearable solution is generalizable to multiple populations. For example, wearable step counting solutions that perform well for healthy individuals often underestimate the number of steps taken in various clinical groups that have slow and shuffled gait. Those that perform well for slow and shuffling gait, often overestimate stepping in healthy groups.
Image Source: Pirker W, Katzenschlager R. Gait disorders in adults and the elderly : A clinical guide. Wien Klin Wochenschr. 2017;129(3-4):81-95. doi:10.1007/s00508-016-1096-4
3. External Factors Impact the “Raw” Signal and Metric Measured
Wearable solutions are often validated within confined laboratory settings. This validation is a crucial first step in understanding specific factors that impact performance. Still, it is very different from the real-world environment in which devices are deployed.
In general, laboratory validations minimize the amount of within and between-person variability, limiting the extent of conditions under which the wearable solution is tested. For example, through standardized procedures, laboratory experiments control the amount of “noise” or unwanted artifact that can saturate the raw signal.
In this sense, we make it easy for the device to accurately estimate metrics of interest by ensuring that all data captured is “good data.” Realistically, sensors used in real-world settings are subject to large amounts of artifact or “bad data,” which aren’t useful to estimate the metric of interest.
Example 1: A good example of this is the effects of car travel on accelerometer estimates of gait. Car travel is one of the most regularly performed behaviors in real-world settings, but laboratory-based validations cannot easily test the effects of riding in a vehicle. A body-worn accelerometer is sensitive to the vibrations generated by the car (e.g., from the engine, or movement of the vehicle over the ground). Such accelerations create confusing signal artifacts that can lead to inaccurate identification of stepping behavior while a person is seated in a car.
Example 2: Common behaviors performed in real-world settings are also frequently performed very differently in the laboratory. For example, when step counting solutions are validated in the laboratory, patients may walk on a treadmill while a researcher manually counts steps. Device estimated steps are then compared to manually counted steps. However, in real-world settings, stepping behavior is not directly comparable to treadmill walking.
In the real-world, stepping can be short and intermittent (such as during activities of daily living), of varying speed, performed on varying terrain or as part of other primary activities (e.g., during sport). In short, laboratory settings do not directly reflect the highly variable and often unpredictable real-world environment in which devices are meant to be used. Knowing which factors impact the sensor signal and the metric being measured is essential to understanding the generalizability (or lack thereof) of the validity testing of wearable sensor solutions.
Most wearable solutions have not been validated to the degree necessary for their acceptance into clinical trials. This lack of validation dictates that small pilot studies with ground truth data be collected and presented to the FDA before trial initialization or that these sensors and digital biomarkers be deployed as secondary or exploratory endpoints.
“One size does not fit all” is our approach as we assist our customers in developing digital biomarkers that may be validated as primary endpoints.
In our eBook 4 Key Principles of Digital Biomarker Discovery, we expand on each of these four principles and provide specific examples from our own work and literature when appropriate. We will also explain how and why these principles are most impactful and relevant when biomarker development is conducted in early stage clinical trials or even earlier pilot studies