This is the next in our series of blog posts to keep you up to date on our ongoing involvement with the FDA Pre-Certification Pilot program. In the first post, we shared some of the “Excellence Principles” and “Common Validating Principles” that the FDA asked us to think about. In our second post, we shared how we think about building software in 101 Questions to Ask Yourself if you are building Medical Software. In this post, we talk about our most recent meeting with the FDA and examine the revolution that is under way. –HL
TL;DR version of this blog post
The FDA is revolutionizing how digital health products get to market. These changes will be a great benefit to public health and will cause digital health solutions to reach the public as quickly as possible while maintaining safety and efficacy.
These FDA changes have leapfrogged over much of the diabetes industry. It’s time for the industry to do more, faster. Imagine that.
Every company in the diabetes industry needs to seriously examine how they think about and interact with the FDA and public when it comes to digital health. Here are a few suggestions:
- Rethink the way you interact with the FDA. Companies that historically have held their cards close to the vest with the FDA should reconsider their approach. The FDA is a partner that wants to get better, safer, more effective digital health solutions to market as quickly as possible. Engage with them early, engage often, and be as open and transparent as you can.
- Participate! Companies that chose not to participate in the Pre-Cert program because they were worried about the impact to current products in the pipeline should have participated. Even if you didn’t apply or weren’t selected, you can still be a part of this revolution. Register to attend the January 30-31 public meeting about the Pre-Cert program. Engage with the other Pre-Cert participants to help make sure we get this right together. Also consider joining the conversation in our #fda-precert public Slack channel.
- Put public health first. Companies that have carefully designed and crafted their products to avoid regulatory oversight should reconsider doing more to benefit public health. If you left a decision support algorithm out of your product to keep it at Class 1 instead of Class 2, instead think “I’m not afraid of Class 2, and I can do more to benefit public health if I do it. I’ll engage with the FDA and figure out how to do it quickly, safely and effectively.”
- Ditch conservative, old-school regulatory processes. Companies should be asking, “How could we make sure our software/device is safe and effective in the most efficient way possible?” The FDA is totally open to novel approaches to quality systems. If your quality system is modeled after a guidance document written before the Internet existed, you’re doing it wrong, it’s slowing you down, and it’s impacting your organization’s employee satisfaction and bottom line.
- Listen to feedback. Companies that that are not taking advantage of customer feedback channels like social media are abdicating their responsibility to public health. The FDA, and your customers, want you to hear their feedback so that it helps make your products better. You need to do better.
If we all get this right, it is going to have incredible benefits for public health, not just in the diabetes community, but across the board:
- Companies that can demonstrate that they are good at building software (and eventually hardware devices, too) will get a “fast lane,” (think TSA Precheck) to get their products out more quickly.
- Companies like Apple, Google and Fitbit will be able bring all of their software and consumer electronics expertise to bear and deliver digital health products the way they already know how to do, without onerous regulatory implications.
- Smaller companies like OneDrop, MySugr, Glooko and Tidepool will be able to make a bigger impact because it will be easier to deliver digital health solutions that previously had cumbersome regulatory implications.
- We’d also love to see open source, do-it-yourself projects able to participate. Imagine being able to download official versions of Loop, OpenAPS or Nightscout from an app store, and have it work seamlessly and reliably with interoperable pumps and sensors.
So let’s go, diabetes industry and community: don’t stand idle. Get involved! Make some bold changes. Help this revolution succeed.
It will be good for your business and, more importantly, it will be good for public health.
Why this is important
Let’s start with an astonishing statement: When it comes to digital health, the FDA has leapfrogged industry. The FDA is now pulling industry to go faster, but industry has not yet responded. Many companies are still being guided by old-school regulatory groups that believe they need to comply with old-school regulatory guidance, much of which was invented before the Internet even existed. This overly conservative approach is causing digital health products to take way too long to get into the public’s hands, and this delay is a detriment to public health and corporate bottom line.
On the bright side, the FDA gets this, and is actively trying to fix it. They understand, and openly admit, that prior attempts to regulate software and other digital health projects have backfired. Industry adherence to slow-moving, old-school regulations and guidance documents has had terrible unintended consequences that have been detrimental to public health:
- Some companies that really should be shipping medical products are choosing not to because they didn’t want to undergo burdensome regulatory processes.
- Some companies are choosing to not add features to their products that could be of benefit to the public health, because they don’t want the additional regulatory burden.
- Some companies can’t get funded because VCs are scared off by anything that could be impeded by regulatory processes. This means that some potentially revolutionary healthcare innovations never even get off the ground.
Without intending to do so, the agency tasked with protecting the public health and safety has hurt it by previously enforcing regulations that simply could not keep up with the pace of technical innovation. These are unintended consequences, but consequences nonetheless.
Fortunately, things at the FDA are now changing.
On December 20-21, 2017, Tidepool had the honor of spending two days, in person, with Bakul Patel and the team from the FDA that is driving the Precertification Pilot Program. The FDA has done site visits with all of the pre-cert pilot participants, and is now preparing for a public meeting.
We covered a lot of ground in this meeting:
- We shared our own “scorecard,” a work in progress based on 101 Questions for how we think about building safe, high quality software.
- We shared a presentation with examples of tools and processes that we use.
- We talked a lot about the use of metrics and Key Performance Indicators (KPIs) as measures of delivering safe, high quality software, and the pros and cons of doing it this way. (More on this below)
- We talked about scaling the precertification processes so that they would work for both large, well-funded companies like Apple and Google, small companies like Tidepool and even do-it-yourself projects like Nightscout, Loop and OpenAPS. (More on this below, too).
Using Metrics and KPIs – Will it be sufficient?
We talked a lot at our meeting with the FDA about the use of metrics and KPIs as a method for validating that a company has a “culture of quality and organization excellence.”
In short, the FDA’s vision is that there will be a library of Culture of Quality and Organizational Excellence (CQOE) questions and KPIs, and that each company can in essence say, “here’s the dashboard of KPIs that we look at to make sure we are shipping safe, effective products.” Appropriately, they think about these KPIs along five dimensions:
- Patient Safety
- Product Quality
- Clinical Responsibility
- Cybersecurity Responsibility
- Proactive Culture
I will admit that I went in mildly skeptical about a KPI-driven approach. As I wrote last month, the way Tidepool thinks about building high-quality software is a little different than the draft questions the FDA proposed. I still have some concerns that no approach can be fully quantitative, as there’s the chance it will become overly rigid, causing unintended consequences like before, or that the KPIs will become easy to game, and will not meaningfully benefit public health.
That said, the conversation we had with the FDA around metrics and KPIs exposed that there are indeed some interesting ways to have quantitative measures. The FDA team was inquisitive and open to our approach to building software, and also pushed us think about how some of these measures could be turned into KPIs that we would use on a regular basis as good indicators of safety and quality.
Tidepool and KPIs
Here’s an example of how a KPI could be used effectively, based on one of the “deep dives” that Tidepool did with the FDA team:
It’s critically important for companies delivering software and medical devices to understand and mitigate the risks involved with a product. Here’s the evaluation process Tidepool uses. It’s similar to the risk analysis template that a lot of companies use and is derived from ISO 14971. (Sidenote: Hey AAMI, ISO and lots of others: important standards docs like this should be free and public if you want them to be used and followed!).
In Tidepool’s system, like many others, risk is analyzed along the dimensions of severity and probability. Anything that is green in the chart above is OK to ship with. Anything that is red is definitely not OK to ship, and we should drop everything to fix the issue. Yellow risks need to be mitigated with a plan in place to fix.
You may have noticed risk level 11, which is the possibility of shipping with a known Catastrophic issue (one that could potentially kill someone), even if it’s highly Improbable. I wanted to address this, as this is something that folks may wonder about. Imagine this hypothetical scenario: You are delivering a closed loop app that has a risk of a lethal overdose to 1 in 2 million people. But by not delivering the system at all, those 2 million people will stay on self-administered open loop therapy, which has a 1 in say 200,000 chance of delivering a lethal overdose. In that case, it would be better to deliver the system since it would save lives.
Capturing the hypothetical risk of a bug, however, only tells half of the story. What about the issues that your customers report, perhaps by social media, or the hazards that actually occur after you ship? (In this context, “hazard” is a “risk” that actually happened to users. It’s real – not hypothetical).
It’s important for a company to take seriously actual customer reports of issues that occur, and demonstrate that they aren’t affecting product quality or safety in a way that they didn’t anticipate, and that the company isn’t seeing repeat issues that should have been previously mitigated or fixed.
What KPIs might you measure to see how you are doing in this regard? It might look like this:
(Apologies for the handwriting, but we want you to see the real work.)
In other words, a good KPI to measure your follow-through might be, for each risk level:
Risks Mitigated / Risks Identified
If this ratio approaches 1, you are demonstrating good follow-through.
Another KPI might be:
Number of Level 12-25 Hazards Reported / Level 12-25 Identified Risks Mitigated
You certainly would want this KPI to approach zero.
Finally, you could imagine a third KPI being:
Recurrent Level 12-25 Bugs
You also would want to be a very small number. A large number would indicate that you aren’t fixing bad issues that come up, or that you don’t have the right automated tests in place to ensure that once fixed, they don’t come back later.
Why Openness and Transparency Matter
We had a wonderful conversation about openness, transparency and reproducibility.
In short, we believe that when a company shares more about what they do, they can build safer systems. Being open allows others to inspect what you do and how you do it. In the best case scenario, others can build and reproduce what you have done, and alert you to safety or cybersecurity issues. Karen Sandler has written and spoken at length about this topic with respect to cardiac devices.
That said, it’s unreasonable to assume that all companies will be open and transparent. Some companies simply consider their product development processes and details to be top secret, and we are OK with that. But companies need to understand that within healthcare, in some cases you’re asking patients to trust your products with their lives. And often patients don’t have a choice of what devices their insurance is willing to cover. Openness builds trust between your customers and your products. Similarly, a responsible disclosure program can help facilitate third parties to review your work and provide constructive feedback.
Being closed about your products and processes means that you have an even greater responsibility to convey information and put processes and measures in place that will help convince yourself, and the public, that you are building safe and effective systems.
A vision of the end-game for the DIY community
One of my favorite parts of the conversation with the FDA was about do-it-yourself open source projects like Nightscout, Loop and OpenAPS. I have been very vocal about about my love and support for these projects – I even had the honor of sharing the importance of projects like these with President Obama. There is simply no doubt in my mind that by being on OpenAPS and Loop, my daughter and hundreds if not thousands like her, have been getting safer, more effective therapy for, wait for it, THE LAST TWO YEARS. That’s two years sooner than she could have gotten with any commercially available system. #WeAreNotWaiting.
Discussing DIY projects like Nightscout, OpenAPS and Loop with the FDA
At our meeting with the FDA, we asserted that however the Pre-Cert program evolves, not only does it need to keep learning and iterating, it also needs to scale. Not only should it work for large, well funded companies like Apple, Google and Samsung, it also needs to accommodate projects like Nightscout, Loop and OpenAPS.
Here’s how this could play out: First, imagine that an insulin pump company delivers an open protocol pump, perhaps catalyzed by JDRF’s Open-Protocol Automated Insulin Delivery System Initiative. Now imagine that a project like Loop or OpenAPS, or even a small company that bases their app on open source code, applies to the Pre-Cert program. Once that organization demonstrates its commitment to quality and organizational excellence, it can then deliver a pre-built app in the app stores. An app that you can easily download and use, that’s supported by great processes. An app that can improve and be updated in months, not years, with an expedited review process.
This pace of innovation can only occur when both do-it-yourselfers and industry can iterate quickly within the framework of a pragmatic regulatory process.
This is a world we really like. And it’s a world that the FDA wants to make happen.
What do you think about all this? What other metrics or KPIs should be considered? What else should we be thinking about as these conversations continue? Leave a comment and let us know!