This is the next in our series of blog posts to keep you up to date on our ongoing involvement with the FDA Pre-Certification Pilot program. In the first post, we shared some of the “Excellence Principles” and “Common Validating Principles” that the FDA asked us to think about. In our second post, we shared how we think about building software in 101 Questions to Ask Yourself if you are building Medical Software. In this post, we talk about our most recent meeting with the FDA and examine the revolution that is under way. –HL
The FDA is revolutionizing how digital health products get to market. These changes will be a great benefit to public health and will cause digital health solutions to reach the public as quickly as possible while maintaining safety and efficacy.
These FDA changes have leapfrogged over much of the diabetes industry. It’s time for the industry to do more, faster. Imagine that.
Every company in the diabetes industry needs to seriously examine how they think about and interact with the FDA and public when it comes to digital health. Here are a few suggestions:
If we all get this right, it is going to have incredible benefits for public health, not just in the diabetes community, but across the board:
So let's go, diabetes industry and community: don't stand idle. Get involved! Make some bold changes. Help this revolution succeed.
It will be good for your business and, more importantly, it will be good for public health.
Let's start with an astonishing statement: When it comes to digital health, the FDA has leapfrogged industry. The FDA is now pulling industry to go faster, but industry has not yet responded. Many companies are still being guided by old-school regulatory groups that believe they need to comply with old-school regulatory guidance, much of which was invented before the Internet even existed. This overly conservative approach is causing digital health products to take way too long to get into the public's hands, and this delay is a detriment to public health and corporate bottom line.
On the bright side, the FDA gets this, and is actively trying to fix it. They understand, and openly admit, that prior attempts to regulate software and other digital health projects have backfired. Industry adherence to slow-moving, old-school regulations and guidance documents has had terrible unintended consequences that have been detrimental to public health:
Without intending to do so, the agency tasked with protecting the public health and safety has hurt it by previously enforcing regulations that simply could not keep up with the pace of technical innovation. These are unintended consequences, but consequences nonetheless.
Fortunately, things at the FDA are now changing.
On December 20-21, 2017, Tidepool had the honor of spending two days, in person, with Bakul Patel and the team from the FDA that is driving the Precertification Pilot Program. The FDA has done site visits with all of the pre-cert pilot participants, and is now preparing for a public meeting.
We covered a lot of ground in this meeting:
We talked a lot at our meeting with the FDA about the use of metrics and KPIs as a method for validating that a company has a "culture of quality and organization excellence."
In short, the FDA's vision is that there will be a library of Culture of Quality and Organizational Excellence (CQOE) questions and KPIs, and that each company can in essence say, "here's the dashboard of KPIs that we look at to make sure we are shipping safe, effective products." Appropriately, they think about these KPIs along five dimensions:
I will admit that I went in mildly skeptical about a KPI-driven approach. As I wrote last month, the way Tidepool thinks about building high-quality software is a little different than the draft questions the FDA proposed. I still have some concerns that no approach can be fully quantitative, as there’s the chance it will become overly rigid, causing unintended consequences like before, or that the KPIs will become easy to game, and will not meaningfully benefit public health.
That said, the conversation we had with the FDA around metrics and KPIs exposed that there are indeed some interesting ways to have quantitative measures. The FDA team was inquisitive and open to our approach to building software, and also pushed us think about how some of these measures could be turned into KPIs that we would use on a regular basis as good indicators of safety and quality.
Here's an example of how a KPI could be used effectively, based on one of the "deep dives" that Tidepool did with the FDA team:
It's critically important for companies delivering software and medical devices to understand and mitigate the risks involved with a product. Here's the evaluation process Tidepool uses. It's similar to the risk analysis template that a lot of companies use and is derived from ISO 14971. (Sidenote: Hey AAMI, ISO and lots of others: important standards docs like this should be free and public if you want them to be used and followed!).
In Tidepool's system, like many others, risk is analyzed along the dimensions of severity and probability. Anything that is green in the chart above is OK to ship with. Anything that is red is definitely not OK to ship, and we should drop everything to fix the issue. Yellow risks need to be mitigated with a plan in place to fix.
You may have noticed risk level 11, which is the possibility of shipping with a known Catastrophic issue (one that could potentially kill someone), even if it’s highly Improbable. I wanted to address this, as this is something that folks may wonder about. Imagine this hypothetical scenario: You are delivering a closed loop app that has a risk of a lethal overdose to 1 in 2 million people. But by not delivering the system at all, those 2 million people will stay on self-administered open loop therapy, which has a 1 in say 200,000 chance of delivering a lethal overdose. In that case, it would be better to deliver the system since it would save lives.
Capturing the hypothetical risk of a bug, however, only tells half of the story. What about the issues that your customers report, perhaps by social media, or the hazards that actually occur after you ship? (In this context, "hazard" is a "risk" that actually happened to users. It's real - not hypothetical).
It's important for a company to take seriously actual customer reports of issues that occur, and demonstrate that they aren't affecting product quality or safety in a way that they didn't anticipate, and that the company isn't seeing repeat issues that should have been previously mitigated or fixed.
What KPIs might you measure to see how you are doing in this regard? It might look like this:
(Apologies for the handwriting, but we want you to see the real work.)
In other words, a good KPI to measure your follow-through might be, for each risk level:
Risks Mitigated / Risks Identified
If this ratio approaches 1, you are demonstrating good follow-through.
Another KPI might be:
Number of Level 12-25 Hazards Reported / Level 12-25 Identified Risks Mitigated
You certainly would want this KPI to approach zero.
Finally, you could imagine a third KPI being:
Recurrent Level 12-25 Bugs
You also would want to be a very small number. A large number would indicate that you aren't fixing bad issues that come up, or that you don’t have the right automated tests in place to ensure that once fixed, they don’t come back later.
We had a wonderful conversation about openness, transparency and reproducibility.
In short, we believe that when a company shares more about what they do, they can build safer systems. Being open allows others to inspect what you do and how you do it. In the best case scenario, others can build and reproduce what you have done, and alert you to safety or cybersecurity issues. Karen Sandler has written and spoken at length about this topic with respect to cardiac devices.
That said, it's unreasonable to assume that all companies will be open and transparent. Some companies simply consider their product development processes and details to be top secret, and we are OK with that. But companies need to understand that within healthcare, in some cases you’re asking patients to trust your products with their lives. And often patients don’t have a choice of what devices their insurance is willing to cover. Openness builds trust between your customers and your products. Similarly, a responsible disclosure program can help facilitate third parties to review your work and provide constructive feedback.
Being closed about your products and processes means that you have an even greater responsibility to convey information and put processes and measures in place that will help convince yourself, and the public, that you are building safe and effective systems.
One of my favorite parts of the conversation with the FDA was about do-it-yourself open source projects like Nightscout, Loop and OpenAPS. I have been very vocal about about my love and support for these projects - I even had the honor of sharing the importance of projects like these with President Obama. There is simply no doubt in my mind that by being on OpenAPS and Loop, my daughter and hundreds if not thousands like her, have been getting safer, more effective therapy for, wait for it, THE LAST TWO YEARS. That's two years sooner than she could have gotten with any commercially available system. #WeAreNotWaiting.
Discussing DIY projects like Nightscout, OpenAPS and Loop with the FDA
At our meeting with the FDA, we asserted that however the Pre-Cert program evolves, not only does it need to keep learning and iterating, it also needs to scale. Not only should it work for large, well funded companies like Apple, Google and Samsung, it also needs to accommodate projects like Nightscout, Loop and OpenAPS.
Here's how this could play out: First, imagine that an insulin pump company delivers an open protocol pump, perhaps catalyzed by JDRF's Open-Protocol Automated Insulin Delivery System Initiative. Now imagine that a project like Loop or OpenAPS, or even a small company that bases their app on open source code, applies to the Pre-Cert program. Once that organization demonstrates its commitment to quality and organizational excellence, it can then deliver a pre-built app in the app stores. An app that you can easily download and use, that’s supported by great processes. An app that can improve and be updated in months, not years, with an expedited review process.
This pace of innovation can only occur when both do-it-yourselfers and industry can iterate quickly within the framework of a pragmatic regulatory process.
This is a world we really like. And it's a world that the FDA wants to make happen.
What do you think about all this? What other metrics or KPIs should be considered? What else should we be thinking about as these conversations continue? Leave a comment and let us know!
Cheers,Howard
We'll keep you updated with all the need to know news and announcements for Tidepool and Tidepool Loop
Donate today to support Tidepool's mission to improve the lives of People with Diabetes through affordable, accessible and interoperable technologies
Want to get the latest news and updates about Tidepool and Tidepool Loop? We promise to keep it interesting in your inbox.
