FDA Pre-Cert Pilot Program: Public Input Workshop and Co-Creation

This is the next in our series of updates on Tidepool's participation on the FDA Pre-Certification Pilot Program. Prior popular posts include:

• A Revolution is Occurring: The FDA is leading it, and you may be missing out
• 101 Questions to Ask Yourself if you are building Medical Software

TL;DR

From January 29 through February 1, 2018, we had the pleasure of participating in the FDA's Precertification Pilot Program public meeting as well as getting together in person with all of the other Pilot Program participants. Here's a summary:

• This program has real momentum. The FDA is really doing this! They (we!) are dramatically reshaping how software will be evaluated as part of their regulatory process. You would not be alone if you are skeptical, but from our perspective, this is really happening, on an aggressive timeline, and will make a dramatic difference to our ability to innovate in ways that will benefit the public health.
• The two-day public workshop was a great opportunity for the public to hear the program's goals and to share their thoughts. Some attendees came hoping to be educated on exactly how the program would work, only to find that they were the "10th pilot participant," expected to take an active role in defining the program.
• Many folks in industry are still have "old school" thinking. Instead of sending software developers or technical leaders to help shape the program, they only sent regulatory professionals and lawyers. Set your software people free!
• The nine pilot participants stayed for an extra day and had a very productive session where we rolled up our sleeves and got busy. Many challenges and lots of work remain, but this group has embraced our role in helping to co-create this program and we committed to maintaining momentum.
• Small companies (like Tidepool and Pear Therapeutics) got a lot of attention. Everyone agrees that this program must scale and work for startups, and hopefully for DIY projects, if it's going to succeed in fostering innovation in digital healthcare.
• There's a misconception among some that the FDA may be lowering the regulatory bar or "racing to the bottom." They are not. Instead this is about changing how software, and more specifically the companies who make software, are evaluated to match modern standards of how great software is built. This will make it easier for organizations that already follow excellent software quality principles to innovate more quickly.

Day 1: "Is this for real?"

The public workshop was held on the NIH campus in Bethesda, MD, on January 30 and 31, 2018. There were over 400 in-person attendees, and lots more folks were watching online. Day one was all about bringing the public up to speed on the goals of the program and what has transpired so far. All nine pilot participants got to be on a morning panel discussion to share our perspective. Then the FDA team shared their perspective on the program and working with the pilot participants.

Getting the public up to speed on the goals and status of the program was no small task. As Pilot participants, we've had months to get our heads wrapped around this, and to share our thoughts directly with the FDA when we thought things weren't quite right.

In this public workshop, the FDA was attempting the squeeze the 2-3 month journey that we've had as Pilot participants into two short days. Not too surprisingly, there was some confusion. Some participants were expecting the program to have more "meat on the bones" and attended to understand how their work would change.

Others attendees were just plain skeptical. As an opinionated and vocal member of the Pilot program, I had multiple people come up to me during the workshop and ask "Is this for real?"

I know, it's hard to believe. But the FDA is really serious about this, and yes, it is really happening. FDA Commissioner Gottlieb is tweeting about it. CDRH director Jeff Shuren gave a very clear keynote about it. And Bakul Patel and his team are very, very serious. This. Is. Happening.

Bakul Patel discusses the goals of the program

The afternoon of the workshop included presentations from representatives of CMMI, EQFM, Baldridge and Balanced Scorecard, all established "organization maturity" evaluation models. No doubt, there's a ton that can be learned from existing models about how high quality organizations can be evaluated. But I'll be honest: these models typically make my eyes roll back in my head.

While there are certainly startups that use them, they tend to be filled with buzzwords, acronyms and processes that will scare away many entrepreneurs and small companies. We're more aligned with Lean Startup and Agile software development methods, frameworks that help projects iterate quickly.

As part of this FDA Pre-cert Pilot program, we're going to need to find the best of all of these models and build an easy-to-understand framework that allows for rapid iteration while still maintaining a very high bar for patient safety, efficacy, cybersecurity and clinical responsibility. Fortunately, the team at the FDA is extremely open to feedback like this, and making the program work for small and big companies alike was a common thread of conversation.

Day 2: Co-Creating with 400 of our Closest Friends

Day two of the workshop was all about brainstorming and co-creating with the other workshop attendees. Everyone was assigned to a team. There were 10 teams that varied in size from about 25-50.

We had three co-creating, brainstorming sessions with the goal of soliciting the public's input. Bakul kept referring to the public as the "10th Participant" in the process. I was co-moderator of Team 5, along with Catherine Bahr of the FDA. Attendees included pilot participants J&J, Pear Therapeutics, and Roche, as well as Abbott, Bayer, Dexcom, Lilly, Medtronic and many other companies both inside and outside of the diabetes industry.

Breakout 1, "Enablers and Drivers" took a little while to get into the groove as most attendees were still getting their heads wrapped around the goals of the program. But we made great progress in identifying key issues around processes that might enable (or inhibit) achieving the program's goals. We also took the opportunity to start using words that everyone could agree on. :)

Breakout 2, "Measuring Results," was quite lively. We started with the suggested topics as suggested by the FDA, and then shredded them and ended up with this:

There were a few key insights that resonated with everyone in the room:

• KPIs are valuable, but don't always tell the whole story.
• There are some things that every company should measure and report to the FDA, especially around defects, safety and efficacy, and there are some things where it's really not the FDA's business (e.g. retention, brand awareness) even if it is the company's business.
• There are primary indicators that the FDA should know, and secondary indicators where the FDA only needs to know that the organization is taking measurements, but they don't need to see their results.
• There are leading indicators (metrics that you can look at before you launch), and trailing indicators (metrics that you look at ongoing, after launch, sometimes called "post-market surveillance").
• Especially when it comes to software and devices, in this connected age, every company should be continually evaluating raw, post-market data and looking for signs of trouble, and taking action based on what they find.

By the time of Breakout 3, "Weighting and Evaluation," we were all trying to find concrete ways of expressing how the program would work and who should participate. This is where lots of conversation about size and maturity of company came in. On the one hand, some participants felt like a company should be well funded and have a proven track record of delivering before being allowed in the program and using up valuable government time and resources.

On the other hand, more entrepreneurial participants saw the program as an incredible opportunity to catalyze investment and innovation; we want startups to feel that if they are willing to apply to the PreCert program, that they have an opportunity to iterate quickly and deliver their innovative products to market faster than they might otherwise with "old school" regulatory processes. From the perspective of smaller startups (like Tidepool and Pear Therapeutics), it's this mentality that fuels innovative, quality, Silicon Valley-like thinking.

Day 3: Sleeves Rolled Up with Pilot Program Participants

I was very happy that the FDA held the public workshop and I look forward to many more. It's critical for this process to be open and transparent, and for the the public's feedback to be heard and incorporated at every step.

But making tangible progress with 400+ people is tricky, even with breakout sessions. I was very glad that the pilot program participants were asked to stay an extra day to spend quality in-person time with the other pilot participants. This was an incredibly productive day, and I left very confident in this group's ability to collaborate effectively and productively.

At the FDA campus, co-creating with the FDA team and other Pilot Participants

We started the day by all sharing our hopes and concerns about the program. Some themes immediately emerged:

• Some of the terminology is throwing people. Not everyone understands terms like "excellence principles" and it's hampering our ability to get down into the details.
• In parallel with developing the "dashboard", a framework for messaging the programs goals and benefits needs to be established.
• We need to create some concrete examples of an assessment dashboard, so that we have something to work with, and so that we can more easily convey how the program will work.

Brandon and I both joined subgroups focused building concrete examples of metrics evaluation dashboards. This was my favorite part of the week. We started by combining the dashboards that Apple and Tidepool had drafted, and then immediately started diving into the details. Folks from Roche, FitBit, Samsung, Pear Therapeutics and more were right there with us. Themes emerged. Details were riffed upon. Brainstorms were had. We were in "flow" and it felt good. Having just come from the public workshop meant that it was all fresh in our heads; we looked at everything through the public feedback lens that had emerged the two days prior.

Co-creating with the teams from Fitbit, Pear Therapeutics, Apple and more

The Pilot participants all agreed that we are going to capitalize on the momentum we generated at the workshop, and attempt to produce a first draft within about a month. Some of the participants from the SF Bay Area also agreed that we would convene again in person.

This revolution is really happening. The FDA is really asking for input. Don't let this opportunity go by. Feel free to comment here (we'll pass it on), or comment on the official docket. Helpful links below.

Cheers!

Howard

• FDA CDRH Digital Health Homepage
• Digital Health Software Precertification (Pre-Cert) Program
• Workshop Details, including Agenda and Presentations
• Federal Register Notice
• Send Email to the FDA Pre-Cert Team: Fdapre-certpilot@fda.hhs.gov