Fair warning – this is an awesomely nerdy blog post. If you just want to learn what Uploader 2.0 is all about and install it, go here.

If you want a behind the scenes look at how and why we built it, this post is for you!

TL;DR

• We just launched Tidepool Uploader 2.0… hooray!
• We’ve removed the requirement to upload to CareLink. If you use a Medtronic 523, 723, or 530G pump (with or without Enlite sensor), you can now upload directly to Tidepool.
• If you’re using a Medtronic 630G, 640G, or 670G, sign up here to be a beta tester of our upcoming release for your pump.
• The Tidepool Uploader is no longer reliant on Chrome. It’s now an independent app for your Mac or PC. For the tech nerds reading this, it’s now an Electron app. (Read more below.)
• The new Tidepool Uploader will let you know when updates are available.

Interview with the Engineers behind Tidepool Uploader 2.0

I asked Gerrit Niezen and Chris McGee some rather basic questions, and asked them to be as detailed as they wanted.

Me: What exactly does the Tidepool Uploader do?

Gerrit & Chris: The Uploader does the heavy lifting getting the data off of compatible diabetes devices and transmitting the data to our servers where it’s securely stored. It requests data from devices using the USB port, and processes the data into a common format known as the Tidepool data model. Almost every device out there has its own unique way of storing and representing diabetes data, so we have to convert the data into a common representation to ensure that apps, like Tidepool, can display the data from various devices in the same way. The Uploader also takes the various timezones and time changes on the device into account – see this blog post for more detail.

How did our old Uploader work?

The first version of our Uploader was a Chrome App, not to be confused with a Chrome Extension – which you’d see next to the address bar. Chrome Apps have their own User Interface (UI), whereas a Chrome Extension merely augments the browser experience. More importantly, Chrome Apps get more access to the machine that the browser is running on than regular web apps, which was particularly advantageous for us because we were able to leverage Chrome’s native USB support for our software to work.

This USB support comes in the form of APIs for two different interfaces: Serial and Human Interface Device (HID). The diabetes devices that we want to communicate with use either of these two interfaces to communicate to computers. Because Chrome Apps have that support built in, we didn’t have to write the low level communication software to make sure we were compatible with Windows and macOS operating systems, we just had to make sure we worked with Google Chrome, and we were good to go on both Windows and MacOS.

This also created some limitations and restrictions around our software, but we felt the advantages of creating a Chrome App, most notably the built-in USB communication support, was a huge advantage to us developing our first Uploader. Additionally, Chrome Apps are relatively easy to distribute through the Chrome Store. So when we published an update, that new code was propagated out to our users seamlessly and automatically.

What prompted the change away from Chrome Apps?

There were two issues.

New Tidepool users sometimes struggled to install the app because getting a Chrome App was a new, and different, experience.

But adding fuel to the fire, Google announced they will no longer support Chrome Apps unless they are running on a Chromebook, running ChromeOS. This means we had to find a solution for the Tidepool Uploader that no longer relied on Google Chrome, the Chrome Store, and Chrome Apps.

Google announced this decision last year, you can read more on their blog – https://blog.chromium.org/2016/08/from-chrome-apps-to-web.html. While there was no firm stop date announced, we knew we had to get to work sooner rather than later to get a new solution developed, tested, verified, and ready for the diabetes community.

Along with the Chrome App, users had to install a Driver Package to successfully liberate their diabetes data. What is the Driver Package?

USB drivers, which are often times proprietary, makes it easier for software to communicate with these diabetes devices over the Serial or HID interfaces instead of over the raw USB interface. That driver is usually installed in Kernel space (technically, this term and location varies depending on operating system), where it has access to talk directly to hardware.

The thing about drivers, though, is that you need Administrative privileges to install them because drivers have access to the hardware on your computer. Drivers operate in a privileged execution space within your Operating System, so there needs to be an additional layer of security around their installation and maintenance.

But once that installation is authorized by something like an Administrator password, the Operating System largely stays out of the way of a new driver. All of this means you have to trust the source of these types of files. Chipset manufacturers, that build the USB microchips used by the device manufacturers in their diabetes devices, digitally sign their USB drivers. We also digitally sign the Tidepool Installer used to install the USB drivers. Unsigned software is untrusted, and can be misused to damage a machine.

We’ve gone to great lengths to ensure that we could have a single, installable driver package that would cover support for all Tidepool-compatible devices like Animas, Dexcom, Omnipod, Medtronic, and others. Being able to redistribute these drivers in a single package makes things much easier for us. Rather than requiring users to go to a handful of different locations or websites to download specific drivers for their different diabetes devices, we bundle all the drivers into a single file – if we support a device, its driver is included in our Driver Package for the Tidepool Uploader.

Connecting your insulin pump to your computer with a USB cable is meaningless to the Tidepool Uploader without the drivers to facilitate data being transferred. Without the Driver Package, the communication to your diabetes devices does not exist and none of this is possible.

Why were the Chrome App and Driver Package separate?

The short and sweet answer is, we couldn’t put drivers into a Chrome App. That’s a development restriction put in place, in part, by Google. Chrome Apps are installed into User Space, which created specific limitations on how far our access as engineers went. Chrome Apps have access only as far as Google Chrome permits.

By requiring our users to install the separate Driver Package, we were able to communicate with the devices from inside of the Chrome App and you could access your diabetes data.

Which brings us to the new Tidepool Uploader. What makes it “new”?

In order to replicate the user experience of the Chrome App, we pieced together components of the Tidepool Uploader through a framework called Electron. These components – Chromium, Node.js, and V8 – form the foundation of the Tidepool Uploader.

Chromium is the open source project that serves as the foundation for Google Chrome. For all intents and purposes, Chromium is a web browser, but it lacks a few of the additional features that Google Chrome has like account syncing and USB communications. (Remember how USB communications is a specific feature available to Google Chrome and Chrome Apps? That’s not available on Chromium.)

We rely on Chromium for the visual experience of the Uploader. Everything you see and click on is handled by Chromium. But without the USB APIs that were available to Chrome Apps included in Google Chrome, an additional component is necessary to allow the Tidepool Uploader to communicate with all of the devices we support. This brings us to node.js.

Node.js is an operating system-level execution environment for JavaScript. Through Node.js, we are able to do a lot of the things behind the scenes that users do not see that we were previously doing with the Chrome App – most importantly, talk to USB devices.

Finally, Electron includes V8, the JavaScript execution engine that powers all of our code.

All of this means the Uploader can do all of the things it was doing before when it was just a Chrome App, but now we are no longer held back by restrictions placed on Google Chrome. While you still have to use Google Chrome to view your data, the act of uploading your diabetes data happens independent of Google Chrome.

Specifically, how does node.js factor into the new uploader?

Node.js forms part of the JavaScript execution and allows us to replicate the driver capabilities that we used to have in our previous Uploader through Google Chrome. Now that we are using the Electron framework for the new Tidepool Uploader, Chromium is able to handle the visual components of the Uploader, but we use node.js libraries to communicate with diabetes devices over USB.

The key thing about node.js is that it enables JavaScript to be run outside a web browser, whereas most web browsers already run JavaScript on their own. Additionally, Node.js actually runs on the same engine that Google Chrome and Chromium use. Which, if you sort all this out, means that what we’ve built to work in Google Chrome (which runs on V8, and is built upon Chromium), can be replicated without the Google Chrome experience by relying on Node.js.

What does this mean moving forward?

Now, we have full control over the installation and setup process for our software. For our users, there is only one file to download and install. The Driver Package, which was previously a separate file, is now bundled with the rest of the Uploader. Part of the Uploader installation process includes running the driver installer. So when we add a new compatible devices to the Tidepool Uploader, users will get a notification for an update and can install the latest version like they would any other software update.

For us, making updates will be much more streamlined. With the Chrome App Store, we had to follow very specific protocols for updating the app. Now, we have more control, leveraging GitHub’s Releases functionality to indicate and propagate updates. The Tidepool Uploader will check for new updates once a day, and every time you launch the app, so there shouldn’t be a risk of missing out on new features or functionality.

Now we are able to create an application that one would think would normally run on a browser, but we have much more intimate hardware communication potential through a stand-alone application, which is what we’ve built with Electron. While things are otherwise unchanged for our users, all of the code we are writing is leveraging more powerful APIs through Node.js than we previously had access to in creating a Chrome App.

The Tidepool Uploader has the same functionality, and when we look at the big picture, it operates the same behind the scenes, but getting this this point took a lot of work – what has changed?

There are a lot of little things we have access to now that we’re building a stand-alone app with Electron. And, there were a lot of things we did with our Chrome App that didn’t make a lot of sense from a “normal web application” development standpoint. A lot of our design and engineering decisions were because we were bound by the Google Chrome and Chrome App environment.

For instance, having an actual page URL is something you don’t have with a Chrome App, but we have that opportunity now in an Electron environment. We aren’t surfacing the URL to users, but when it comes to debugging and troubleshooting, leveraging a URL with parameters being passed from screen to screen will be tremendously helpful if we have a bug to replicate and eventually squash.

URLs mean we have proper navigation through the Uploader experience, this means we can be more precise with updates to the User Experience, which means adding a new page or splash screen is a much easier process. While moving to a true page system will help us a lot in the future, it did represent a fair amount of work separating the components to their proper home in the new codebase.

Because Google Chrome and Electron are using the same rendering engine, Chromium, we didn’t have to change too much to ensure the same visual experience for Tidepool users. But a lot of the outer framework of the Uploader did need to be modified quite a bit. Now that we aren’t held back by size restrictions for Chrome Apps, we can play with space much more liberally as development on the Uploader continues. Creating a more responsive application opens up more opportunities to improve and expand what the Uploader, and the broader Tidepool platform can accomplish within a single application.

Included with the new Tidepool Uploader is a feature for direct Medtronic uploading. What was the motivation behind developing this feature?

Feedback from Tidepool users and the broader diabetes community was the driving force behind our improved support for Medtronic insulin pumps. After we launched support for Paradigm 523 and 723, and MiniMed 530G, we heard the same key message, “I’m glad Tidepool works with my insulin pump, but I wish the data upload process was easier.”

As thrilled as we were to debut our initial support for Medtronic devices, user feedback pushed us to do better. And we knew better had to be a priority for us if we wanted to effectively address the needs of the diabetes community members using Medtronic pumps.

How did Tidepool previously support people with diabetes wearing Medtronic pumps?

Initially, we supported Medtronic’s Paradigm 523 and 723, 530G and Enlite sensor by requiring users upload their data to Carelink. We were then able to download their data from Carelink, upload it to our secure servers, and convert that data into our data format.

What is Medtronic direct uploading?

Direct Medtronic uploading is the next phase of our growing support for insulin pump compatibility with Tidepool. The Tidepool Uploader now communicates directly with Medtronic’s Paradigm 523, 723, and 530G pumps to upload data directly from the pump to Tidepool.

In order to utilize Medtronic Direct, users just need a Bayer Contour Next Link meter.

How can users wearing a Medtronic Paradigm 523, 723, or 530G upload directly to Tidepool?

If you’re creating a Tidepool account for the first time, be sure to select “Medtronic Direct from Bayer Contour Next Link” as one of your devices when you are configuring the Tidepool Uploader.

If you already have a Tidepool account, when you log into the new Tidepool Uploader, click the arrow next to your name, select “Choose Devices”, uncheck “Medtronic (from CareLink)”, and select “Medtronic Direct from Bayer Contour Next Link”.

The Tidepool Uploader will ask for a 6-digit serial number on the back of your pump. After you’ve entered the serial number, and connected your Bayer Contour Next Link meter to your computer, you can upload your Medtronic data directly to Tidepool.

Why is a Bayer Contour Next Link meter required?

Bayer Contour Next Link meters wirelessly communicate with compatible Medtronic insulin pumps, which is a fantastic feature on its own. Also important for us is the fact that the Contour Next Link meters are compatible with USB 2.0 and 3.0. Many folks were having trouble with the grey CareLink stick because it is not compatible with USB 3.0 ports found on most modern computers. Most importantly, Bayer publicly published the protocol specification for their Contour Next Link meters on their website.

What’s next for Tidepool?

We’re constantly evaluating feedback from the Tidepool community to prioritize what updates and features we should be focusing on next. With the release of direct Medtronic support in the Tidepool Uploader, we’re getting ready to launch a public beta for Medtronic 6-series insulin pumps. Anyone interested in participating in our Medtronic 6-series beta should fill out this form.

Is all of this work we’ve done available to the public?

Absolutely! As always, part of our mission as a nonprofit is to share everything we learn and do with the community. All of Tidepool’s source code is available at https://github.com/tidepool-org

—

There are a lot of these tech-based conversations that happen internally at Tidepool that I’d love to share with you all more regularly. If you have specific questions about the “how” and “why” of Tidepool, please let me know and I’ll ask around.

Christopher Snider – christopher@tidepool.org