Creating Vial Connect - An ETL platform to manage millions of EMR/EHR healthcare records

Vial hero image - Woman staring at computer screen



Google Cloud



Frontend Development


Interested in learning more?

Send us a message 👋

About Vial

Vial is a Contract Research Organization (CRO) that delivers faster, better, cheaper trial results across Dermatology, Ophthalmology, GI, Oncology, Neurology, and Cardiology. They connect with healthcare data providers to find patients that would benefit the most from taking part in current clinical trials.

As a result of needing to work with large amounts of data from Electronic Health Record (EHR) APIs, CSVs, XML files, and more, Vial approached Lazer to help design, architect, and build a greenfield solution to this. We ended up creating a fully-fledged ETL platform, powered by the Google Cloud Platform (GCP), that would allow them to ingest millions of healthcare records, transform that data into different meaningful structures, and load it into Vial’s databases and systems, in a HIPAA compliant manner. All so that medical professionals could easily find and filter patients that would be most suitable for recruitment into clinical trials.

Our Approach

Circular icon of the number one with a black background

Understanding the vision behind Vial Connect

One of Vial’s core goals is to help clinics find more patients for trials. Typically the best way to do that is by accessing Electronic Health/Medical Records (EHRs/EMRs), searching and filtering patients by characteristics, and then reaching out to them for recruitment purposes. Thus we knew we had to create an ETL platform to build a frontend solution for this.

Multiple tablet screens displaying Vial's app
Multiple tablet screens displaying Vial's app
Circular icon of the number two with a black background

Planning for data differences and mapping EHR records to Vial’s ideal format

One of the first EHR systems we engaged with was ModMed. However, very quickly we learned there was differences in the terminology. For example, ModMed referred to allergies and problems as intolerances and conditions in their systems. So we need to create a plan and mapping strategy that would take all the terminology differences from EHRs and relate them to the terminology we needed.

Tablet view of Vial's screening form
Circular icon of the number three with a black background

Understanding HIPAA compliance needs and planning for it

When dealing with any patient data, we needed to ensure we were dealing with it in a compliant manner. Before architecting our ideal solution, we took core pieces we needed such as authentication, database access, access control, encryption, audit logs, and more, and thought through what was needed to be HIPAA compliant. By leveraging Google Cloud, we could ensure that we were creating a strategy that would allow us to be fully compliant.

Tablet view of Vial's Investigator's Global Assessment on a tablet screen
Circular icon of the number four with a black background

Passing technical demos and certifications with EHR systems

Oftentimes, in order to get access to EHR APIs you need to pass a technical evaluation with them to showcase that you have a thorough understanding of their systems. With ModMed, we successfully passed their evaluation and ensure that ModMed could audit our work.

Tablet view of Vial's metrics dashboard
Circular icon of the number five with a black background

Establishing a complete architecture with Google Cloud

In order to create a scalable, reliable ETL platform, we leveraged Google Cloud extensively. Google Cloud services we used included Cloud SQL for PostgreSQL, Kubernetes Engine, Cloud Logging, Identity and Access Management, Cloud Storage, BigQuery, Looker for GCP, and Firebase. Outside of GCP, we used Github Actions for our test suite and deployment.

Image of Google Cloud Platform's logo
Circular icon of the number six with a black background

A caching strategy to ingesting hundreds of thousands of medical data

Once our architecture was created, we now needed to create the pipeline to ingest, transform, and load the hundreds of thousands of medical data we needed to deal with. Very quickly, we learned that rate-limiting would be an issue, so we developed a caching strategy at every step of the ingestion process.

Image of an App Architecture diagram
Circular icon of the number seven with a black background

Scaling to millions of medical data

Creating a reliable ETL pipeline was our first goal. After, we performed optimizations to ensure that it could support millions of medical records and more EHR systems in the future.

Image of 2 iphones with Vial's app on the screen
Circular icon of the number eight with a black background

A central dashboard using Retool

To create a quick and optimal frontend to access the data, we leveraged Retool to query the appropriate databases and filter through data.

A tablet and iphone view of Vial's medication tracker sceeen
Vial logo

The Result

Working with the Vial team was incredible, and working on a project with such a great mission was the cherry on top. At the end of the engagement, we successfully created and launched Vial Connect. With it came search and filtering, integration with 2 ModMed clinics, the integration with EHRs, CSV downloads, storage of millions of patient records, and a successful plan for the future. We’re excited to see Vial continue to grow!