Big data is changing the way every industry operates, and healthcare is no exception. The use of big data in healthcare stands to affect everything from healthcare costs to the quality and accuracy of care.

It's one of the reasons why careers in big data are booming. According to LinkedIn's 2020 Emerging Jobs Report, the data science field is continuing to grow, and there are even signs that data science roles are replacing or augmenting roles traditionally filled by statisticians.

Learn more about how big data stands to solve some of the industry's biggest challenges from two Abbott data engineers, as well as what they see as the critical skills for success as a data engineer.

What Is Big Data?

Big data refers to the massive amounts of information — think petabytes and exabytes — that an organization creates or consumes every second of the day. What's critical isn't necessarily the amount of data but rather its organization. You can find valuable information within the chaos, such as patterns or trends that may identify a new business opportunity or change the way you're delivering a service.

That's where data engineers come in. Data engineers are tasked with organizing data and making it usable for other groups across an organization. That often means creating the systems and processes behind gathering data and making it easy to work with.

"I see big data pairing nicely with advanced analytics," explains Devin Stompanato, an Abbott data engineer, pictured below. "It's not just big data, it's big data and advanced analytics. Big data means you have a large set of data. Advanced analytics is what you do with that large set of data."

Sources of Big Data in Healthcare

Stompanato and his colleague Sam Jarvis, also a data engineer, both work on the Big Data and Advanced Analytics (BDAA) team at Abbott. The two work mostly with Abbott internal data, including data about and from the Architect and Alinity diagnostic machines. If you have your blood drawn at the doctor's office, the sample may be run through one of these machines to get a result. Then, certain data may be sent back to Abbott in order to improve the machines or the patient experience.

But beyond what they work with on a day-to-day basis, Jarvis, pictured below, notes that there are "innumerable" sources of data in healthcare.

"Sources could really be anything from a patient's clinical experience to medical billing coding, and specific assay outcomes," Jarvis said.

The team is constantly receiving data from around the globe, making it crucial for them to process and clean raw log files so the data is in a storable and usable format. Some data sets contain duplicate records that Jarvis and Stompanato need to ensure are eliminated. They may also add error checking to filter invalid records, as well as parse data, dividing it into more manageable chunks.

This work allows data scientists and others within the organization to ask questions about the data set without having to clean the data themselves. The time savings is significant: The information that used to take days for data scientists to uncover now takes minutes or seconds.

Using Data to Solve Healthcare's Biggest Challenges

Jarvis and Stompanato have seen first-hand how big data is affecting the healthcare industry.

"Big data can reveal patterns and trends, especially relating to human behavior and interactions," Jarvis said.

But another important component of big data in healthcare is that it lets data scientists create more advanced and powerful predictive algorithms and models. With more data comes the ability to spot more patterns and trends that can more accurately predict events, such as machine failure, before it happens.

Stompanato notes how big data allows data scientists to do things they haven't been able to do before, like running algorithms on much larger data sets.

"Now that we are providing a higher volume of data and better tools to query it, business users can ask questions they haven't been able to ask before," he said.

Stompanato compares it to asking for feedback from three people versus 300 million people. Having a large sample size makes an answer much more meaningful.

"Big data has liberated us from those limitations," he said.

Jarvis agrees, saying that much of the data they work with feeds into algorithms that pertain to predictive failure. Because of the predictive element, an engineer can fix an issue before it has even been flagged. This proactive approach helps increase efficiencies and improve the quality of Abbott's machines.

Jarvis also points out Abbott's emphasis on ensuring data integrity.

"Collectively, the world has more and more data points," she said. "Correctly managing those [is crucial], as that can have a compounding impact on insights."

Working as an Engineer Within Big Data in Healthcare

Working as a data engineer isn't something Stompanato anticipated when he was in college. He learned about big data, but only considered pursuing a career in it when he experienced it first-hand. He started as a business system analyst intern at Abbott before being accepted into Abbott's IT Professional Development Program. The program allowed him to try out four jobs, each for six months, including a position in the Netherlands as a systems analyst and engineer, as well as a position on the BDAA team as a data engineer.

Stompanato and Jarvis agree it's critical to remain open-minded and pursue all your interests. Jarvis also says it's crucial for data engineers to not be afraid to fail.

"Fail fast and iterate," she says. "Learn from your failure and look forward. This idea can be applied to your work methods and your own exploration of big data. If you didn't like what you first found in this industry, try a different resource, reach out to a new person."

Both noted the importance of communication. As Jarvis says, you may be talking to a data scientist or leadership team about big data, and those two groups see data in different ways.

"You need to speak the same language," she says.

Being able to speak to both groups allows data engineers to translate company initiatives and strategy to data scientists to accomplish a specific goal. It also means they can explain to leadership the design decisions data scientists make and how those decisions meet the intended company objective.

And though data engineering may be seen as a very objective career, Stompanato calls out the need for creativity.

"I now need to solve a problem at the scale of terabytes or hundreds of terabytes of data," he said. "I can't even download that on my computer, let alone look through it. That would take me hundreds of lifetimes. So, I need to be creative in how I solve problems at scale."

"Big data really touches everything now," Jarvis said. "Every industry vertical and job component is impacted by big data and the insights it can drive. Find your interest, lean into it and you likely will find success."