How to solve the 'hidden layers of data' problem at the heart of GDPR compliance
Editor's note: The following is a guest article from Matt Glickman, Vice President, Customer and Product Strategy at Snowflake Computing.
Data privacy has been one of the hottest topics of 2018, as the General Data Protection Regulation (GDPR) dramatically changed the rules on how businesses must collect, track, and store consumer data.
The conversation will no doubt continue into 2019, and with renewed vigor as focus shifts to passing similar legislation in the United States. This summer, California passed a bill that many have dubbed "GDPR lite," going into effect in 2020.
Industry has yet to see how the U.S. government will seek to govern consumer data privacy on the federal level, but one thing is clear: if they haven't already, U.S.-based organizations must put into place the infrastructure and processes to address data privacy control concerns.
From a technology perspective, the biggest challenges companies face is building the capabilities needed to meet the GDPR's "right to be forgotten" clause. That is, the ability to isolate data and eliminate specific data sets at a customer's request.
GDPR appears on its surface to be all about privacy, but what it's really about is control. Specifically, the ability to put control back in the hands of the consumer by forcing businesses to accommodate their requests to be "forgotten" and ensure that all personally identifiable information (PII) has been deleted. This type of request can get very complicated, very quickly.
At the center of this challenge lies the "hidden layers of data" problem.
Here's how it works: for a long time, when data needed to be shared from point A to point B (across departments, for example), it required making copies. These copies then spread out across an organization, forming data marts in numerous locations, oftentimes even beyond those known or categorized by the organization.
This is an operational nightmare, as organizations needing to track down data profiles to accommodate user requests for deletion and other regulatory obligations are left to scavenger hunt for the data they need.
Think about an example common to every company: data backups. Every system basically holds onto data forever in some way, shape or form. Not only is this an issue with GDPR — which requires that consumers be able to delete their data on a requested date — but the inevitable data leakage means that customer data gets copied and moved enough times that it no longer is completely trackable.
Here's how can industry fix this problem:
Consolidate data marts as much as possible
Finding and consolidating every data mart is not realistically attainable at the enterprise level, but going from 1,000 to 20, for example, would be good progress.
Every additional degree of consolidation businesses can achieve makes the problem exponentially easier to manage and takes them a step closer to the ultimate goal: creating a "single source of truth." That is, collapsing all data flows into one central location that can serve as the system of record for all data across the organization.
Share, don't send
Creating a single source of truth for data has long been considered the holy grail of data management. But can it actually be done? Absolutely. Most enterprises just don't realize it yet.
Today, advances in encryption and stateless computing are creating new ways to store and share data. Specifically, they make it possible to keep data in one location and securely grant access to it on demand.
In the past, the only practical way to share data was to make copies. Now, rather than sending copies of data, businesses can simply manage access.
In other words, these new technologies make it possible to create a system where everyone interacts with the exact same data set. By no longer having to replicate data, and therefore having it exist in fewer places, compliance with the "right to be forgotten" becomes much easier.
Leverage the cloud's elastic scale
The cloud plays a big role in making it all work. Specifically, it's the elastic nature of cloud that underpins the approach.
In traditional production environments, resources are constrained. On-premises installations have hard speed and storage limits, are inflexible to work with, and require significant resources to manage.
The traditional model disincentivizes centralized data processing and instead pushes the resource demands out to the edges, which leads to copies and data marts. Industry is, not surprisingly, seeing some of the unintended consequences of this model play out today.
But when businesses move to a cloud-native data model where users have access to shared data sets without the need to copy and transfer them, they can comply with requests for data at any time without interrupting the production workflow.
As a result, organizations no longer need to copy data so that others can process it in their own environment, and businesses always have the compute power they need to comply.
Apply the same levels of compliance globally
In many ways, runaway complexity is at the heart of the problem GDPR aims to remedy. This is a problem that is only getting worse as more businesses gather more data about more people.
To that end, enterprises retrofitting their data systems in a post-GDPR world are well-served by applying the same levels of data governance globally. Not doing so means having different rules for different areas and likely employing a patchwork approach to the systems used to manage data, resulting in more complexity and a need of more resources to manage everything. It also means more room for error and data leakage.
Today, GDPR violations can lead to fines of up to €20 million (approximately $23 million), or 4% of worldwide annual revenue, whichever is higher. European Union regulators are expected to announce the first round of fines by the end of the year.
Enterprises who choose to embrace the much-needed spring cleaning of their data and consolidate it into a single source of truth stand to gain a competitive advantage that will carry them well into the future.
Doing so will not only allow them to accommodate the new requirements and operate without fear of enforcement, but also transform their businesses.