How the Modern Data Stack is Changing the Way Enterprises Use Data Experts

An opinion piece by the CEO of Rivery, a SaaS ELT platform for data professionals based in Israel and New York

Itamar Ben Hemo

October 12, 2022

5 Min Read

Business leaders are confronted with an unstoppable tide of data, and a lack of experts to make sense of it. In the decade from 2010 to 2020, capturing, creating and copying of data rose by 5,000% worldwide, reaching 59 trillion gigabytes, according to 2021 IDC research.

But data experts, who are often trained with degrees in fields such as machine learning and computer science, are in short supply worldwide, with almost half of businesses struggling to recruit workers with hard data skills, according to British Government statistics.

Thankfully, the way businesses deal with the data available to them is changing.

New job roles, new technologies and new organizational cultures around data offer new ways for business leaders to extract value from data. Crucially, business leaders can now derive insights from data without relying on a single overworked central data team outputting business insights into the form of reports and visualizations. Training internal talent in basic data skills will be key to this transformation.

What is a ‘data expert’ anyway?

There is a growing trend for the artificial divisions between data experts and business users to break down, with data experts becoming more business-minded and business users learning to ‘self-serve’ with data.

One aspect of this is the rise of roles such as ‘analytics engineer,’ which help to bridge the gap between IT and the data consumers within an organization.

Analytics engineers collaborate with the team to analyze the data, to ensure that the business can use the high-quality insights generated from their work. Together with wider teams, these engineers help to set up and activate a truly modern data stack.

Upgrading employees with data skills

Rather than relying solely on hiring qualified data experts, business leaders should aim to train their existing workers with data skills. This is powerful because they combine newly acquired data skills with their existing domain expertise to extract maximum value from the data.

Data literacy courses are already becoming common in many companies, and large organizations such as Bloomberg and Adobe are going further, with in-house digital academies dedicated to training workers in how to use data.

Low-code and no-code solutions mean that data is more accessible to workers in departments such as sales or marketing. These ‘data citizens’ will be able to extract value from data without waiting for a separate team of data experts or scientists to do it for them.

Technology plays its part

Democratizing access to data within your organization and unlocking the business value of data requires the right technological tools. Data management is an important tool to ensure data is delivered to the right team within your business, in a condition where it can be used, without the bottlenecks and delays that can come from relying on a central data team.

Data management deployments automate procedures into one framework, making it simpler for business users to extract value. Along with tools such as data quality management, data validation ensures that data meets the standards required by business users.

Perhaps even more important is Reverse ETL, which turns the normal job of data warehouses on their head to direct a stream of valuable data directly to the teams which need it most.

Reverse ETL reverses the traditional process by which data is loaded into a data warehouse, by first extracting it from a data warehouse and then loading it into your operational systems.

In Reverse ETL, the data is loaded from the data warehouse and then fed directly into business software such as ERP (Enterprise Resource Planning) or CRM (Customer Relationship Management). Sales or marketing teams have data delivered directly into the applications they use in their daily work, so there’s less training required to understand it.

For example, this can be used to deliver personalized offers based on purchase history or more precisely targeted marketing campaigns. It is key to breaking down the barriers between data and the data consumers within a company, and removing a burden from overworked specialist data teams.

Why data mesh matters

Along with these technological changes and job role evolutions around data, there is also a new organizational approach to how data works within companies; a data mesh.

A data mesh offers a decentralized and ‘self-serve’ approach to delivering data throughout an organization. Rather than relying on a centralized data team − where the warehouse is controlled by hyper-specialised experts − data is organized via shared protocols in order to serve the business users who need it most.

The significance of this is that it helps empower teams to access the correct data they need, right when they require it, via the distribution of data ownership across the organization.

Whilst the concept of data mesh isn’t necessarily new, the key to operationalizing this approach effectively is the introduction of a platform or universal interoperability layer that facilitates the connection of domains and associated data assets within it. Companies can then use a platform that will help them to connect all the dots and manage the entire operation, in order to fully operationalize the approach.

Additionally, simply being aware of the business potential of data is no longer enough.

Companies need to be alert to the concept of “data as a product,” as data meshes will be core to enabling the application of the product lifecycle to data deliverables. By applying product thinking to datasets, a data mesh approach will ensure that the discoverability, security and explorability of datasets are retained. Teams are then better prepared to swiftly derive the most important insights from their data.

Engaging with the modern data stack

A central data team can become a bottleneck if analysts and engineers across the business cannot access the data they need, when they need it. It is essential that teams can access data within the systems and processes that they’re already using, while providing users with the skills and tools they need to self-serve.

By introducing data operations such as Reverse ETL, companies can prevent unnecessary bottlenecks and inefficiencies within the modern data stack, to enable teams to act on data in real-time and make key decisions.

Only through effective engagement with the modern data stack will companies be able to effectively pave the way for a future in which every employee is a data citizen, and where a data mesh enables companies to truly realize the potential of data.

Get the newsletter
From automation advancements to policy announcements, stay ahead of the curve with the bi-weekly AI Business newsletter.