During a time of competitive markets and economic headwinds, organizations are looking to add more agility, efficiency and intelligence to their internal decision-making and processes. In fact, according to a new IDC report, by 2023, 60 percent of enterprise intelligence initiatives at European enterprises will focus on shortening data-to-decisions time by 30 percent and driving higher agility and resilience.

However, only 20 percent of IT and data leaders say they've achieved better data insights from their IT modernization and cloud migration strategy. While organizations have embraced cloud data warehouses and data lake platforms to scale analytics and improve their competitive edge, getting maximum value from these platforms requires them to modernize the end-to-end data lifecycle. 

IDC believes that a trusted and automated data movement platform that securely delivers data into the cloud data warehouse helps to eliminate one of the biggest barriers to accelerating data strategies.

IDC Spotlight: The value of automated data pipelines and cloud analytics

DOWNLOAD

The hidden cost of not automating your data pipelines  

The top challenges holding European organizations back include data quality, data silos and legacy infrastructure. The more significant challenges are a lack of trust in insights, a data and analytics skills shortage and complexities in synthesizing multiple sources of information for effective insights.

Processes that involve manually building and maintaining data pipelines from various sources across the enterprise result in delays, inaccuracies and constant maintenance. Enterprises surveyed by IDC say they spend 12-18 months developing these capabilities and data access controls. Another study by Wakefield Research shows the average data engineer spends 44 percent of their time maintaining data pipelines – costing companies an average of $520,000 per year. 

The value of automated data movement to accelerate business insights 

Modern data teams face daunting challenges when it comes to data integration. An ever-increasing number of data sources, ever-changing schemas, the need for data quality and multiple destinations complicate development teams' ability to keep up with demand and can cause an increase in data latency.

The answer to these problems is automated ELT, the modern successor to traditional ETL (extract, transform, load). Automated ELT provides nearly instant, self-service access to data and allows data development teams to allocate engineering resources to important business-supporting activities. 

An automated data movement platform uses the ELT methodology alongside 200+ prebuilt connectors that automatically adapt as schemas and APIs change to ensure consistent and reliable access to data. This enables organizations to unify and continually synchronize data from disparate sources, including databases, applications, files and events. All of which leads to faster, more accurate data insights.

Given that 54 percent of European cloud users use Google Cloud, according to IDC's 2021 Multicloud Survey, and 24 percent cite "extensive usage," integration with Google’s BigQuery should be a requirement of any automated data platform selected in order to bridge the gap between disparate data sources and Google analytics solutions. 

Finally, to maintain a strong security posture, an enterprise-grade, end-to-end, fully managed and secure data cloud integration platform with the highest level of protection for sensitive data is required. The platform should include end-to-end data encryption and private networking capabilities to mitigate the risk of data traversing the public internet and data residency options. The option for organizations to select their cloud service provider and cloud geography as well as the specific cloud region will further ensure regional compliance and data security requirements are met.  

Google Cloud and Fivetran: Open cloud meets automated data movement

DOWNLOAD

Real-time data movement is key to GroupM’s client reporting  

Building a modern data stack that combines Google Cloud and Fivetran allows organizations to start small and scale fast while achieving greater agility and efficiency. For global media agency Group M, it was able to accelerate data ingestion from 15 business application sources using Fivetran as its automated data movement solution.

Previously, GroupM's data team pulled marketing data directly into Google Sheets. Pipelines would occasionally fail, caused by issues that were hard to detect, and formatting problems were prevalent in the spreadsheets. Preparing data for analysis in Google BigQuery, GroupM’s data warehouse was proving to be labor-intensive work – at a time when clients wanted faster access to more business insights.

Fivetran’s fully-managed, prebuilt data connectors helped GroupM pull data from a wide variety of sources, including Facebook and Google Ads, directly into Google BigQuery. Its self-healing and zero maintenance, managed service architecture along with fast access to pre-configured connectors sped up ingestion from up to 15 main sources. Fivetran also backfilled its customers' historical data, saving the team further time and avoiding time-consuming manual syncs – ultimately saving the firm up to 75 hours a month, or nearly two weeks of work, to provide marketing analytics reports to its clients. 

“Our responsibility is onboarding data that gives value back to our major clients by helping them decide when and where to spend their advertising budget. Fivetran helps us to do that…Instead of spending our time maintaining pipelines and collecting data, we are actually using the data,” says Herman Mull, Data Analyst, at GroupM.

Automated data movement should be a core competency of any business 

The current economic climate demands that organizations find ways to do more with less. Data teams need to be able to spend less time maintaining pipelines and more time extracting value from the data.

A trusted and automated secure data movement platform like Fivetran allows organizations to get the most from their data – and their cloud investment.

IDC Spotlight: The value of automated data pipelines and cloud analytics

DOWNLOAD