The concept of a data pipeline is not a new idea, and many businesses have used them (whether they know it or not) for decades, albeit in different forms than we see today. But as we continue to see the exponential growth of business data year over year, data pipelines are becoming more imperative for businesses to have.
In the world of data analytics and business analysis, data pipelines are a necessity, but they also have a number of benefits and uses outside of business intelligence, as well. Today we are going to discuss data pipeline benefits, what a data pipeline entails, and provide a high-level technical overview of a data pipeline’s key components.
Table of Contents
A little bit about who we are...
ReconInsight is a business analytics software and services company. Our mission is to provide the software, services, and education you need to achieve success in your business analytics program. If you have time and want to read more about who we are, check out this letter from our founders and our code of excellence.
Disclaimer: While the majority of this blog post is focused on general education around data pipelines, we do mention our business analytics software, Ri360, from time to time. Just want to let you know up front.
How will my business benefit from a data pipeline?
Any business can benefit when implementing a data pipeline. For starters, every business already has the first pieces of any data pipeline: business systems that assist with the management and execution of business operations. Such as a CRM, Customer Service Portal, e-commerce store, email marketing, accounting software, etc.
In this post, we will highlight a data pipeline’s functional benefits as a technology. This means we won’t be spending time talking about things like cost savings, increased sales, or operational efficiencies. Data pipelines can help businesses achieve all of those things, but to do so requires human change in addition to the technology. Today we are putting a spotlight on how awesome data pipelines are all on their own.
Functional data pipeline benefits:
Business Autonomy
All business data, organized and normalized in one unified location streamlines everything.
Business Analysis & BI
A robust data pipeline creates strategic freedom and is an important piece to successful business analytics.
Productivity
Save everyone at your business time as they go about their day-to-day responsibilities. No more double-checking information.
Data Management
Automatic and consistent. Know your data.
Data Security
Data pipelines provide peace of mind that your data is stored securely.
Data Compliance
Access to all of your customer data is the first step for compliance with privacy regulations.
What is a data pipeline?
Data pipelines are created using one or more software technologies to automate the unification, management and visualization of your structured business data, usually for strategic purposes.
What affects the complexity of your data pipeline?
Number of different data sources (business systems)
Types of data sources, whether they are complex or simple
Type of connectivity to the data sources
Volume of data
Velocity of data
We have found that business size or industry do not directly impact the complexity of a data pipeline, it all depends on how you run your business and what technologies you are currently using.
What software should you use to create a data pipeline?
Data pipelines can consist of a myriad of different technologies, but there are some core functions you will want to achieve. A data pipeline will include, in order:
Data Processing
Data Store
User Interface
Now, we will dive in to technical definitions, software examples, and the business benefits of each.
Step 1: Data Processing.
Also called: ETL (Extract, Translate, Load).
Data processing extracts business data from all data sources. Then heavy-duty data cleansing and translation occurs. Once the data passes inspection and is normalized for the business-user, the data is loaded into the data store.
Data Processing Value-adds
Data Consistency - Automatically extract and load data from your various systems at regular intervals with the same technique
Data Confidence - Data cleansing provides accurate reporting and visualization and a fail-safe to notify you if/when you have dirty data
Strategic Freedom - Data translation includes data mapping and applying custom business logic to your data which creates flexibility when manipulating the information in the user interface.
What is data cleansing?
Data cleansing reviews all of your business data to confirm that it is formatted correctly and consistently; easy examples of this are fields such as: date, time, state, country, and phone fields. This is especially important when data is being extracted from multiple systems and may not have a standard format across the business.
During the cleansing process there are three things that can happen to the data:
Data is identified as clean and ‘approved’ for translation.
Data is identified as dirty, the data processor automatically fixes the problem, and the data is ‘approved’ for translation.
Data is identified as dirty but the data processor is unable to fix it. In this case, the data processor will either send a notification to the business user and quarantine the data until it is resolved; or if it is a significant issue the data processor will stop all activity and notify the business user that processing is on hold until the dirty data is addressed.
Without a data processor there is no easy way of knowing whether or not you are reporting on dirty data. The only alternative is to manually review all business data on a regular basis, which would be considered an impossible task for most.
What is data translation?
Data translation involves mapping and applying custom business logic to cleansed business data. These processes can be simple or complex, but the goal is always to organize and normalize your data for storage and consumption.
A few examples of data translation:
Creating relationships between objects in different systems.
When you unify your CRM, Accounting system, and marketing platform, each will have customer company information and contact information. With data translation, John Doe will be represented once in your user interface with a complete view of his information from all three systems.
Create new fields that may not exist in any of your systems.
Normalizing standard fields in all systems, such as state fields.
Software Examples
Alteryx
Tibco Jaspersoft ETL
Ri360
Step 2: Data Store.
Also called: Data warehouse, database (relational or non-sql), or data storage.
If the data from disparate business systems is an outline, and the data processor creates a rough draft, then the data store is the data’s final draft. Depending on who you talk to or what you read, some may argue that the data store is optional, but we believe that the security and efficiency it provides drastically changes the value for the business and end-user. Just like one could argue a rough draft may be enough, but it is always better to go the extra step and save a clean, organized final draft.
Data store value-adds
Business Autonomy - secure, unified storage of all business data.
Data compliance - stay in compliance with data privacy regulations such as GDPR and CCPA.
Efficient reporting - the user interface pulls information directly from the data store, without it you risk lag time and confusion if the user interface must pull information from the data processor.
Software Examples
MongoSQL
Hadoop
Ri360
MySQL
Oracle RDBMS
SQL Server
Step 3: User Interface.
Also called: Reporting, dashboarding, data visualization, data analytics.
The user interface is the end-result of all the data processor’s and data store’s efforts. It is the interface that the business user interacts with in their day-to-day work to report and analyze business data.
In our experience, there can be confusion about what a reporting tool / data analytics tool provides. More often than not, reporting tools are surface-level and do not include data processing and/or data storage. Be sure to find a user interface that meets your data pipeline needs.
User Interface Value-Adds
Business-user friendly and easy-to-use
Strategic business analysis / business intelligence
Productivity
Business data democratization
Software Examples
PowerBI
Tableau
Qlik
Ri360
Final Thoughts: A Complete Data Pipeline.
Done correctly and with purpose, data pipelines can dramatically change how a business is run. The technology alone provides quick wins for the business once implementation is complete, and it opens the doors to new business practices that may not have been an option before.
Most answers you will find for ‘what is a data pipeline?’ start with ‘a combination of technologies.’ Which is true, businesses usually find different tech to accomplish each part of their data pipeline.
You may have noticed that we listed Ri360 as a software example for all three data pipeline components (that wasn’t an accident). We have spent the last 13 years of business perfecting and streamlining our software, Ri360, to be a complete data pipeline.
At ReconInsight, our answer to ‘what is a data pipeline?’ has always been, “Ri360.”
