As enterprises embracing artificial intelligence move from initial pilots and trial projects through deployment and into production at scale, many are realizing the critical importance of reliable data management and agile, responsive data operations (DataOps) processes to improve trust in the data used for AI and business intelligence. With an emphasis on agility, automation and continuous delivery, DataOps has been pitched by some advocates as the antithesis of traditional, manual, batch-based approaches to data management. Although products in each category continue to serve specific roles, it has become clear that a holistic approach to data management and operations is required if enterprises are to harness and nurture data from multiple sources to fuel success with AI.
Data is integral to AI. Large volumes of data are required to train models. Data freshness is important to inferencing in interactive applications, and data quality is fundamental to ensuring that the output of agentic and generative AI initiatives can be relied upon. Poor
The term DataOps emerged just over a decade ago to describe a set of tools, practices and a philosophy used to support the continuous processing and delivery of data in the face of constant change. ISG defines DataOps as the application of agile development, DevOps and lean manufacturing by data engineering professionals to support data production. The need for agility, collaboration and automation has accelerated interest in DataOps. ISG
The emphasis on agility, collaboration and automation initially separated DataOps products from existing approaches to data management, which were traditionally batch-based, manual and rigid tools and practices. However, the distinction between DataOps and traditional data management tools is clearer in theory than in practice. In recent years, many providers of traditional data management products have added capabilities to improve automation and acceleration, while DataOps providers have expanded into market categories previously led by established data management providers. The lines between previously distinct product categories have blurred considerably.
This is the primary reason why, this year, we are running a single research program to create our Buyers Guides for Data Management and Data Operations simultaneously. As part of the overall Data Management and Operations Buyers Guide research, we will continue to assess the individual market segments of both data management and DataOps. Data management software combines data governance, data quality, master data management, data integration and data intelligence to ensure that enterprises collect, store and process data in accordance with strategic goals and regulatory requirements. DataOps encompasses the development, testing, deployment and orchestration of data integration and processing pipelines, along with improved data quality and validity via data monitoring and observability, enabling the development and consumption of data products.
Running the programs concurrently enables us to combine elements of data management and DataOps to reflect how enterprises use them. For example, I have previously written about the complementary nature of data quality and data observability. While data quality software helps users identify and resolve specific data quality problems, data observability software automates the detection and identification of the causes of data quality problems.
Many providers now offer functionality that spans both data observability and data quality, while some that historically focused on data quality have adopted the term data observability, and vice versa. While these products are assessed separately in our respective Buyers Guide for Data Quality and Buyers Guide for Data Observability, this year we will also assess providers’ ability to address requirements for both in our Buyers Guide for Data Quality and Data Observability.
I have also examined the evolving landscape for platforms that enable the development, sharing and consumption of data products. This landscape includes dedicated platforms for developing and sharing data assets with a product-thinking approach, as many providers of data intelligence software also incorporate integrated features for data product development, discovery and management. While data intelligence platforms are not required to deliver data products, the holistic, business-level view of data production and consumption they support facilitates the delivery of data as a product. We continue to assess data product-related capabilities within the Buyers Guide for Data Intelligence, as well as assessing data intelligence-related capabilities within the Buyers Guide for Data Products. Creating the combined Buyers Guide for Data Intelligence and Data Products better enables the identification of providers that address the requirements for both.
My Analyst Perspective on the overlap between data pipelines, data orchestration and data integration highlights the transition to combined functions. Given the increasing complexity of evolving data sources and requirements, data orchestration is essential to automate and coordinate the creation, scheduling and monitoring of data pipelines that automate and accelerate the flow of data supporting operational and analytics initiatives. Although point-to-point data integration remains a primary use case for data pipelines, not all data pipelines perform data integration.
Our new Buyers Guide for Data Engineering reflects the holistic practice of developing and managing pipelines that collect data as it is generated by multiple applications and systems, unify and combine it as required and deliver it to multiple applications to support operational, analytic and AI use cases. The Buyers Guide for Data Engineering reflects the requirements for data pipelines of all types, along with higher-level data operations tasks related to pipeline testing, deployment and orchestration. We continue to assess the specific requirements for each of these via our Buyers Guide for Data Pipelines, our Buyers Guide for Data Orchestration and our Buyers Guide for Data Integration.
An enterprise must have trust in data to make data-driven business decisions. Establishing and maintaining trust in enterprise data accelerates the delivery of analytics and artificial intelligence projects, providing the confidence required to make agile business decisions. It is also a complex process that depends on multiple factors, including data governance, master data management and data quality. These product categories are also complementary. Master data modeling results in the generation of data catalog entries or enterprise glossary information that can be shared across the enterprise, as well as with partners and suppliers, and can form the basis of agreed definitions that can be implemented by data governance products and used to support data quality initiatives.
Our new Buyers Guide for Data Integrity assesses software products that provide enterprises with confidence that data used for operational, analytic and AI use cases can be trusted for accuracy, completeness, consistency and fit for purpose. It does so by combining the core elements of data governance (such as data usage, data lineage, data quality, data security and access control) and data quality (including accuracy, completeness, consistency, timeliness and validity), as well as MDM (including data validation, matching, merging, enrichment and data modeling). We continue to assess standalone products for each of these elements with our Buyers Guide for Data Governance, Buyers Guide for Master Data Management and the Buyers Guide for Data Quality.
In combination, the data management and operations product landscape comprises a variety of platforms and specialist tools that can and should be used together to address the full breadth of requirements for data management and DataOps. I recommend that enterprises evaluating products and providers pay close attention to the functional capabilities on offer for the specific use case, as well as the context of a holistic approach to data management and operations.
Regards,
Matt Aslett