ISG Research is happy to share insights gleaned from our latest Buyers Guide, an assessment of how well software providers’ offerings meet buyers’ requirements. The Data Observability: ISG Research Buyers Guide is the distillation of a year of market and product research by ISG Research.
Maintaining data quality and trust is a perennial data management challenge, often preventing enterprises from operating at the speed of business. In addition to automating
This is achieved using data observability, which ISG Research defines as providing the capabilities for monitoring the quality and reliability of data used for analytics and governance projects as well as the reliability and health of the overall data environment.
There has been a Cambrian explosion of data observability software providers in recent years, inspired by the observability platforms that provide an environment for monitoring metrics, traces and logs to track application and infrastructure performance.
To monitor and measure anything, it must first be instrumented, so a baseline requirement for data observability software is that it collects and measures metrics from data pipelines, data warehouses, data lakes and other data-processing platforms. Data observability software also collects, monitors and measures information on data lineage (dependencies between data), metadata (describing the attributes of the data, such as its age, volume, format and schema) and logs of human- or machine-based interaction with the data.
In addition to collecting and monitoring this information, some data observability software also enables the creation of models that can be applied to the various metrics, logs, dependencies and attributes to automate the detection of anomalies. Data observability software may also offer root cause analysis and the provision of alerts, explanations and recommendations to enable data engineers and data architects to accelerate the correction of issues, as well as take preemptive action to prevent data quality issues from reoccurring.
The metrics generated by data observability also form a critical component of the development and sharing of data products, providing the information by which data consumers can gauge if a data product meets their requirements in terms of a variety of attributes, including validity, uniqueness, timeliness, consistency, completeness and accuracy.
The importance of trust in data has arguably never been greater. As enterprises aspire to be more data-driven, it is critical to trust the data used to make those decisions. However, only 1 in 5 (20%) participants in ISG’s Analytics and Data Benchmark Research are very confident in the ability to analyze the quantity of data needed to make informed business decisions.
Assessing the quality of data used to make business decisions is not only more important than ever but also increasingly difficult, given the growing range of data sources and the volume of data that needs to be evaluated. Poor data quality processes can result in security and privacy risks as well as unnecessary data storage and processing costs due to data duplication. Without trusted and reliable data, enterprises may make decisions based on old, incomplete, incorrect or poorly organized data—or worse, no data.
Enterprises have previously sought to improve trust in data using data quality tools and platforms to ensure that data used in decision-making processes is accurate, complete, consistent, timely and valid. These are assessed in ISG’s Data Quality Buyers Guide.
Data observability complements the use of data quality products by automating the monitoring of data freshness, distribution, volume, schema and lineage as well as the reliability and health of the overall data environment.
The use of automation is an important characteristic of data observability software, expanding the volume of data that can be monitored while also improving efficiency compared to manual data monitoring and management. Automation is also integrated into data quality tools and platforms, however, to the extent that automation should not be considered a defining characteristic that separates data quality from data observability. A clearer distinction can be drawn from the scope and focus of the functionality. Data quality software is concerned with the suitability of the data for a given task. In comparison, data observability is concerned with the reliability and health of the overall data environment.
While data quality software helps users identify and resolve data quality problems, data observability software automates the detection and identification of the causes of data quality problems, such as avoiding downtime triggered by lost or inaccurate data due to schema changes, system failures or broken data pipelines, potentially enabling users to prevent data quality issues before they occur.
Data observability tools monitor not just the data in an individual environment for a specific purpose at a given point in time but also the associated upstream and downstream data pipelines. In doing so, data observability software ensures that data is available and up to date, avoiding downtime caused by lost or inaccurate data due to schema changes, system failures or broken data pipelines.
The two approaches are largely complementary. For example, when the data being assessed remains consistent, data quality tools might not detect a failed pipeline until the data has become out of date. Data observability tools could detect the failure long before the data quality issue arises. Conversely, a change in address might not be identified by data observability tools if the new information adhered to the correct schema. It could be detected—and remediated—using data quality tools.
The reciprocal nature of data quality and data observability software products is supported by the fact that some providers offer products in both categories. Others offer products that could be said to include functionality associated with both data observability and data quality.
Data observability is an important aspect of Data Operations, which provides an overall approach to automate data monitoring and the continuous delivery of data into operational and analytical processes through the application of agile development, DevOps and lean manufacturing by data engineering professionals in support of data production. In addition to the emergence of standalone data observability software specialists, we also see this functionality being included in wider DataOps platforms. This is a trend we expect to continue.
ISG asserts that through 2026, two-thirds of enterprises will invest in initiatives to improve trust in data through automated data observability tools addressing the detection, resolution and prevention of data reliability issues. Potential adopters of data observability are recommended to explore how the software can help increase trust in data as part of a broader evaluation of the people, processes, information and technology improvements required to deliver data-driven decision-making.
However, the evolution of data observability is still in its early stages. When evaluating data observability software, potential adopters are advised to pay close attention and assess products carefully. Some data observability products offer quality resolution and remediation functionality traditionally associated with data quality software, albeit not to the same depth and breadth. Additionally, some providers previously associated with data quality have adopted the term data observability but may lack the depth and breadth of pipeline monitoring and error detection capabilities.
The ISG Buyers Guide™ for Data Observability evaluates software providers and products in key areas, including the detection, resolution and prevention of data reliability issues. This research evaluates the following software providers that offer products to address key elements of data observability as we define it: Acceldata, Ataccama, Bigeye, Collibra, Dagster Labs, DataKitchen, DataOps.live, DQLabs, Great Expectations, IBM, Informatica, Monte Carlo, Precisely, Qlik, RightData, Soda and Validio.
This research-based index evaluates the full business and information technology value of data observability software offerings. We encourage you to learn more about our Buyers Guide and its effectiveness as a provider selection and RFI/RFP tool.
We urge organizations to do a thorough job of evaluating data observability offerings in this Buyers Guide as both the results of our in-depth analysis of these software providers and as an evaluation methodology. The Buyers Guide can be used to evaluate existing suppliers, plus provides evaluation criteria for new projects. Using it can shorten the cycle time for an RFP and the definition of an RFI.
The Buyers Guide for Data Observability in 2024 finds Monte Carlo first on the list, followed by DQ Labs and Acceldata.
Software providers that rated in the top three of any category ﹘ including the product and customer experience dimensions ﹘ earn the designation of Leader.
The Leaders in Product Experience are:
The Leaders in Customer Experience are:
The Leaders across any of the seven categories are:
The overall performance chart provides a visual representation of how providers rate across product and customer experience. Software providers with products scoring higher in a weighted rating of the five product experience categories place farther to the right. The combination of ratings for the two customer experience categories determines their placement on the vertical axis. As a result, providers that place closer to the upper-right are “exemplary” and rated higher than those closer to the lower-left and identified as providers of “merit.” Software providers that excelled at customer experience over product experience have an “assurance” rating, and those excelling instead in product experience have an “innovative” rating.
Note that close provider scores should not be taken to imply that the packages evaluated are functionally identical or equally well-suited for use by every enterprise or process. Although there is a high degree of commonality in how organizations handle data observability, there are many idiosyncrasies and differences that can make one provider’s offering a better fit than another.
ISG Research has made every effort to encompass in this Buyers Guide the overall product and customer experience from our data observability blueprint, which we believe reflects what a well-crafted RFP should contain. Even so, there may be additional areas that affect which software provider and products best fit an enterprise’s particular requirements. Therefore, while this research is complete as it stands, utilizing it in your own organizational context is critical to ensure that products deliver the highest level of support for your projects.
You can find more details on our community as well as on our expertise in the research for this Buyers Guide.