Market Perspectives

ISG Research is happy to share insights gleaned from our latest Buyers Guide, an assessment of how well software providers’ offerings meet buyers’ requirements. The Data Orchestration: ISG Research Buyers Guide is the distillation of a year of market and product research by ISG Research.

While data-driven enterprises stand to gain a competitive advantage by responding faster to worker and customer demands for more innovative, data-rich applications and personalized experiences, this increasingly relies on a complex array of data pipelines to support agile, continuous data processing. Given the increasing complexity of evolving data sources and requirements, it is essential to automate and coordinate the creation, scheduling and monitoring of data pipelines as part of a DataOps approach to data management.

This is the realm of data orchestration, which ISG Research defines as providing the capabilities to automate and accelerate the flow of data to support operational and analytics initiatives and drive business value via capabilities for the monitoring and management of data pipelines and associated workflow.

At the highest level of abstraction, data orchestration covers three key capabilities: data collection, including data ingestion, preparation and cleansing; data transformation, additionally including integration and enrichment; and data activation, making the results available to compute engines, analytics and data science tools, or operational applications.

This may sound very much like the tasks that data management practitioners have undertaken for decades. As such, it is fair to ask what separates data orchestration from traditional approaches to data management.

Viewing data management challenges through the lens of today’s data-processing requirements is key to understanding why data orchestration is a necessary and differentiated approach that goes beyond traditional data management.

Being data-driven requires a combination of people, processes, information and technology improvements involving data culture, data literacy, data democracy and data curiosity. Encouraging workers to discover and experiment with data is a key aspect of being data-driven that requires new, agile approaches to data management.

Additionally, the increasing reliance on real-time data processing is driving requirements for more agile, continuous data processing. The rapid adoption of cloud computing has fragmented where data is generated and stored. Enterprise data is increasingly spread across multiple data centers and cloud providers.

Traditional approaches to data management are rooted in point-to-point batch data processing, whereby data is extracted from its source, transformed for a specific purpose and loaded into a target environment for analysis. These approaches are unsuitable for the demands of modern analytics environments, which instead require agile data pipelines that can traverse multiple data-processing locations and evolve in response to changing data sources and business requirements.

Given the increasing complexity of evolving data sources and requirements, there is a need to enable the flow of data across an enterprise through new approaches to the creation, scheduling, automation and monitoring of workflows. Traditionally, individual tasks related to these requirements have been addressed with a variety of specialist tools as well as manual effort, hand-coded scripts and expertise.

In comparison, data orchestration tools are designed to automate and coordinate the sequential or parallel execution of a complete set of tasks via data pipelines, typically based on directed acyclic graphs that represent the relationships and dependencies between the tasks. The capabilities delivered by data orchestration fall under three categories: pipeline monitoring, pipeline management and workflow management.

As is often the case with new approaches to data and analytics, the requirements for data orchestration were first experienced by digital-native brands at the forefront of data-driven business strategies. One of the most prominent data orchestration tools, Apache Airflow, began as an internal development project within Airbnb, becoming an Apache Software Foundation project in 2016. Workflow automation platform Flyte was created and subsequently open-sourced by Lyft, and Metaflow was developed and open-sourced by Netflix.

Data orchestration is not just for digital natives, however, and a variety of software providers have sprung up with offerings based around these open-source projects, as well as other development initiatives, to bring the benefits of data orchestration to the masses.

In addition to stand-alone data orchestration software products and cloud services, data orchestration capabilities are also being built into larger data-engineering platforms addressing broader data management requirements, including data observability, often in the context of data fabric and data mesh.

Whether stand-alone or embedded in larger data-engineering platforms, data orchestration has the potential to drive improved efficiency and agility in data and analytics projects. ISG asserts that by 2027, more than one-half of enterprises will adopt data orchestration technologies to automate and coordinate data workflows and increase efficiency and agility in data and analytics projects.

Adoption of data orchestration is still in the early stages and is closely linked to larger data transformation efforts that introduce greater agility and flexibility. If an enterprise’s data processes and skills remain rooted in traditional products and manual intervention, then data orchestration is not likely to be a quick fix. However, alongside the cultural and organizational changes involved in people, processes and information improvements, data orchestration has the potential to play a key role in the technological improvement involved in becoming more data-driven. All enterprises are recommended to explore how the orchestration of data pipelines can help increase the potential for improved data-driven decision-making as part of a broader evaluation of the people, processes, information and technology improvements required to deliver data-driven decision-making.

The orchestration of data pipelines is just one aspect of improving the use of data within an enterprise. In addition to the development, testing and deployment of data pipelines, DataOps also encompasses data observability, which has a complementary role to play in monitoring the health of data pipelines and associated workflows as well as the quality of the data itself.

The combination of healthy and well-orchestrated data pipelines and data observability is also complementary to developing and delivering data products, ensuring that data consumers can trust the provenance and quality of the data that is made available across the enterprise.

Data orchestration is also integral to the development and delivery of applications driven by artificial intelligence and generative AI, complementing MLOps, which serves the collection of artifacts and orchestration of processes necessary to deploy and maintain AI/ML models. Specifically, data orchestration can be used to automate and accelerate the flow of data from multiple sources, including existing applications and data platforms as well as the output of large language models and vector databases. Almost one-half (49%) of participants in ISG’s 2023 Application Development and Maintenance Study expect to AI-enable applications by embedding AI and ML models into current applications and processes.

The ISG Buyers Guide™ for Data Orchestration evaluates software providers and products in key areas, including data pipeline management, workflow management and pipeline deployment. This research evaluates the following software providers that offer products to address key elements of data orchestration as we define it: Alteryx, Amazon Web Services, Astronomer, BMC, Cloudera, Dagster Labs, Databricks, DataKitchen, DataOps.live, dbt Labs, Google, Hitachi, IBM, Informatica, Infoworks, K2View, Keboola, Matillion, Microsoft, Nexla, Prefect, Rivery, Saagie, SAP, Stonebranch, Y42 and Zoho.

This research-based index evaluates the full business and information technology value of data orchestration software offerings. We encourage you to learn more about our Buyers Guide and its effectiveness as a provider selection and RFI/RFP tool.

We urge organizations to do a thorough job of evaluating data orchestration offerings in this Buyers Guide as both the results of our in-depth analysis of these software providers and as an evaluation methodology. The Buyers Guide can be used to evaluate existing suppliers, plus provides evaluation criteria for new projects. Using it can shorten the cycle time for an RFP and the definition of an RFI.

The Buyers Guide for Data Orchestration in 2024 finds Databricks first on the list, followed by Microsoft and Alteryx.

Software providers that rated in the top three of any category ﹘ including the product and customer experience dimensions ﹘ earn the designation of Leader.

The Leaders in Product Experience are:

• Microsoft.
• Databricks.
• Google.

The Leaders in Customer Experience are:

• Databricks.
• Microsoft.
• SAP.

The Leaders across any of the seven categories are:

• Informatica, which has achieved this rating in five of the seven categories.
• Microsoft and SAP in three categories.
• AWS, Databricks and Google in two categories.
• Alteryx, Astronomer, BMC, Keboola and Stonebranch in one category.

The overall performance chart provides a visual representation of how providers rate across product and customer experience. Software providers with products scoring higher in a weighted rating of the five product experience categories place farther to the right. The combination of ratings for the two customer experience categories determines their placement on the vertical axis. As a result, providers that place closer to the upper-right are “exemplary” and rated higher than those closer to the lower-left and identified as providers of “merit.” Software providers that excelled at customer experience over product experience have an “assurance” rating, and those excelling instead in product experience have an “innovative” rating.

Note that close provider scores should not be taken to imply that the packages evaluated are functionally identical or equally well-suited for use by every enterprise or process. Although there is a high degree of commonality in how organizations handle data orchestration, there are many idiosyncrasies and differences that can make one provider’s offering a better fit than another.

ISG Research has made every effort to encompass in this Buyers Guide the overall product and customer experience from our data orchestration blueprint, which we believe reflects what a well-crafted RFP should contain. Even so, there may be additional areas that affect which software provider and products best fit an enterprise’s particular requirements. Therefore, while this research is complete as it stands, utilizing it in your own organizational context is critical to ensure that products deliver the highest level of support for your projects.

You can find more details on our community as well as on our expertise in the research for this Buyers Guide.