Analyst Perspectives

Databricks recently hosted its Data+AI Summit in San Francisco, an event that attracted 22,000 attendees. That’s a far cry from the Spark Summit I attended in 2016. As pointed out in my coverage of Databricks massive funding round earlier this year, the company was originally founded as a provider of cloud-based Apache Spark services. Since its inception, Spark has been associated with processing big data, and over time this has become almost synonymous with AI. Consequently, the event became the Spark+AI Summit and eventually the Data+AI Summit. The association between data and AI has become the focus of attention for many enterprises. The top challenge organizations reported in ISG’s Data and AI Programs Study was making data usable for AI applications.

We have written about many of the challenges associated with using data for AI. Databricks made several announcements to address some of these issues. As my colleague has pointed out, enterprise needs for real-time applications driven by online predictions and recommendations has increased the requirements for operational data platforms coupled with analytical data platforms. Toward that end, Databricks announced Lakebase, a serverless Postgres database integrated with the lakehouse. Lakebase is based on Databricks’ recent acquisition of Neon and is designed to support agentic AI processes as well as other operational workloads.

As we’ve written previously, Enterprises also require data pipelines to ensure data is integrated and processed in the sequence required to generate business intelligence (BI) and support the development and deployment of applications driven by artificial intelligence (AI). Databricks announced Lakeflow Designer, a no-code/low-code data pipeline tool to make it easier to create data integration and preparation pipelines. The pipelines generate code which can be deployed into production and can also be edited by data engineers that prefer to work with code. These pipelines are also integrated with Unity Catalog, the Databricks data catalog offering.

Data catalogs provide an inventory of data available to the enterprise including the lineage of that data. They enhance governance and, as we’ve written, they improve confidence in the data and they improve the organization’s ability to govern and manage data. One problem is the proliferation of catalogs leading to challenges integrating information across the various catalogs. Databricks is attempting to establish Unity Catalog as a standard that others can support. As part of their efforts to embrace a wider portion of the data and analytics market, they’ve introduced Unity Catalog Metrics. For years I’ve pointed out the disconnect between analytics governance and data governance. Unity Catalog Metrics allows enterprises to capture and share the definitions of their KPIs and business metrics ensuring not only visibility but consistency across these metrics. Today, the definition of the metrics is based on SQL. That’s a good start, but hopefully, the languages for defining business metrics will be expanded in the future in order to capture richer business metrics definitions.

Also related to analytics, Databricks has made significant enhancements to its business intelligence offering which it refers to as AI/BI. The company touted its business intelligence capabilities with new features such as cross filtering between displays, brushing and other features which are common and necessary to meet business requirements. Part of what has created an opportunity for Databricks in the business intelligence market is the disruptive force of generative AI (GenAI). Although business intelligence is a mature market, GenAI is changing the face of analytics and creates the potential for disintermediation of analytics vendors by data platform vendors. Databricks has incorporated GenAI in Genie to provide a natural language interface and automated insights.

While GenAI has provided a new way to interact with data and analytics via a chat interface, as I’ve written, it still leaves too much of the exercise to the reader. Agentic AI is an extension or evolution of GenAI to help automate actions instead of just providing information. There is much promise and there are many challenges associated with agentic AI. In preparing our recent AI Platforms Buyers Guide we observed that only 17% of AI platform providers included agent design tools and even fewer (11%) offered agent evaluation tools. To address these needs, Databricks introduced Agent Bricks. Users can create agents by providing a description of the task(s) that need to be accomplished. In addition, tools are included to evaluate the agent’s performance and to optimize the deployment of the agent with consideration for cost, performance and accuracy.

With various new features designed to move functionality closer to business users—including Lakeflow Designer, AI/BI and Agent Bricks—the company has also introduced Databricks One to simplify the user experience. Users don’t need to understand clusters or use notebooks. Databricks One surfaces AI/BI Genie and enables navigation among different domains which are collections of data and dashboards.

And getting back to its roots, Databricks made several Apache Spark announcements. In the Day Two keynotes, Matei Zaharia, Databrick CTO and co-founder as well as the creator of Apache Spark, introduced Apache Spark 4.0. Databricks also contributed declarative pipelines to the Spark open source project.

The event showcased the popularity of Databricks and included multiple announcements that should be welcomed by customers and prospects. Company executives also recognize and acknowledge that there is more work to be done. Agentic AI is still in its infancy and will continue to evolve. We’ve evaluated Databricks in various data and AI ISG Buyers Guides and they were rated Exemplary repeatedly including AI Platforms, Agentic and Generative AI, Data Integration, Data Pipelines, Real Time Data and others. We recommend that enterprises considering data and AI platforms consider Databricks in their evaluation set.

Regards,

David Menninger