Agile Data Engine designs high-performance Data Vault model with Databricks

At the Willibald Challenge of the DDVUG, providers of data warehouse automation can demonstrate how they solve general and advanced problems in connection with the Data Vault modeling approach. They have to solve various tasks using a sample data set provided. A fictitious company was created for this purpose: the seed and plant shop Willibald. It consists of two databases that represent 13 typical problems in the creation of a data warehouse. The Solita subsidiary Agile Data Engine (ADE) recently took up the challenge as the seventh provider - with Databricks in its luggage. ADE also presented the collaborative solution at the latest Databricks Bootcamp on February 20 in Frankfurt am Main.

"Our participation in the Willibald Challenge proved to be a rewarding experience to test Agile Data Engine's capabilities in a demanding environment," explains Matti Karell, Managing Director of Agile Data Engine. "Databricks' support for SQL-based queries, data lineage and BI integration perfectly matched the capabilities of our platform and our goals for the Challenge."

The compatibility between Agile Data Engine and Databricks SQL enabled ADE to quickly set up a robust, high-performance data vault model with all the necessary components. From multi-active and effective satellites to transactional and hierarchical joins, Databrick's advanced Data Vault components could easily handle complex data requirements without bottlenecks.

Matti Karell: "Databricks' Unity Catalog Lineage Tracking brought transparency and efficiency to our data management practices and made it easy to track data flows and transformations across the entire pipeline. The result is improved data governance."

Agile Data Engine helps organizations design, automate and orchestrate their Data Vault implementation. However, implementing a Data Vault model at scale requires a certain amount of technical effort to optimize the data layout of the growing objects. Predictive Optimization (PO) in Databricks simplifies this process by automating table management via Unity Catalog and the Data Intelligence Platform. PO intelligently optimizes the data layout for tables managed by Unity Catalog, resulting in improved query performance and lower storage costs.

There are two common challenges when building data platforms: ensuring high quality data and making it available to users quickly. However, it is often unclear whether they can fully utilize all available data for insights and decisions. This is where AI/BI Genie from Databricks comes into play, an AI-powered interface with data intelligence that enables business users to perform data analysis independently in natural language. Other features that make a Databricks Lakehouse ideal for implementing Data Vault with Agile Data Engine include Databricks' Serverless Warehouse, Variant Datatype and Autoloader

agentur auftakt
Bergmannstr. 26
80339 München
Germany

https://www.agentur-auftakt.de

Herr Frank Zscheile
+49-89-2622 4718

presse@agentur-auftakt.de

Agile Data Engine is a Finnish DataOps provider and subsidiary of the data management specialist Solita with over 2,200 employees. Agile Data Engine's core product is the all-in-one software platform of the same name for the development, provision and operation of data products and pipelines. Companies can use it to automate cloud data warehouse processes and make optimum use of their data warehouse. The first version of the platform was created in 2016 as part of Solita data projects. Over 40 well-known customers in the Nordic countries are already using the product. Since 2022, the Agile Data Engine spin-off, which now has more than 25 employees, has been responsible for further development and sales of the platform. The company is headquartered in Helsinki and has its German headquarters in Munich.
www.agiledataengine.com

This release was published on openPR.