The data platform ketchup effect!
Data contracts + unified architecture as drivers for adoption
There are many ways to operate a data platform team, but ultimately the success of the data platform is defined by how many (data) product teams adopt it.
At Mathem we are pivoting from a data engineering team building data pipelines and data models to a data platform team providing capabilities to squads/teams to take on ownership and build pipelines and data models. Three major technological changes that have been crucial to enable this pivoting are:
Data contracts - we’ve shifted from proto that essentially only defined schemas. It was cumbersome and didn’t validate payload before deploy, nor schema evolution compatibility and no meta-data. Now we use YAML contracts with validations on PRs and merge and a review process where analytics engineers (main consumer) can communicate with squads (producers) before new data streams are added.
Unified architecture - regardless data input port (events, cdc, files, APIs) we now pipe data through the same streaming processing component that enforces the contracts before data is ingested to our data warehouse (BigQuery). We want to avoid multiple ingestion services and only maintain and develop one core component while offering flexibility in terms of input ports. Also, the new component (StreamProcessor) is message based rather than the previous topic based dataflow solution. This means we can now run all data streams through the same job and add new data streams without any downtime or redeployments. Hence we get much better resource utilization and cost per GB processed has dropped to 25% of the legacy solutions and it keeps getting better with each data stream added.
DBT as a service. Previously we used composer (airflow) for transformation, validation and data exports/feeds. Not only was/is it cumbersome to use by product teams and maintain but also not cheap and it threw a number of random kubernetes related alerts. Now we offer DBT core as a cloud run service to be used by analytics engineers and data scientists. It is completely serverless with CI/CD and a PR process that foster communication on a completely different level. The service also generates docs, lineage and alerts (Elementary). It is also used to generate feeds/exports by defining logic in dbt that export to cloud storage where we have an event driven solution to transfer the export. This enables product teams to own and maintain the logic without the involvement of the platform team. We also get the same lineage and alerts on feeds. Contracts are also used to generate silver layer models and hence removing manual and repetitive work from producers and platform team.
So what happens when you get a good product/market fit of your platform, i.e. a better data product developer experience?
-> You get a ketchup effect!!!
In 4 years we onboarded roughly 45 data streams. With the new platform we have onboarded 82 data streams in 7 months and this week we had 5 new data contract PRs from different squads who want to onboard new data streams. Also, the dbt service repo get around 20-40 PRs/week to add/modify one or more models which is at least 10x compared to the composer based solution.
Cost savings, less maintenance, etc. is all great, but remember that the success of the data platform is defined by how many product teams adopt it!
P.S. to give context to the numbers, Mathem tech + data < 100 FTE so adoption of data contracts in squads and DBT in the data team is almost 100%.
P.S.S. Ketchupeffekt (Swedish): It's used to describe a situation or event where not much happens for a long time, and then a lot happens at once. The significance of ketchup will be clear to anyone who has ever tried to pour the sauce from a glass bottle -- usually it takes several seconds before any comes out, and then you get a huge amount.
Ketchupeffekt is often used in a sporting context, for example to describe a game or match with no goals for the majority of the time, and then several in the last few minutes, or a team who have an unremarkable season and then multiple spectacular wins in their final games. https://www.thelocal.se/20181211/swedish-word-of-the-day-ketchupeffekt
Hey Robert, great content. Thanks for sharing! You are allowing teams inside your org run DBT as part of a batch process, triggering the Cloud Run service, right? How are you guys orchestrating the execution of those containers?