That Streaming Ingestion system with protobuf and generic Dataflow is very similar to something we created (ours was Avro and generic Dataflow!). We called it the Data Platform Gateway (DPG). We avoided the infra costs by having one stream, but that made the stream very difficult to use, and also made the autonomy problem worse.
Our next iteration became what we called data contracts (as compared to the DPG, which was mostly "just" schemas).
I find it interesting that at similar times different organisations were treading similar paths almost in isolation. Well, maybe not isolation, since there's probably loads of ideas and trends we were aware of that were influencing us both in similar ways, even if subconsciously. But I think one of the great things we have today is the low barrier entry to sharing ideas, and hopefully that's accelerating how things change for the better in our industry.
Thanks Andrew, yes I recognize quite a lot from your posts and your book :) Our next generation was a one stream solution but with messages carrying an entity attribute that let us look up respective contract (cached) for each message. More on that in the next post.
yes, that is something that strikes me as well when attending summits or reading posts, we are all trying to solve similar problems and often end up with similar solutions :)
Interesting write up, thanks for sharing!
That Streaming Ingestion system with protobuf and generic Dataflow is very similar to something we created (ours was Avro and generic Dataflow!). We called it the Data Platform Gateway (DPG). We avoided the infra costs by having one stream, but that made the stream very difficult to use, and also made the autonomy problem worse.
Our next iteration became what we called data contracts (as compared to the DPG, which was mostly "just" schemas).
I find it interesting that at similar times different organisations were treading similar paths almost in isolation. Well, maybe not isolation, since there's probably loads of ideas and trends we were aware of that were influencing us both in similar ways, even if subconsciously. But I think one of the great things we have today is the low barrier entry to sharing ideas, and hopefully that's accelerating how things change for the better in our industry.
Looking forward to the next post!
Thanks Andrew, yes I recognize quite a lot from your posts and your book :) Our next generation was a one stream solution but with messages carrying an entity attribute that let us look up respective contract (cached) for each message. More on that in the next post.
yes, that is something that strikes me as well when attending summits or reading posts, we are all trying to solve similar problems and often end up with similar solutions :)
Very interesting read. You could just not imagine the pain with migrating BI-tools. Isn't that supposed to be "easy"?!