That Streaming Ingestion system with protobuf and generic Dataflow is very similar to something we created (ours was Avro and generic Dataflow!). We called it the Data Platform Gateway (DPG). We avoided the infra costs by having one stream, but that made the stream very difficult to use, and also made the autonomy problem worse.
Our next iteration became what we called data contracts (as compared to the DPG, which was mostly "just" schemas).
I find it interesting that at similar times different organisations were treading similar paths almost in isolation. Well, maybe not isolation, since there's probably loads of ideas and trends we were aware of that were influencing us both in similar ways, even if subconsciously. But I think one of the great things we have today is the low barrier entry to sharing ideas, and hopefully that's accelerating how things change for the better in our industry.
Interesting write up, thanks for sharing!
That Streaming Ingestion system with protobuf and generic Dataflow is very similar to something we created (ours was Avro and generic Dataflow!). We called it the Data Platform Gateway (DPG). We avoided the infra costs by having one stream, but that made the stream very difficult to use, and also made the autonomy problem worse.
Our next iteration became what we called data contracts (as compared to the DPG, which was mostly "just" schemas).
I find it interesting that at similar times different organisations were treading similar paths almost in isolation. Well, maybe not isolation, since there's probably loads of ideas and trends we were aware of that were influencing us both in similar ways, even if subconsciously. But I think one of the great things we have today is the low barrier entry to sharing ideas, and hopefully that's accelerating how things change for the better in our industry.
Looking forward to the next post!