Interesting post, I've shared this in my newsletter!
I've too found myself spending most of my time on some Data Platform projects writing Infrastructure as Code for data lakes and databases and networking them together.
Those projects have often had the domain experts building the dashboards themselves: some are like Analytics Engineers doing the modelling and pipelines too, but some just make dashboard and reports based on data presented.
Though I'm not sure I'd want to give up the building of pipelines and dashboards - there something satisfying about building everything needed for a end to end data pipeline, even if it is a bit exhausting to wear too many hats!
Really love the data-engineering != data-engineer. Really feeling like the data engineer role is to a certain extent a replacement for traditional BI tools - as we have more and more tools trying (and succeeding) to improve data engineer work, we are shifting the data engineer work on other part of the stack. Where there is more value to bring.
Very well written, Robert! A lot of good points. One question: you are mentioning testing as an area that is important, but I dont hear you (or anybody else) talk about a specific test role. Is that you opinion that should be carried out by the data engineers/analysts/scientists end-to-end?
Interesting post, I've shared this in my newsletter!
I've too found myself spending most of my time on some Data Platform projects writing Infrastructure as Code for data lakes and databases and networking them together.
Those projects have often had the domain experts building the dashboards themselves: some are like Analytics Engineers doing the modelling and pipelines too, but some just make dashboard and reports based on data presented.
Though I'm not sure I'd want to give up the building of pipelines and dashboards - there something satisfying about building everything needed for a end to end data pipeline, even if it is a bit exhausting to wear too many hats!
Double click a couple more times and what you are saying is that these roles are ultimately merging into software engineering.
Here is my take on the future of data products. Something's got to give!
https://open.substack.com/pub/dataproductleader/p/the-third-data-revolution
Very good read !
Really love the data-engineering != data-engineer. Really feeling like the data engineer role is to a certain extent a replacement for traditional BI tools - as we have more and more tools trying (and succeeding) to improve data engineer work, we are shifting the data engineer work on other part of the stack. Where there is more value to bring.
Big echo with my last blog post https://medium.com/@benoit.pimpaud/data-engineer-is-a-transitional-job-ed0074c89646
I would like to thank all data engineers for their hard work on improving data collecting 😊😇
Very well written, Robert! A lot of good points. One question: you are mentioning testing as an area that is important, but I dont hear you (or anybody else) talk about a specific test role. Is that you opinion that should be carried out by the data engineers/analysts/scientists end-to-end?