FRONT ROW AT THE BACKEND
Great that you have arrived, these are the best seats in the house!
HOW ESSENT IS BUILDING A DOPE DATA PLATFORM (YES, IT’S LITERALLY CALLED DOPE)
Welcome to the backend of Essent’s IT department, where innovation meets impact. I’m Jan Pieter, but you can call me JP. I’m a DevOps engineer at Essent, and in this podcast, you’re getting front, row access to what it means to work in a DevOps, driven energy tech company. In the last episode, we ventured into the data jungle with birthday boy Yannick. If you missed that, definitely give it a listen. It sets the scene perfectly for what we’re diving into today. And now, we’re going deeper, with the right tools to cut through the vines. Joining me today is Bjorn Mooijekind, one of the masterminds behind Essent’s self, service data platform, which, yes, is actually called DOPE. “I’m Bjorn, I’ve been at Essent for about two and a half years now. Before this, I worked at companies where we built data platforms too, but here we’re doing it better. There’s a strong focus on governance and building nice stuff for users.”
FROM CENTRAL BOTTLENECKS TO SELF, SERVICE POWER
At the heart of the DOPE platform is a mission to shift away from bottlenecks and toward empowerment. Historically, data went through a centralized funnel, think multiple teams and sprint cycles just to get one dashboard up. “If you wanted a report, you’d need to wait through three sprints. And if something needed changing, it would loop back through all those teams again. That takes time.” So, Bjorn and the team flipped the script. DOPE is all about self, service, enabling data analysts and engineers to onboard, process and analyze data with minimal dependency, all while embedding governance automatically. “It's easy to give access to data, but treating it properly, with governance in place, that’s the challenge. We built DOPE to do that, behind the scenes.”
WHAT’S UNDER THE HOOD OF DOPE?
Techies, this one’s for you. DOPE is a layered stack, and here are some of the key ingredients:
- DBT (Data Build Tool) for data transformation
- CloudQuery as the EL (Extract, Load) tool
- Snowflake for data storage
- Kubernetes clusters running the show
- FluxCD for GitOps, style deployment
- Argo Workflows and Argo Events for orchestration
- Prometheus and Grafana for monitoring
- Terraform for infrastructure, as, code
- Everything, as, code: no clicking allowed
- “We don’t do any clicking. Everything is code.”
The team has moved through various setups, AWS Batch, Fargate, ECS – and eventually landed on Kubernetes for performance and scalability.
“We tested it with 12,000 parallel tasks. It works. Fast and responsive. Tasks spin up in about four seconds, which is impressive compared to the 20 seconds we had before.”
BETA PHASE AND BEYOND
The platform is currently in beta, but it’s already gaining traction. “Most feedback is about things we already know, some fragility, lack of documentation, things like that. But those are being addressed. For instance, validation is now built into the implementation. Even documentation can be auto, generated.” There’s even talk of UI elements in the future, especially for data loading. “I don’t love UIs for complex stuff like SQL, but for EL, yeah, maybe. For less technical users, that could be a real help.”
BUILT BY ENGINEERS, FOR ENGINEERS
What makes DOPE particularly effective is that it’s being built by people who use it. Bjorn and his team are all data engineers. They know the pain points, and they’re designing for real, world usage. “You can write your own extractors, work with streaming data, or just write SQL. It supports both ends of the spectrum.” The project began with just two people, Bjorn and his colleague Rick, about a year and a half ago. It has since grown into a five, person team with diverse skills spanning cloud, Kubernetes, CLI tooling in Python, and services in Go. “We came from many places. We tested tons of tools , Airflow, Prefect, Mage, Flight, but Argo Workflows on Kubernetes is where we landed. It's fast, scalable, and works with how we build.”
WHAT’S NEXT?
The official release is just around the corner, and Bjorn encourages everyone to give DOPE a spin.
“If people want to try it, come find us. Our product owner, Mark, is awesome and happy to onboard new teams. We’re also running a community for data engineers where we share insights and explore each other’s infrastructures.”
No matter where you sit on the data engineer spectrum, SQL lover or stream wizard, DOPE is designed to make your life easier.
Stay tuned: next up, we’ll talk to Karol about how Essent’s cloud team became a strategic partner in making this all possible.