FRONT ROW AT THE BACKEND
Great that you have arrived, these are the best seats in the house!
FROM BOTTLENECKS TO BREAKTHROUGHS: HOW ESSENT’S DATA TEAMS ARE ACCELERATING CHANGE
Welcome to the heart of Essent's data revolution. In the latest episode of our podcast, “Front Row at the Back End,” host and DevOps engineer JP (Jan-Pieter) sits down with Yannick van der Wende, Scrum Master at Essent, to explore the hidden world behind the dashboards, where terabytes of data power everything from government reports to energy trading strategies.
The Data Dilemma: “You might need to wait nine weeks…”
Data at scale isn’t just about size, it’s about speed, accuracy, and accessibility. As Yannick puts it, “The lead time to get new data in is very high. A lot of people want data, and that’s where the bottleneck starts to come up.”
He illustrates this with a real example: a seemingly simple data request that took “two days of actual work, but a nine-week wait due to backlog.” This delay stems not from technical complexity, but from demand outpacing delivery, an all-too-familiar reality in enterprise-scale environments.
From Manual Releases to Daily Deployments
Before Yannick stepped in, some data teams at Essent released updates just once or twice per sprint, with much of the work done manually. Fast-forward to today: “They now deliver twice a day, and we're aiming for three,” Yannick shares.
The key? Investing in CI/CD pipelines tailored for data. “We’re working with billions of rows. Just one run on production takes four to six hours,” he says. “It’s not just about moving fast, it’s about building reliability into scale.”
Dependencies, Democratization, and the Data Mesh
Essent’s current structure involves multiple specialized teams, one pulling in data, another handling warehousing, and others focused on analysis and reporting. This sequential setup introduces delays. “We’re still a bit old-school,” Yannick admits. “We’re trying to streamline this with better technology and team alignment.”
Enter the DOPE platform, Essent’s answer to data democratization. Instead of data teams acting as gatekeepers, DOPE empowers every department to access and use data safely and efficiently.
“Our role becomes more about laying the ground rules and acting as consultants to help others use the platform,” Yannick says. “We want the entirety of Essent to be able to use data to their advantage.”
Lessons in Scale: “Data engineering is different… and still the same.”
Reflecting on his journey from .NET developer to Scrum Master for data teams, Yannick shares a moment of humility: “I used to joke that my job was just moving data from A to B. I don’t make that joke anymore.”
Working with data teams opened his eyes to the complexity behind the scenes, from supporting scientific models and analytics, to handling compliance reporting with zero tolerance for errors. “Sometimes you think a data source isn’t critical, until five people are at your desk asking why it’s broken.”
Shoutout to the Cloud Empowerment Team
Before signing off, Yannick tips his hat to the unsung heroes behind the scenes: “The cloud team at Essent has been a tremendous help. They keep things running and offer invaluable support.”
What’s Next?
In the next episode, we speak to Bjorn about ensuring data engineers hit the ground running with the help of our evolving data platform.