Essent’s Modernization Journey
I am Anthony Roes, a Product Owner within Essent and in this blogpost I would like to tell you a bit about the modernization journey that we are currently on and the cool things we have realized along the way. As you can see in the picture below, we are on a modernization track that involves:
1) the consolidation of several legacy Data Warehouses into a new Cloud DWH platform
2) Upgrading the Data Analytics Platform from Qlikview to Qlik Sense
Background Snowflake
My first interaction with Snowflake was when I followed one of their free ‘virtual hands-on labs’. I saw the simplicity of using Snowflake’s web user interface, but my mind was really blown when I saw an “undrop” statement that is made possible by Snowflake’s unique time-travelling functionality. I thought to myself how many people could have benefitted from this functionality when they accidentally dropped a table/database.
So, what is Snowflake? Snowflake is a cloud-based data warehousing company based in California and founded in 2012. Snowflake helps organizations more than ever before. Its cloud data platform is capable of running any workload, enables secure and governed access data to all and offers unlimited performance and scalability modern enterprises need. All this in an easy-to-use platform delivered as a service. It is cloud agnostic and can run on AWS, Microsoft Azure, and Google’s cloud platform. Snowflake was the brainchild of two experts: Benoit Dageville and Thierry Cruanes. Their view was that data warehouses were and still are rigid, expensive, and difficult to use. They decided to revolutionize the industry by running the warehouse on the cloud while factoring in speed, ease of use, costs, and elasticity. Furthermore, Snowflake differs from the rest of the pack in terms of their payment model. Users pay per usage which provides an option to decrease the total cost of ownership.
Why move to Snowflake
So, what are some concrete benefits of moving to Snowflake? Well, Vikash Kumar has already published a very comprehensive [article] about the reasons why you should use Snowflake so allow me to dissect the benefits mentioned in his article. First, Snowflake data warehouse has a hybrid of shared-disk and shared-nothing database architectures. Making it accessible from all compute nodes and processes query using MPP compute clusters. Secondly, the Database Storage Layer ensures high data availability being replicated across multiple disk devices. Third, Snowflake’s Virtual Warehouse Processing layer consist of one or more virtual warehouses depending on the requirements of an organization. These warehouses do not share compute recourses making it an easy option to separate certain workloads to avoid contention and allow for horizontal scalability. And finally, Snowflake’s Cloud Services Layer which are fully maintained by Snowflake for metadata, security, query compilation, optimization, and warehouse management. Below you can find a representation of the implementation of Snowflake at Essent.
Background of the Solution to support the Affordability Initiative at Essent
As Snowflake was being introduced to our engineers at Essent, some geopolitical events started happening in the World, which greatly impacted the energy sector. A war was initiated in Eastern Europe and suddenly, the supply of gas to the rest of Europe dramatically decreased. As a result, gas prices increased dramatically as well as the pressure on energy companies dealing with this situation. Shortly after, headlines in all Dutch newspapers popped up that consumers should anticipate increased energy bills, especially at the end of the year. Knowing these problems would be a hot topic for Essent in the coming period and with the newly introduced possibilities of Snowflake, the Engagers Agile Release Train (SAFe ART) acted in true Agile fashion and changed priorities.
Becoming a Tech Company using Snowflake
To best explain how we see ourselves becoming a true tech company, a few words from our colleague [Geert Bossuyt]: “It seems obvious that energy and utilities need to transform into something sustainable and less dependent on foreign sources. Which is easier said than done. Essent and E.ON are positioned well to have a genuine impact on the global energy transition: we work with suppliers, customers, and all other parties involved. We do experiments, learn what works and also learn what doesn’t work. One of the crucial enablers for Essent to have this impact is a fast and reliable IT Delivery. This has to do with short feedback cycles, fast releases, small increments, high performance, and a great user experience. Taking on this role will help us to become a true Tech Company. Essent tries to help the customer”. So how did we implement this philosophy facing our current crisis?
Essent’s customer assistance
As the affordability of the energy bill is a big challenge for many customers, Essent wants to avoid any surprises around the yearly annual energy bill. First, some hypotheses were made to analyse if there would indeed be an increased amount of “bill shocks” especially nearing the end of the year. Let me explain what a bill shock exactly is: a billshock occurs when the yearly annual bill is higher than the sum of all installment amounts that have been paid over the year by the customer. The difference in what has been paid and still has to be paid can be so high that it surprises or “shocks” customers. Whereas some customers have increased their monthly bill amount (Termijnbedrag), this is not the case for all, making the bill shock specifically in the current market situation a bigger shock then it usually is. Once the hypothesis of an increased bill shock was confirmed a plan was implemented to effectively gather insights in order to be able to help the customers that will be receiving a relatively high yearly annual energy bill.
Context for BTI Betaalbaarheid why were these insights required
It was clear that to help our customers, insights were needed on the termijnbedragen (monthly bill) and adviestermijnbedragen (recommended monthly bill). Once the relevant data was collected from different sources (SAP & Calculation Result Store) a prototype could be started to merge these two datasets. In true SAFe Agile fashion priorities were changed during an already started Program Increment (Quarterly plan). We started midway a Program Increment to validate the value of the insights in a Prototype and followed up in the next Program Increment and delivered a fully modelled data product that serves as input for the Termijnbedrag Dashboard. It provides insights on the delta between calculated consumption and the sum of all instalment amounts enriched with customer data (customer segment, contract type, type of product, etc.) which enables the users to make a customer selection and pinpoint exactly those customers that are in danger of suffering a significant bill shock. That selection can in turn be used to take action and pro-actively help our customers and find a solution for their particular situation. We are currently in the middle of this exercise and compared to the past we have the insights to take action to avoid these situations in the future.
Main Obstacles in Snowflake & Insights
Quickly switching gears and spending time on the prototype is something that we are able to handle. However, in hindsight there were quite some challenges in combining datasets with two source systems and each having their own data model. It took careful collaboration from Architects, Data Stewards, Data Analysts, Data Engineers and our business. Ultimately being able to provide our business requestors with the right insights that can be used to take data-driven actions. Snowflake made the process faster and easier and the termijnbedrag data product is now available for reuse in our nice and shiny Cloud Datawarehouse.