At Essent we embarked on a transformative path in software development and API management, transitioning from traditional infrastructure to a monorepo architecture. This shift revolutionized our approach to crafting APIs, overcoming challenges posed by limitations in our legacy software and embracing a more streamlined software delivery. Our journey embodies resilience, innovation, and a focus on APIs as invaluable, customer-centric products.
SOFTWARE DELIVERY AS A CATALYST
In today's dynamic business landscape, successful software projects serve as pivotal contributors to a company's overall performance. Software delivery performance directly impacts operational efficiency, customer satisfaction, and ultimately, the company's bottom line. The ability to deliver high-quality software consistently and swiftly has become synonymous with competitive advantage.
Successful software projects are intrinsically linked to operational performance. When software projects are executed efficiently (meeting deadlines, adhering to quality standards, and addressing customer needs) they invariably enhance operational excellence. These projects streamline workflows, reduce operational redundancies, and create an environment conducive to innovation and growth.
Enterprises striving for software excellence often turn to industry standards and frameworks for guidance. DevOps Research and Assessment (DORA) research has emerged as a beacon, identifying best practices in software development and delivery. DORA's research-backed insights highlight key practices that, when implemented effectively, significantly improve the likelihood of successful software projects.
While designing solutions in the Personalization domain, the identification and adoption of DORA-endorsed best practices quickly became a reference when aiming to enhance software delivery performance. Embracing practices like continuous integration, automated testing, and deployment pipelines not only streamline development cycles but also elevate the quality and reliability of software deliverables. We hoped to foster a culture of continuous improvement and innovation within the organization by implementing these best practices.
In essence, successful software projects aren't just isolated wins; they are integral components of a company's success story. By aligning software delivery performance with best practices identified by DORA, we attempted to pave the way for not just efficient software development but also for enhanced operational performance, driving Essent towards sustained success.
THE PERSONALIZATION TEAM
Essent's journey towards innovation began with a strategic initiative: The Personalization Team. We set the ambitious mission to elevate customer experience and deliver tailored marketing solutions. The team's core objective was clear: to leverage technology to create personalized, real-time energy offerings, and elevate Essent's customer-centric approach within the energy landscape. The ultimate goal was to create an API which outputs the best possible personal offer Essent can make, to maximize the likelihood of our prospect to become a customer or remain a customer of Essent.
When the Personalization team started, we found ourselves in quite an interesting situation. At the heart of our operational infrastructure lay Tibco BusinessWorks. BusinessWorks is an enterprise-grade software platform which allows organizations to create, orchestrate services, capture, and publish events from internal or external applications and technologies. It mainly focuses on the transformation and mapping of XML-based messages, which are processed through a message bus.
However, as technology evolved, Tibco's limitations became evident. While it facilitated low-code processes and operated both on-premise and in our AWS-backed cloud environments, it posed challenges in adapting to modern software engineering practices. The inability to scale efficiently, integrate seamlessly with contemporary methodologies, and align with our Personalization Team's objectives became increasingly evident.
Driven by the aspiration to innovate and enhance customer experiences, we faced uphill battles with the vast amounts of data we needed to process and provide on a daily basis. Implementing real-time, personalized offerings and refining Essent's customer engagement strategies required more than an integration platform, it required software engineering. The limitations like scaling difficulties, integration complexities, and agility impediments slowed down our efforts, prompting a reevaluation of Essent's technological approach.
Essent's goal to make services more personal led to changes in how they work. We will look more closely at the problems the Personalization Team faced and how Essent is strongly moving forward with new technology.
COMBINING LEGACY INFRASTRUCTURE WITH BEST PRACTICES
Essent's legacy landscape, supported by Tibco, was the base for all our operations. However, these legacy systems, while functional, presented roadblocks to delivering great software, fast. The existing way of working became incompatible with industry-recognized best practices essential for software success. The methodologies used with our legacy infrastructure were incompatible with the best practices identified by DORA. The inability to seamlessly apply continuous integration, automated testing, and streamlined continuous deployments within the existing architecture created a conflict between our operational practices and the standards that define successful software projects.
Recognizing the incompatibility between our existing practices and the industry-endorsed best practices, we realized a transformative paradigm shift. A transition that would align Essent's operations with the principles driving successful software projects and operational excellence.
EMBRACING THE MONOREPO
Recognizing the limitations imposed by legacy landscape and the divergence from industry best practices, we needed a method to help teams use these best practices, without having to invest a lot of time for each new project. We also needed a way to easily onboard teams into our way of working, provide knowledge sharing, and reuse as much as possible. The decision to transition to a monorepo stemmed from a series of challenges encountered within our existing development setup. The fragmentation across projects, code duplication, versioning struggles, and dependency management complexities became evident impediments to our quest for efficiency and innovation.
Of course this transition did not happen overnight. we actually began by defining a clear exit strategy: a roadmap to offboard applications from the monorepo. This strategy made sure we never took any steps that irreversibly made us dependent on the monorepo setup, and thus ensured a structured path forward, mitigating potential hurdles in the transition process. This exit strategy also helped us get sufficient buy-in from stakeholders. The next step was picking the monorepo framework (Nx) and the migration of three existing domains into a new repository.
The adoption of the monorepo immediately yielded positive outcomes. The consolidation of projects within a single repository drastically reduced code duplication, streamlined version control, and empowered teams to collaborate seamlessly. This new cooperation set the stage for better efficiency and new ideas. It sparked more innovation.
TOOL ADOPTION FOR ENHANCED DEVELOPMENT: SERVERLESS FRAMEWORK, AWS CDK, AND DECORATORS
As we navigated towards a more efficient and innovative tech landscape, tool adoption became imperative. The Serverless Framework became our go-to solution for API development, enabling rapid deployment, scalability, and efficient utilization of AWS services. Its specialized approach towards API development complemented our quest for agility, allowing teams to focus on code and functionality without the complexities of infrastructure management. The ability to run everything offline also improved the development cycles by cutting out the need to run a deployment between each change.
Simultaneously, the integration of AWS CDK broadened our horizons beyond API development. The CDK empowered Essent to efficiently manage diverse AWS resources—databases, queues, topics—providing a comprehensive solution for our infrastructure needs.
In conjunction with our tool and framework selection, we also introduced decorators. These decorators could add metadata to handler functions and models, which facilitated the seamless generation of serverless.com configurations directly from the codebase. With this addition, adding a new API, backed by a dedicated lambda and AWS API Gateway, became as simple as adding a decorator onto a function. The following sample code, together with the referenced models, is all that would be needed to deploy a new API, including API security, input validation, monitoring and alerting:
The integration of Serverless Framework, CDK, and our own decorators had an immediate impact on Essent's development landscape. It helped us achieve our goal of bringing down development and deployment time of a new API to under 10 minutes. This was a huge improvement compared to the previous record of two weeks. The streamlined deployment processes, coupled with the ability to manage diverse resources efficiently, empowered teams to innovate swiftly and enhance their development experiences.
THE ENERGY CRISIS
Just as we had everything in place, and rolled out our new APIs for personalized offering, something unexpected happened. Due to the ongoing war in Ukraine, and the following effects on the energy markets, we suddenly found ourselves in a very different reality. Energy suddenly became a popular product, and our customers got way more interested in their consumption. We started noticing an increased load on our entire IT landscape. The traffic we had to process during a single day was scaling at a faster rate than the rate by which we could scale our legacy systems. This resulted in an outage which we could only solve by throttling the concurrent sessions. This, of course, was not a sustainable situation. We added a waiting queue in front of our websites and started working on a more permanent solution.
In response to the scalability issues and operational challenges faced during the crisis, a strategic decision was made to rebuild critical Tibco services in our new stack. While initially this transition began as a stabilization measure, the outcomes revealed exponential improvements in system stability with each API migrated, not just for those specific API's, but also for API's that hadn't even been migrated yet.
For the conversion of these services, we adopted an iterative approach, prioritizing critical APIs for conversion based on their impact on operational efficiency and customer experiences. For instance, starting with APIs that caused a lot of internal load and errors, allowed for gradual but impactful changes. Automated Load and Regression Testing became fundamental to our strategy. By setting strict nonfunctional requirements and conducting comprehensive tests, we ensured that each converted API met both the performance and quality standards. The process consisted of a few phases; analysis of the existing service, a rebuild into Typescript, strong test coverage requirements, snapshot tests using offline recorded datasets, regression tests by replaying immutable requests between the different scenario's, and finally a controlled canary release of the new API.
Without the adoption of the monorepo architecture, these Tibco conversion efforts would not have been possible. Bringing projects together made it easier to see the whole picture, efficient code sharing, and streamlined the process. We could now really focus on building the perfect factory line to perform the conversion, which we all deemed a fundamental requirement for orchestrating and supporting the complex conversion process.
FOSTERING A COLLABORATIVE CULTURE
Through this journey, technological transformation extended beyond the realms of code; it's about cultivating a thriving ecosystem built on collaboration, shared knowledge, and a collective pursuit of innovation. The monorepo adoption not only streamlined our technical operations but also redefined our cultural dynamics and day-to-day interactions. In alignment with our goal of enabling new teams to be operational within an hour, the setup process has been meticulously designed for accessibility and ease. Our recommended toolsets, configurations, and standardized processes ensure that newcomers quickly acclimate to our environment.
In order to emphasize sharing and collaboration, we categorized components into distinct sections like Cloud Constructs, Apps, Libraries, and Tools. This classification isn't just organizational; it's an embodiment of our commitment to shared knowledge and code reuse. Within this structure, the Apps and Libraries take center stage. Applications, each with dedicated GitLab (child) pipelines and responsible teams, provide the framework for individualized CI/CD processes, promoting autonomy while upholding essential standards.
The CI/CD pipeline, empowered by Nx's 'affected' check mechanism, stands as a testament to our commitment to efficient and targeted workflows. This mechanism dynamically determines which projects are (indirectly) affected by a certain code change and schedules their jobs only when that is the case. This optimizes testing, linting, and deployments, minimizing disruptions while maximizing productivity.
Maintaining this collaborative culture demands a delicate balance between delivering immediate business value and investing in collective contributions. Our ongoing conversations about allocating time for developers to contribute outside their immediate domains underscore Essent's dedication to nurturing a culture of shared success.
Essent's journey through technological transformation embodies resilience, innovation, and a pursuit of excellence. Our transition from legacy infrastructures to a more agile, collaborative, and innovation-driven landscape stands as a testament to our commitment to enhancing customer experiences and operational efficiency.
The adoption of a monorepo architecture, complemented by strategic tool selection and an emphasis on best practices, has not only addressed immediate challenges but paved the way for sustained innovation. By tackling limitations posed by legacy systems and aligning our operations with industry-endorsed practices, we've laid the groundwork for Essent's future technological success.
Essent's technological journey is a testament to our adaptability, innovation, and commitment to pioneering advancements. With a solid foundation established, we stride forward, embracing change, and continuously redefining what's possible in the dynamic energy sector.
Why Google stores billions of lines of code in a single repository
If you have any lingering questions, do not hesitate to drop a comment below!