INTRODUCTION: THE CATALYST FOR CHANGE
In software development (or any job for that matter), some of the most impactful innovations emerge from real challenges. Several months ago, our team faced a series of incidents that highlighted gaps in our merge request review process. While these incidents were successfully resolved, they sparked an important realization: we needed to fundamentally strengthen our code review workflow.
THE JOURNEY TO AI-POWERED CODE REVIEWS
Our initial strategy followed traditional best practices - implementing multiple approval gates, standardizing templates, and establishing stricter merge conditions.
During this research I also discovered there are some solutions that use Machine Learning or Artificial Intelligence to review code for potential defects. One of these examples is Amazon CodeGuru Reviewer. Unfortunately, we couldn't give that a try, because it's mostly focused on Java and Python. However, the concept seemed very interesting. And how hard could it be to build this ourselves (right?)
After exploring various options, I turned my attention to AWS Bedrock - a fully managed service that provides access to multiple foundation models from leading AI companies. What made Bedrock particularly interesting was not just its impressive lineup of models (including Claude 3.5 Sonnet, which is the default model for MergeMate ), but its built-in features for security, privacy, and responsible AI use.
But before diving into the technical implementation, it's worth understanding what makes foundation models so powerful for code review. These models are essentially massive neural networks trained on vast amounts of data, including code repositories. Think of them as having absorbed patterns and best practices from millions of lines of code, enabling them to identify potential issues and suggest improvements.
The path from concept to production-ready solution wasn't straightforward. Like many projects, MergeMate evolved through several iterations, each teaching us valuable lessons about scalability, reliability, and developer experience.
EVOLUTION THROUGH LEARNING
Our first implementation took the most straightforward approach - a script running in our CI/CD pipeline. While this seemed logical at first, it quickly revealed several challenges that would shape our future design decisions:
- With around 300 repositories in our team, each with its unique pipeline configuration, maintaining consistent code review coverage became a significant challenge
- Pipeline failures, whether from script errors or API issues, would block developers from merging their changes, causing unnecessary delays
- The solution lacked centralized monitoring, making it difficult to track usage patterns and troubleshoot issues
- Cost tracking per team was virtually impossible, limiting our ability to optimize and scale the service
- Updates to the script required changes across multiple repositories
These challenges made it clear that MergeMate needed a more robust, centralized architecture. This led me to pivot towards a fully serverless architecture that would address these pain points while providing better scalability and maintainability.
MERGEMATE IN ACTION: A USER'S PERSPECTIVE
Let's walk through how MergeMate works from a user's perspective, from setup to receiving AI-powered code reviews.
Getting started with MergeMate is straightforward – I designed it to be "set and forget" in under 5 minutes. Simply visit your GitLab project settings, add a webhook that points to the API Gateway endpoint, include your API key, and you're ready to go. Want MergeMate to focus on specific aspects of your code? You can add optional custom instructions like "Always check for TypeScript best practices" or "Focus on security patterns."
From there, MergeMate automatically jumps into action whenever a merge request is created or updated. Here's what you'll see:
First, MergeMate will greet you with a welcome message:
While MergeMate provides valuable code reviews, every developer has their own preferences. That's why opting out is as simple as replying with an emoji - MergeMate will remember the choice and stay quiet on future merge requests.
Let's look at an example of MergeMate in action. Here, it caught several critical security issues in the code changes - from dangerous system commands to hardcoded credentials in version control. For each issue, MergeMate provides both an explanation of the problem and the specific code needed to fix it, like suggesting AWS Secrets Manager for credential management. This kind of proactive feedback helps catch potential issues early, before they reach production.
TECHNICAL DEEP DIVE
MergeMate's architecture is built using AWS serverless services. When a merge request comes in, an AWS Step Function workflow coordinates different Lambda functions to handle tasks like receiving the webhook, analyzing the code with AWS Bedrock, and creating change requests in ServiceNow.
This serverless approach means the system automatically scales up when more teams start using it, with no manual work needed. When usage is low, costs stay minimal since you only pay for the reviews being processed. There's no maintenance overhead - no servers to manage or update. Plus, built-in monitoring and alarms make it easy to spot and fix any issues quickly.
IMPACT AND RESULTS
Since releasing MergeMate as a building block, the numbers speak for themselves:
- 15+ teams actively using the product
- Over 13.000 merge requests reviewed
- Supporting more than 170 different developers
- Reviews spanning 100+ repositories
- Total code reviewed: 100M tokens
And the best part? Each review costs just 2 cents, making it an incredibly cost-effective addition to our development workflow.
BEYOND CODE REVIEW
Mergemate has grown to include features that help with other parts of our development workflow. An example is the ServiceNow integration for managing change requests.
When your merge request includes production changes, simply label it with your team's service name, and MergeMate takes care of the rest:
- Creates an informational change request automatically
- Links the change request to your merge request
- Handles the closure once the changes are deployed
- Manages the entire lifecycle without developer intervention
We've also made MergeMate highly customizable. You can select which AWS Bedrock model to use for your reviews, add specific instructions for code analysis, or even inject some fun into your code reviews. Who says code reviews can't be fun? Here's what happens when you tell MergeMate to embrace its inner pirate:
KEY TAKEAWAYS AND LOOKING FORWARD
When I started MergeMate, I set out to solve a specific problem – innovating and improving our merge request workflow to prevent incidents. After seeing its effectiveness, other teams expressed interest, leading to inner sourcing the solution. Now, MergeMate helps teams across Essent find and resolve potential issues daily.
The numbers and feedback show that teams find real value in MergeMate. By automating code reviews and handling routine tasks like change requests, it helps developers stay focused on what matters - writing and shipping code.
Looking ahead, I’m excited to see how teams will continue to use and customize MergeMate to fit their unique needs. Whether it's through specialized review instructions, different AI models, or yes, even more pirate-speak, MergeMate is evolving alongside our development practices.
DISCLAIMER ON WORKING WITH LARGE LANGUAGE MODELS (LLM)
Even though we are really excited about MergeMate, we believe in being transparent about working with LLMs:
- LLMs may sometimes generate plausible-sounding yet incorrect information.
- Their responses will be inconsistent across different runs (non-deterministic behavior).
- Their knowledge is limited to their training data cutoff date.
Please always keep these limitations in mind.
CLOSING NOTES
A special thanks to Ali Habib and Felipe Macias Diaz for their contributions to this project. MergeMate wouldn't be where it is today without their help. Whether you're just starting with MergeMate or you're one of our regular users, we hope it continues to make your development process a little bit easier (and maybe even more fun with the occasional pirate review!).