INTRODUCTION

In today's world, keeping your data safe and getting consents from users is really important. That's why Essent is excited to share our latest progress in making the digital world safer and more transparent. We've created a new system for managing user consents, which is a big step forward.

Essent knows that protecting user privacy isn't just a rule, it's about building trust online. Our new system is designed to make sure we handle consents in the best way possible. For example, we might ask if it's okay to send someone a newsletter. There are also more complicated permissions, like getting basic information about a customer's energy usage, and any related costs.

WHERE WE CAME FROM

Not long ago, we managed consents in a scattered way. Each team or system handled permissions for their own part of our applications.

In the image above you see a simplistic scenario sketched out where a user wants to request information about their consents. It’s going to make request to several services (A,B,C) that sometimes also lead to those services getting the information from another downstream service (X,Y). A service can be the place where the consent is stored or where the consent is being used.

Our old way of handling consents seemed practical at first, but it caused big problems. Consents were scattered across different systems, making it hard to keep track of everything users agreed to. As our apps grew and teams multiplied, it became even harder to manage.

So, we decided to make a big change. We wanted to centralize consent management, creating one clear system for everything users agree to. This would make things easier for users and ensure we met the highest privacy standards. Besides that, we needed to make sure this system could handle future changes too.

TRANSFORMING CONSENT MANAGEMENT WITH MODERNITY AND EFFICIENCY

Now, let's picture how things will work for the user. Instead of connecting with many services, they just connect with one (A). This main service sends the consent updates to a central data streaming point (K). Other services (X, Y) can then listen to this point and receive the updates.

Now that we understand the basic flow, let's explore the technologies behind our new consent management platform. Think of this platform like a well-coordinated orchestra, where each part works together smoothly. One important component is serverless technology, which allows us to operate without directly managing servers, reducing overhead and focusing more on user experience.

What makes our platform unique is its integration of AWS and Confluent Kafka. AWS provides reliable cloud computing services, while Confluent Kafka handles near real-time data streaming and processing efficiently.

Together, these technologies create a flexible and scalable ecosystem, ensuring our platform meets the demands of modern data governance. Additionally, we've centralized our audit trails, simplifying tracking and monitoring processes. For more details on Confluent Kafka, check out our articles: The Kafka serverless journey - annotations on lambda functions to automate Kafka connector setup & Essent and web data processing using Kafka.

Take a look at the core of our consent management platform. In the upcoming image, each number represents a key part of our infrastructure, with descriptions provided below.

1. Frontend applications: This is where users start the chain by opening their app or website and adjusting their consents.
2. General Account API Gateway: Serving as the gateway for external requests, our API Gateway ensures secure access to our platform. It's a temporary solution until the transition to CRA 2.0 is complete. For more details, refer to Cloud Reference Architecture 2.0.
3. API Gateway: This is the entry point for requests to our AWS account. It houses two Lambda APIs: GetConsents and PutConsents, used for fetching and updating consent information.
4. DynamoDB Table: At the core of our data storage is a DynamoDB table, where consent records are stored and managed. Transactions are also stored here to track when and how consents change. This highly scalable and low-latency database ensures fast access to consent data.
5. DynamoDB Stream: To track changes in real-time, we've set up a DynamoDB Stream linked to our consent table. Any modifications trigger this stream, signaling the need for immediate action. We utilize an outbox pattern instead of directly publishing to Confluent Kafka from here. We’ve also introduced this outbox pattern from point 6 to 7 for the missing functionality of a replayable dead-letter queue on the DynamoDB Stream.
6. ConsentChanged Lambda: When triggered by the DynamoDB Stream, this Lambda captures and prepares consent changes for further processing.
7. SQS Queue: Acting as a buffer, the SQS (Simple Queue Service) queue awaits incoming consent changes. It includes a dead-letter queue setup for replayability and error alerts.
8. PublishConsentChange Lambda: This Lambda fetches queued changes from the SQS queue and prepares and sends them for publishing.
9. Confluent Kafka Topic: Using Confluent Kafka, our platform communicates consent changes via a topic, enabling near real-time streaming and processing of updates.
10. Confluent Kafka Command Topic: In addition to API calls, we use a command-driven approach. Services can publish commands to this topic to trigger consent modifications directly, especially useful for process-related consents. Like a customer leaving meaning we want to revoke all their consents.
11. Listener for Command Topic: A dedicated listener monitors the command topic, intercepting incoming commands. It then orchestrates the necessary actions, updating the DynamoDB table accordingly.
    
    

CONCLUSION

Our consent management platform showcases efficiency and innovation by seamlessly combining advanced technologies. From handling API requests to providing almost instant updates through DynamoDB Streams and Kafka topics, every part is carefully designed for reliability and scalability. As we move forward, our platform remains committed to maintaining data privacy and transparency, creating a safe digital space for all users.