As a software application or system expands, it becomes increasingly important to have a method for tracing and creating a timeline of how the different components of your system interact.

In this article we will discuss the conceptual differences between Tracing, Logging and Auditing. Then we will zoom in on Tracing, more specifically on what options you have for implementing it using Amazon AWS. And we'll close by presenting how we are implementing it at Essent.

TRACING VS LOGGING VS AUDITING

Since these terms often cause confusion, let's begin by clarifying these fundamental concepts.

Tracing is the process of following a program’s flow and data progression. It encompasses a much wider, continuous view of an application. The goal of tracing is to track the flow of data and control through a system. Tracing can be a noisier activity than logging, as it represents a single user’s journey through an entire app stack.

Logging is the process of recording events that occur in a system. It is used to track error reporting and related data in a centralized way. Log files can show any discrete event within an application or system, such as a failure, an error or a state transformation. When something inevitably goes wrong, such transformations in state help indicate which change caused an error. Logging is primarily deployed and used by system administrators on the operational level, intentionally providing a high-level view.

Auditing is the process of recording and reviewing the activity of a system. It is used to track changes to data and system configurations, as well as to monitor user activity. Audit trails record data lifecycle changes and give wide context to modifications. Logs give detailed information on each event. Logs focus on user activities besides system events. Audit trails focus on contextual information besides data changes.

TRACING WITH AMAZON AWS

All methods track the behavior of software systems, but they serve different purposes. The focus of this article will be on Tracing, in the scope of Amazon AWS.

In a nutshell, the idea is to propagate a TraceID (think of it as a flow identifier) between all the components in certain application paths and export all this info to an analysis tool like AWS X-Ray.

When we speak about Tracing, it’s important to define a standard way for components to report their data and control flow, so you can easily capture and visualize it. One of the most famous frameworks for this purpose is OpenTelemetry. From the official website, we read “OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior”.

In AWS, OpenTelemetry is available as a distribution. This distribution allows you to effortlessly send your traces to various AWS monitoring services, with AWS X-Ray being the most notable among them. Using X-Ray, you can view the entire workflow of your application and focus on specific components for detailed insights. Additionally, this distribution can be seamlessly integrated with a broad spectrum of AWS services. Some of the popular services include:

• AWS Lambda: AWS managed Lambda layers for ADOT provides a plug-and-play user experience by automatically instrumenting a Lambda function, packaging OpenTelemetry together with an out-of-the-box configuration for AWS Lambda and X-Ray, in an easy-to-setup layer.
• AWS ECS: you can use a simple sidecar service linked to your tasks (either Fargate or EC2) which will be used to propagate your data.
• AWS EC2: via AWS X-Ray client.
• API Gateway: it can easily pass your TraceID and other relevant headers to the next components.

Now that we've introduced the concepts and gave a high-level overview of how we can achieve Tracing with Amazon AWS, we shall discuss how we are implementing it at Essent.

ESSENT'S IMPLEMENTATION

At Essent, it’s crucial to keep an eye on the workflow and the performance of our systems.
In this final part of the post, we give you a high-level idea on how we are implementing Tracing across our landscape.

As you can see in the image below, a centralized audit account is used as a receiver where all the tracing data will be forwarded to.


Our customers, using a browser or mobile app, perform requests that arrive at our backend. OpenTelemetry is used as protocol and distributor for the tracing data. Those requests are forwarded to an API gateway for integration with our internal backend systems, but also passed to AWS X-Ray, which we then use to visualize and drill down onto the traces.

For added convenience, X-Ray exports the traces to our dashboarding tool, Prometheus. This enables everyone in the organization to easily search for and visualize the specific information they need.

Keep in mind the image depicts a simplified view of our implementation, as elements like Security are not displayed.

The solution we've described is currently in the implementation phase. Essent's Cloud Platform team is developing the framework that will be utilized by our stream-aligned teams to trace our customers' journeys across the entire landscape.

If you are interested in understanding our implementation in more detail, feel free to reach out in the comments section below.

And if you're interested in being a part of the creation and implementation of such initiatives, please take a look at our current job openings.

Thank you for reading, see you soon!