How we built DMARC Management using Cloudflare Workers
What are DMARC reports
DMARC stands for Domain-based Message Authentication, Reporting, and Conformance. It’s an email authentication protocol that helps protect against email phishing and spoofing.
When an email is sent, DMARC allows the domain owner to set up a DNS record that specifies which authentication methods, such as SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail), are used to verify the email’s authenticity. When the email fails these authentication checks DMARC instructs the recipient’s email provider on how to handle the message, either by quarantining it or rejecting it outright.
DMARC has become increasingly important in today’s Internet, where email phishing and spoofing attacks are becoming more sophisticated and prevalent. By implementing DMARC, domain owners can protect their brand and their customers from the negative impacts of these attacks, including loss of trust, reputation damage, and financial loss.
In addition to protecting against phishing and spoofing attacks, DMARC also provides reporting capabilities. Domain owners can receive reports on email authentication activity, including which messages passed and failed DMARC checks, as well as where these messages originated from.
DMARC management involves the configuration and maintenance of DMARC policies for a domain. Effective DMARC management requires ongoing monitoring and analysis of email authentication activity, as well as the ability to make adjustments and updates to DMARC policies as needed.
Some key components of effective DMARC management include:
- Setting up DMARC policies: This involves configuring the domain’s DMARC record to specify the appropriate authentication methods and policies for handling messages that fail authentication checks. Here’s what a DMARC DNS record looks like:
v=DMARC1; p=reject; rua=mailto:[email protected]
This specifies that we are going to use DMARC version 1, our policy is to reject emails if they fail the DMARC checks, and the email address to which providers should send DMARC reports.
- Monitoring email authentication activity: DMARC reports are an important tool for domain owners to ensure email security and deliverability, as well as compliance with industry standards and regulations. By regularly monitoring and analyzing DMARC reports, domain owners can identify email threats, optimize email campaigns, and improve overall email authentication.
- Making adjustments as needed: Based on analysis of DMARC reports, domain owners may need to make adjustments to DMARC policies or authentication methods to ensure that email messages are properly authenticated and protected from phishing and spoofing attacks.
- Working with email providers and third-party vendors: Effective DMARC management may require collaboration with email providers and third-party vendors to ensure that DMARC policies are being properly implemented and enforced.
Today we launched DMARC management. This is how we built it.
How we built it
As a leading provider of cloud-based security and performance solutions, we at Cloudflare take a specific approach to test our products. We “dogfood” our own tools and services, which means we use them to run our business. This helps us identify any issues or bugs before they affect our customers.
We use our own products internally, such as Cloudflare Workers, a serverless platform that allows developers to run their code on our global network. Since its launch in 2017, the Workers ecosystem has grown significantly. Today, there are thousands of developers building and deploying applications on the platform. The power of the Workers ecosystem lies in its ability to enable developers to build sophisticated applications that were previously impossible or impractical to run so close to clients. Workers can be used to build APIs, generate dynamic content, optimize images, perform real-time processing, and much more. The possibilities are virtually endless. We used Workers to power services like Radar 2.0, or software packages like Wildebeest.
Recently our Email Routing product joined forces with Workers, enabling processing incoming emails via Workers scripts. As the documentation states: “With Email Workers you can leverage the power of Cloudflare Workers to implement any logic you need to process your emails and create complex rules. These rules determine what happens when you receive an email.” Rules and verified addresses can all be configured via our API.
Here’s how a simple Email Worker looks like:
export default {
async email(message, env, ctx) {
const allowList = ["[email protected]", "[email protected]"];
if (allowList.indexOf(message.headers.get("from")) == -1) {
message.setReject("Address not allowed");
} else {
await message.forward("inbox@corp");
}
}
}Pretty straightforward, right?
With the ability to programmatically process incoming emails in place, it seemed like the perfect way to handle incoming DMARC report emails in a scalable and efficient manner, letting Email Routing and Workers do the heavy lifting of receiving an unbound number of emails from across the globe. A high level description of what we needed is:
- Receive email and extract report
- Publish relevant details to analytics platform
- Store the raw report
Email Workers enable us to do #1 easily. We just need to create a worker with an email() handler. This handler will receive the SMTP envelope elements, a pre-parsed version of the email headers, and a stream to read the entire raw email.
For #2 we can also look into the Workers platform, and we will find the Workers Analytics Engine. We just need to define an appropriate schema, which depends both on what’s present in the reports and the queries we plan to do later. Afterwards we can query the data using either the GraphQL or SQL API.
For #3 we don’t need to look further than our R2 object storage. It is trivial to access R2 from a Worker. After extracting the reports from the email we will store them in R2 for posterity.
We built this as a managed service that you can enable on your zone, and added a dashboard interface for convenience, but in reality all the tools are available for you to deploy your own DMARC reports processor on top of Cloudflare Workers, in your own account, without having to worry about servers, scalability or performance.
Architecture
Email Workers is a feature of our Email Routing product. The Email Routing component runs in all our nodes, so any one of them is able to process incoming mail, which is important because we announce the Email ingress BGP prefix from all our datacenters. Sending emails to an Email Worker is as easy as setting a rule in the Email Routing dashboard.
When the Email Routing component receives an email that matches a rule to be delivered to a Worker, it will contact our internal version of the recently open-sourced workerd runtime, which also runs on all nodes. The RPC schema that governs this interaction is defined in a Capnproto schema, and allows the body of the email to be streamed to Edgeworker as it’s read. If the worker script decides to forward this email, Edgeworker will contact Email Routing using a capability sent in the original request.
jsg::Promise ForwardableEmailMessage::forward(kj::String rcptTo, jsg::Optional<jsg::Ref> maybeHeaders) {
auto req = emailFwdr->forwardEmailRequest();
req.setRcptTo(rcptTo);
auto sendP = req.send().then(
[](capnp::Response res) mutable {
auto result = res.getResponse().getResult();
JSG_REQUIRE(result.isOk(), Error, result.getError());
});
auto& context = IoContext::current();
return context.awaitIo(kj::mv(sendP));
}
In the context of DMARC reports this is how we handle the incoming emails:
- Fetch the recipient of the email being processed, this is the RUA that was used. RUA is a DMARC configuration parameter that indicates where aggregate DMARC processing feedback should be reported pertaining to a certain domain. This recipient can be found in the “to” attribute of the message.
const ruaID = message.to
- Since we handle DMARC reports for an unbounded number of domains, we use Workers KV to store some information about each one and key this information on the RUA. This also lets us know if we should be receiving these reports.
const accountInfoRaw = await env.KV_DMARC_REPORTS.get(dmarc:${ruaID})
- At this point, we want to read the entire email into an arrayBuffer in order to parse it. Depending on the size of the report we may run into the limits of the free Workers plan. If this happens, we recommend that you switch to the Workers Unbound resource model which does not have this issue.
const rawEmail = new Response(message.raw)
const arrayBuffer = await rawEmail.arrayBuffer()
- Parsing the raw email involves, among other things, parsing its MIME parts. There are multiple libraries available that allow one to do this. For example, you could use postal-mime:
const parser = new PostalMime.default()
const email = await parser.parse(arrayBuffer)
- Having parsed the email we now have access to its attachments. These attachments are the DMARC reports themselves and they can be compressed. The first thing we want to do is store them in their compressed form in R2 for long-term storage. They can be useful later on for re-processing or investigating interesting reports. Doing this is as simple as calling put() on the R2 binding. In order to facilitate retrieval later we recommend that you spread the report files across directories based on the current time.
await env.R2_DMARC_REPORTS.put(
`${date.getUTCFullYear()}/${date.getUTCMonth() + 1}/${attachment.filename}`,
attachment.content
)
-
We now need to look into the attachment mime type. The raw form of DMARC reports is XML, but they can be compressed. In this case we need to decompress them first. DMARC reporter files can use multiple compression algorithms. We use the MIME type to know which one to use. For Zlib compressed reports pako can be used while for ZIP compressed reports unzipit is a good choice.
-
Having obtained the raw XML form of the report, fast-xml-parser has worked well for us in parsing them. Here’s how the DMARC report XML looks:
example.com
<[email protected]
http://example.com/dmarc/support
9391651994964116463
1335521200
1335652599
business.example
r
r
none
none
100
192.0.2.1
2
none
fail
pass
business.example
business.example
fail
business.example
pass
- We now have all the data in the report at our fingertips. What we do from here on depends a lot on how we want to present the data. For us, the goal was to display meaningful data extracted from them in our Dashboard. Therefore we needed an Analytics platform where we could push the enriched data. Enter, Workers Analytics Engine. The Analytics engine is perfect for this task since it allows us to send data to it from a worker, and exposes a GraphQL API to interact with the data afterwards. This is how we obtain the data to show in our dashboard.
In the future, we are also considering integrating Queues in the workflow to asynchronously process the report and avoid waiting for the client to complete it.
We managed to implement this project end-to-end relying only on the Workers infrastructure, proving that it’s possible, and advantageous, to build non-trivial apps without having to worry about scalability, performance, storage and security issues.
Open sourcing
As we mentioned before, we built a managed service that you can enable and use, and we will manage it for you. But, everything we did can also be deployed by you, in your account, so that you can manage your own DMARC reports. It’s easy, and free. To help you with that, we are releasing an open-source version of a Worker that processes DMARC reports in the way described above: https://github.com/cloudflare/dmarc-email-worker
If you don’t have a dashboard where to show the data, you can also query the Analytics Engine from a Worker. Or, if you want to store them in a relational database, then there’s D1 to the rescue. The possibilities are endless and we are excited to find out what you’ll build with these tools.
Please contribute, make your own, we’ll be listening.
Final words
We hope that this post has furthered your understanding of the Workers platform. Today Cloudflare takes advantage of this platform to build most of our services, and we think you should too.
Feel free to contribute to our open-source version and show us what you can do with it.
The Email Routing is also working on expanding the Email Workers API more functionally, but that deserves another blog soon.
