Concluding the Experiment: SPF Flattening

“SPF Flattening” was invented by dmarcian as part of the initial release of the SPF Surveyor. For many years the functionality was flagged as “experimental.” Today, we’re concluding the experiment and sharing what we’ve learned.

Sender Policy Framework (SPF) Background

Email operators use SPF to identify themselves and their infrastructure when sending email across the internet. SPF was originally created by email operators around 2000 in response to Joe Job Attacks. At that time, spammers were making victims out of legitimate email operators by pretending to be them while sending spam. Legitimate operators were blamed for sending spam simply because there was no easy way to identify email operators and their infrastructure.

Email operators publish SPF records to describe their email infrastructure. These records are specially formatted strings of text that live in the DNS. When receiving email, an email server looks at the domain of the sending operator’s email address (known as the reverse path, bounce address, envelope from, MAIL FROM and more) to retrieve the domain’s SPF record.

If the email infrastructure described by the SPF record is the same infrastructure that is attempting to deliver email, then the receiver has successfully identified the email sending operator and their infrastructure.

Today’s SPF Challenges

SPF records face three distinct maintenance challenges:

1. SPF has been around for a very long time.
2. How SPF works is largely misunderstood, even by experienced IT technicians.
3. To interoperate with DMARC, SPF requires email operators to be associated with the domain that end users see in an email’s “From:” header.

SPF has been around for 20+ years. For many years, people did their best to list their email-sending infrastructure. Servers would be added to SPF records but almost never removed as there was no easy way to determine if a server was still in use.

To make matters worse, how SPF works is largely misunderstood. Email operators are distinctly different from the people that write and send email, but a common misunderstanding is that SPF checks the “From:” domain of an email instead of the domain of the operator, known as the reverse-path.

This misunderstanding is widespread and causes SPF records to become bloated with infrastructure that is not actually used to send email by an operator. In the worst cases, email vendors identify themselves using the domain of their clients, causing the client to assume the role of “email operator” even though the client is specifically paying the email vendor to deal with email operations.

Before DMARC came along, email operators could identify themselves using a reverse-path that pointed to their own operation. There was no need for email operators to be associated with the “From:” domain that people see in their email applications.

However, SPF is one of the two technologies that can be used to achieve DMARC compliance; the other is DKIM. In order to use SPF to build DMARC compliance, DMARC requires the reverse-path of the email operator to be the same domain or a subdomain of what people see in the “From:” header of an email. To interoperate with DMARC, too many email vendors end up identifying themselves using the top-level domain of their clients, causing additional SPF record bloat as described above.

“Too Many DNS Lookups” as a Symptom

When SPF’s challenges are added up, SPF records can be inaccurate, unnecessarily long to the point of running into SPF’s built-in processing limits, and expose an organization to risk due to overly-broad and uncontrolled authorization.

dmarcian’s SPF Surveyor was the first public tool to highlight SPF’s built-in processing limits—commonly referred to as the “Too many DNS lookups” problem. Simply put, the SPF specification tries to protect receiving email servers from SPF records that would otherwise force the receiving server to consume too many resources while attempting to process an SPF record and deliver email.

Following are reasons that SPF records hit the Too Many DNS Lookups problem:

• people add things to SPF records without ever removing entries,
• people are told to “include:” infrastructure into their top-level SPF records until SPF’s built-in limits are reached, and
• the need to interoperate with DMARC can cause SPF records to become extremely bloated.

At dmarcian, we’ve come to understand large and unwieldy SPF records—most commonly experienced as the “Too Many DNS Lookups” error—as a symptom of larger issues: misunderstanding of how SPF works, incomplete vendor management practices when it comes to vendors that send email on behalf of an organization, and a lack of best-practice email segmentation within an organization.

The Dangers of SPF Flattening

SPF Flattening attempts to work-around the “Too Many DNS Lookups” error without addressing the underlying issues that caused the error in the first place. Making the error go away without paying attention to the underlying issues can lead to a bad place. IT staff at an organization unwittingly ends up being responsible for the behavior and infrastructure of all vendors/email-operators that send email using the organization’s top-level domain:

• From a reputational perspective, receivers cannot easily differentiate between different types of emails. For example, invoices and newsletters share the same domain, meaning that spammy newsletters can cause invoices to end up in spam folders, simply because receivers see the email coming from the same domain.
• From an operational perspective, vendors do not receive the signals they need to effectively manage the email they’re sending on behalf of the organization. For example, if a vendor sends an email to a bad recipient that causes a bounce message to be generated, that bounce message will flow to the IT staff and not to the vendor. As a rule, IT staff do not forward these types of signals to their vendors, leaving the vendors to “fly blind” and gradually negatively impact the reputation of the organization (see above).
• From a security perspective, over-authorization due to bloated SPF records is another way of saying “unnecessary entries in SPF records create attack surfaces.” If an adversary is able to break into (or simply rent) any piece of infrastructure that is listed in an SPF record, that adversary is able to send DMARC-compliant email, effectively circumventing controls and using those controls to hijack the reputation of the organization to deliver fraud.

Given the dominant role of email in cyber attacks, working around SPF’s symptomatic error message without addressing the underlying issues goes against the mission of Information Security and Risk Management professionals.

dmarcian has resisted the urge to turn SPF Flattening into a product or service, mainly because the need for SPF Flattening goes away when an organization adopts best practices and segments their vendors and email streams. Organizations that move in this direction end up with better controls, less attack surface, operational efficiency, and reputational resilience that comes from compartmentalizing vendors and limiting any spill-over effects of a potential cyber incident.

The End of the SPF Flattening Experiment

dmarcian has learned that some IT teams are constrained in their ability to affect change in areas that go beyond servicing of tickets or supporting existing groups. These teams are not chartered to improve business processes or to address security or risk management issues, placing the rollout of dmarcian-style best practices (which can span groups) out of reach.

For these teams, dmarcian views “SPF Flattening” as a stop-gap solution until the organization can align itself to adopt best practices which can alleviate the current load/issues that IT teams experience. The “stop-gap” part recognizes that IT teams in this situation need time without SPF errors in order to collaborate with other groups to roll out more substantial and sustainable changes.

The danger that dmarcian sees in these situations is that IT teams leave the stop-gap solution in place and never get around to fixing the underlying issues (which they are arguably not chartered to address).

Use DMARC Data to Inform Decisions

For over a decade, dmarcian has recommended using DMARC data to assess the current state of an organization, identify steps necessary to build DMARC compliance, and to inform project planning with a goal of implementing best practices.

These steps help everyone—including constrained IT teams—make informed decisions.

Before taking the extraordinary steps of using SPF Flattening to work around SPF’s built-in limits, use DMARC data and dmarcian’s platform to gain insight into the situation.

If you need help with managing your SPF records, get in touch with us and feel free to use our DMARC resources, which are available to everyone, whether you have a dmarcian account or not.