Why SPF Is So Funky in Today’s Modern World
Sender Policy Framework (SPF) has been around for a long time and enjoys a rich history reaching back to 1997, with the SPF Project itself starting in June 2003. However, there are some challenges with the way SPF works, leading to confusion, misunderstanding, and misleading advice. In this article we take a look at these issues and at the barriers that are keeping SPF from being better.
SPF lacks feedback
A notable issue with SPF is that the people deploying it have to do so with very little information from the outside world. SPF lacks functionality similar to DMARC’s RUA and RUF reporting feedback mechanisms. Without reporting from the outside world, SPF records have to be constructed and maintained using only data from an administrator’s immediate knowledge.
In the pre-DMARC era, people first made SPF records based on their own knowledge of their own infrastructure. SPF records published may or may not have been very accurate in terms of authorizing all legitimate pieces of infrastructure sending on behalf of a domain, but those SPF records at least authorized the infrastructure that the administrator knew about. From the perspective of the people and machines processing inbound email, inaccurate SPF records meant that decisions on whether or not to deliver email could not be based solely on SPF processing.
Over time, administrators and other people with access to SPF records would add more infrastructure to existing SPF records. Very rarely would anyone remove anything, as the risk of accidentally disrupting a legitimate email flow was present. No one knew if an existing entry in an SPF block was still in use, and tracking down this information was often impractical and sometimes impossible.
SPF – for Email Operators, by Email Operators
SPF records—lists of authorized servers—exist as text resources of an internet domain. This means that email operators differentiate themselves and their infrastructure using different internet domains.
Email operators that share the same domain effectively turn an internet domain into an undifferentiated and shared pool of reputation that forces email receivers to treat all email from the domain the same.
Spammy marketing email remains undifferentiated from human-to-human corporate email, which in turn remains undifferentiated from the email sent by a compromised vendor.
Email operators use domain-based SPF to identify/differentiate themselves and their infrastructure.
SPF Checks the Reverse-path of Email
The way SPF works and what it checks within an email is often misunderstood. This confusion has led to many articles being published over the years that provide incorrect guidance to non-technical people who were just trying to get SPF into place.
Here’s how it works: When a server delivers email, the sending server shares an email address where error reports and notifications regarding the email can be sent. This address is called the reverse-path and is used primarily by email operators.
The receiving server uses the domain of the reverse-path to look up an SPF record. If that SPF record lists the sending server, then the reverse-path is legitimate.
Today, the reverse-path goes by many names: bounce address, RFC5321.MailFrom, “SMTP MAIL FROM,” Return-Path, envelope address, and even more. These numerous naming conventions can be quite confusing.
Another common misunderstanding is mistaking the “From:” address—the visible address within an email—for the reverse-path. This inaccuracy causes people to add infrastructure to their SPF records based on the “From:” address instead of the reverse-path.
The result is that SPF records become incredibly large, often authorizing huge sections of the internet to send on behalf of an internet domain and resulting in a bloated SPF record. Then, people seek solutions to manage incredibly large SPF records—all based on a misunderstanding of how SPF works.
Combined with the fact that there now exists many services that will send email on behalf of someone else’s domain (for example, a newsletter service, hosted mailbox providers like Google Suite or Microsoft Office 365, and SaaS applications), getting a clear understanding of how SPF works and how to maintain SPF records has become a complicated matter.
The end result for the pre-DMARC era was that people published SPF records that were often inflated by authorizing pieces of the internet that should not be authorized and exceeded the processing limits of SPF. This error is often seen as the “Too many DNS-lookups” notification popularized by dmarcian’s SPF Surveyor; it’s also inaccurate because of the confusion on how SPF works.
DMARC Changes How People View and Deploy SPF
When DMARC hit the scene in 2012, the picture began to change. For the first time, administrators were provided accurate data regarding how domains are used with internet-facing email. With a DMARC record in place, administrators were able to check the accuracy of their SPF records against DMARC reporting data and receive guidance on deploying SPF properly. This ability caused a minor revolution in the world of SPF as people began to witness how badly it had been implemented and supported.
DMARC also ushered in the “domain alignment” requirement. Alignment forces email operators to identify themselves using a sub-domain so that the email operator can both identify themselves and the infrastructure they’re managing. DMARC’s requirement changes SPF in a significant way, leading to deployment challenges and SPF refinement.
The internet-wide cleanup of bad SPF guidance continues to this day. People still run into the “Too many lookups” error without being given clear guidance on how to resolve them. Third-party senders of email still lack guidance on how to send email on behalf of others in a way that is friendly to SPF administration and maintenance. And DNS administrators still struggle with making SPF easy for non-technical people.
Barriers to Better SPF
With clear guidance available today, the cleanup of SPF is now largely an effort of providing education, offering better guidance, and adding functionality to remove some of the mysteries that surround SPF.
At dmarcian, we like to think about internet-scale problems and how to solve them. One way of thinking that has really helped us is to consider what the deployment barriers are to a better internet.
In the case of SPF, some of the barriers we’ve identified are:
- Education: People have been given incorrect information from otherwise authoritative sources on how SPF works. It takes a lot of work to “correct the record.”
- Vested Interests: Commercial entities exist that could be threatened by a fully adopted and maintained internet-wide SPF layer. Companies that thrive on secret email filtering sauce would have to contend with less of a need for secret sauces. Companies that benefit from the existing difficulty in understanding and maintaining SPF would see a direct threat to their model of business.
- DNS Management: Most DNS software requires a level of expertise that escapes most people. Just because someone can purchase an internet domain does not mean they know about TXT records and SPF constructs.
- Legacy Infrastructure: People dealing with SPF today often do not know that they can use DMARC to inform their SPF activities. When someone new to SPF is presented with a problematic legacy going back 15+ years, they might find the task daunting.
SPF has been around for a long time. dmarcian has been around since the DMARC specification was published, and we’re using DMARC to make SPF easier for people doing the work to make email better.
If you need help figuring out SPF, give our resources a try.
Want to continue the conversation? Head over to the dmarcian Forum
