SPF (Sender Policy Framework) has been around for a long time and enjoys a rich history reaching back to 1997, with the SPF Project¬†itself starting in June 2003. (a timeline history of SPF can be found here).¬†However, there are some challenges with SPF’s feedback and it is often misunderstood. We take a look at these issues, and at the barriers that are keeping SPF from being better

SPF lacks feedback

A notable issue with SPF is that the people deploying it have to do so with very little information from the outside world. SPF lacks functionality similar to DMARC’s feedback mechanisms (in the form of RUA and RUF reporting). Without reporting from the outside world, SPF records have to be constructed and maintained using only data from an administrators immediate knowledge.
In the pre-DMARC era, people first made SPF records based on their own knowledge of their own infrastructure. SPF records published may or may not have been very accurate in terms of authorizing all legitimate pieces of infrastructure sending on behalf of a domain, but those SPF records at least authorized the infrastructure that the administrator knew about. From the perspective of the people and machines processing inbound email, the inaccurate SPF records meant that decisions on whether or not to deliver email could not be based solely on SPF processing.
Over time, administrators and other people with access to SPF records would add more infrastructure to existing SPF records. Very rarely would anyone remove anything, as the risk of accidentally disrupting a legitimate email flow was present. No one knew if an existing entry in an SPF block was still in use, and tracking this information down was often impractical and sometimes impossible.

SPF is misunderstood

To make SPF administration even worse, the way SPF works and what it checks within an email is often misunderstood. This misunderstanding has caused many articles to be published over the years that provide incorrect guidance to non-technical people who were just trying to get SPF into place. Combined with the fact that there now exists many services that will send email on behalf of someone else’s domain (for example, any newsletter service, hosted mailbox providers like Google Suite or Microsoft Office 365, and SaaS applications), getting a clear understanding of how SPF works and how to maintain SPF records has become a complicated matter.
The end result for the pre-DMARC era was that people published SPF records that were often bloated (authorizing pieces of the Internet that should not be authorized), exceeded built in processing limits of SPF (often seen as the “Too many DNS-lookups” error that has been popularized by dmarcian’s SPF Surveyor tool), and inaccurate (due to confusion on how SPF works).
When DMARC hit the scene in 2012, the picture began to change. For the first time, administrators were provided accurate data regarding how domains are used with Internet-facing email. Administrators were able to check the accuracy of their SPF records against DMARC reporting data. This ability caused a minor revolution in the world of SPF as people began to witness how badly it had been implemented and supported.
The Internet-wide cleanup of bad SPF guidance continues to this day. People still run into the “Too many lookups” error without being given clear guidance on how to resolve them. Third-party senders of email (like newsletters) still lack guidance on how to send email on behalf of others in a way that is friendly to SPF administration and maintenance. DNS administrators still struggle with making SPF easy for non-technical people.

Barriers to better SPF

With clear guidance available today, the cleanup of SPF is now largely an effort of education, fixing bad guidance, and adding functionality to remove some of the mysteries that surround SPF.
At dmarcian, we like to think about Internet-scale problems and how to solve them. One way of thinking that has really helped us is to consider what the deployment barriers are to a better Internet. In the case of SPF, some of the barriers we’ve identified are:
  • Education: People have been given incorrect information from otherwise authoritative sources on how SPF works. It takes a lot of work to “correct the record”.
  • Vested Interests: Commercial entities exist that could be threatened by a fully adopted and maintained Internet-wide SPF layer. Companies that thrive on secret email filtering sauce would have to content with less of a need for secret sauces. Companies that benefit from the existing difficulty in understanding and maintaining SPF would see a direct threat to their model of business.
  • DNS Management: Most DNS software requires a level of expertise that escapes most people. Just because someone can purchase an Internet domain does not mean they know about TXT records and SPF constructs.
  • Legacy Infrastructure: People dealing with SPF today often do not know that they can use DMARC to inform their SPF activities. When someone new to SPF is presented with a problematic legacy going back 15+ years, they might find the task daunting.
SPF has been around for a long time. We at dmarcian have been around equally as long, and we’re using DMARC to make SPF easy. Before wasting time on the open Internet trying to figure out SPF, give our resources a try.