SPF (Sender Policy Framework) has been around for a long time and enjoys a rich history reaching back to 1997, with the SPF Project itself starting in June 2003. (a timeline history of SPF can be found here). However, there are some challenges with SPF’s feedback and it is often misunderstood. We take a look at these issues, and at the barriers that are keeping SPF from being better
A notable issue with SPF is that the people deploying it have to do so with very little information from the outside world. SPF lacks functionality similar to DMARC’s feedback mechanisms (in the form of RUA and RUF reporting). Without reporting from the outside world, SPF records have to be constructed and maintained using only data from an administrators immediate knowledge.
In the pre-DMARC era, people first made SPF records based on their own knowledge of their own infrastructure. SPF records published may or may not have been very accurate in terms of authorizing all legitimate pieces of infrastructure sending on behalf of a domain, but those SPF records at least authorized the infrastructure that the administrator knew about. From the perspective of the people and machines processing inbound email, the inaccurate SPF records meant that decisions on whether or not to deliver email could not be based solely on SPF processing.
Over time, administrators and other people with access to SPF records would add more infrastructure to existing SPF records. Very rarely would anyone remove anything, as the risk of accidentally disrupting a legitimate email flow was present. No one knew if an existing entry in an SPF block was still in use, and tracking this information down was often impractical and sometimes impossible.
To make SPF administration even worse, the way SPF works and what it checks within an email is often misunderstood. This misunderstanding has caused many articles to be published over the years that provide incorrect guidance to non-technical people who were just trying to get SPF into place. Combined with the fact that there now exists many services that will send email on behalf of someone else’s domain (for example, any newsletter service, hosted mailbox providers like Google Suite or Microsoft Office 365, and SaaS applications), getting a clear understanding of how SPF works and how to maintain SPF records has become a complicated matter.
The end result for the pre-DMARC era was that people published SPF records that were often bloated (authorizing pieces of the Internet that should not be authorized), exceeded built in processing limits of SPF (often seen as the “Too many DNS-lookups” error that has been popularized by dmarcian’s SPF Surveyor tool), and inaccurate (due to confusion on how SPF works).
When DMARC hit the scene in 2012, the picture began to change. For the first time, administrators were provided accurate data regarding how domains are used with Internet-facing email. Administrators were able to check the accuracy of their SPF records against DMARC reporting data. This ability caused a minor revolution in the world of SPF as people began to witness how badly it had been implemented and supported.
The Internet-wide cleanup of bad SPF guidance continues to this day. People still run into the “Too many lookups” error without being given clear guidance on how to resolve them. Third-party senders of email (like newsletters) still lack guidance on how to send email on behalf of others in a way that is friendly to SPF administration and maintenance. DNS administrators still struggle with making SPF easy for non-technical people.
With clear guidance available today, the cleanup of SPF is now largely an effort of education, fixing bad guidance, and adding functionality to remove some of the mysteries that surround SPF.
At dmarcian, we like to think about Internet-scale problems and how to solve them. One way of thinking that has really helped us is to consider what the deployment barriers are to a better Internet. In the case of SPF, some of the barriers we’ve identified are: