A (Brief) History of Spam Filtering and Deliverability
To better understand what email deliverability means today and be able to explain the spam filtering and blocking decisions of ISPs, it helps to take a look at the origins and evolution of email marketing.
Only by understanding the past is it possible to have a real insight into what facilitates success in email marketing today, which aspects are obsolete and why, and which are becoming increasingly important. This can only be answered by looking at the concurrent evolution of email technology & standards, email marketing, email abuse & spam filtering (especially by, but not limited to, ISPs).
So much information is available on the topic of email deliverability that it often is overwhelming. Many recommendations of the past are not applicable anymore, so focusing on what is still relevant is crucial.
This article aims to help you bring your promotional email campaigns up to date rather than simply counting on what has worked in the past. Learn about the fundamental changes that have taken place around spam filtering and email deliverability over the years.
The 1990s – Just send it!
For most consumers the 1990s mark the beginning of popular Internet use.
There are many possible starting dates for “the Internet” but the World Wide Web as we know is usually traced back to 1991 when CERN (the European Organization for Nuclear Research) published a paper known as the New World Wide Web Project. 1996 marked another important milestone – specifically for email – when Hotmail became the first web-based email provider.
As is usually the case with innovations, regulation only emerged some time later. More and more marketers discovered the Internet and email as a cheap and easy way to sell their products.
But in the absence of any form of regulation, this era soon became the ‘Wild West’ of email marketing. Standards that we take for granted today such as sender authentication, ISP relations, whitelisting, etc. were as yet unknown and the attitude of marketers was ‘Just send it!’ Misuse was rife and the dangers and limitations of this approach soon became apparent.
The late 1990s / early 2000s – The postmaster era
With email abuse rising, ISPs had to take measures to avoid cluttering their customers’ mailboxes with unwanted, irrelevant emails. To counter the increasing number of (supposed) spam, ISPs started to use crude filters based on keywords, patterns or special characters. Blacklists began to emerge, listing the IPs of known bad senders.
While partially effective, these measures were still inaccurate and caused many false positives – emails were filtered or blocked which were perfectly fine and actually wanted by recipients. ISPs tried their best to improve matters by guessing what their users actually wanted in their inbox, but early attempts to create appropriate and accurate filters failed spectacularly.
The fact that email spam filtering was so inaccurate led ISPs to offer channels of communication to senders and ESPs. This in turn resulted in whitelisting for legitimate senders, to get around the limitations of the static, content-based spam filters; in other words, the beginning of the “Postmaster departments” at ISPs. This worked pretty well for a while but of course this new way to get senders to deliver emails was not bulletproof. Misuse continued as the ISPs often still didn’t have enough data to prove the claims of their senders to be legitimate and their emails to be solicited. Having good ‘ISP relations’ was crucial as it was still human decisions which determined if your emails were blocked, put in the spam folder or delivered to the inbox.
Mid to late 2000s – Reputation and list maintenance
During the mid-to-late 2000s, more measures were added to prevent unsolicited emails. The next major development in email deliverability included the standardization of reputation scores. The ‘SenderScore’ and other reputation systems emerged at this time. False positives were further reduced by adding sender authentication mechanisms like SPF and SenderID, and following them also Domainkeys and DKIM (Domainkeys Identified Mail).
A sender’s reputation was improved due to proper list maintenance. ISPs assumed (and still do) that legitimate senders would take efforts to clean their contact databases regularly. Hard bounce and complaints clean-up (FBL) have become a standard must-have for sustainable email marketing. Reputation was an important development for the industry as around this time many Postmaster departments at ISPs were becoming increasingly obsolete in favour of more automated solutions which had become necessary due to the huge increases in email volume. In these reputation scores lie the origins of the ‘next big thing’: “Engagement Metrics”
2010 to the present – The shift towards engagement metrics
The recent years have seen important changes to email spam filtering and assessing what recipients actually want to receive. Instead of relying on their own judgement and trying to second-guess the market, email clients have put the power firmly in the hands of their users, by offering more sophisticated features for managing incoming mail. This in turn has given ISPs the opportunity to embrace ‘Big Data’, recording all the inbox actions of all their users to determine what really constitutes “wanted” or “unwanted” emails – the so-called ‘Engagement metrics’. This vast amount of data has helped to get rid of false positives almost entirely.
Of the new features that we now regard as standard mail clients, arguably the most important is the security requirement to download images, since this gives us the best indication of a mail open.
But ISPs now have a wide range of actions they can follow to measure a sender’s engagement metrics, for example:
- Moving an email to spam or trash is a negative reaction of a recipient and reflects badly on the sender
- Keeping an email in a personal folder for a long time or even replying or forwarding shows positive engagement of the contact.
- Adding the sender to the address book is the best indication of all that an email is genuinely wanted.
Note: A simple line at the top of an email asking your customers to add you to their address book is an easy and effective way to guarantee inbox delivery.
Thanks to the ISPs listening so carefully to their users, they now know better that you how your recipients really regard your emails. Today there is no way to convince an ISP that a certain mail stream is wanted if the reactions of recipients show the opposite (low response, complaints, emails immediately deleted or moved to trash, etc.).
If the majority of a sender’s emails are delivered to the spam folder or even being blocked, it can be safely assumed that there is a flaw somewhere along the email marketing workflow, be that in the technical setup (e.g. sender authentication), the registration process (optin), list acquisition, content and/or sending strategy, which is leading to low engagement with subscribers. These flaws can be identified by email deliverability experts, and with the source of the problem identified, senders can easily implement fixes.
With these significant changes, the role of Deliverability experts has changed too. In the beginning this was limited to addressing deliverability problems (mostly too many false positives) at ISPs and/or trying to find a way around filters by tweaking the content and adapting new processes and new technologies meant to disguise problematic mail streams. Today, deliverability means most of all to consult and advise marketers on how to create and maintain transparent, sustainable email programs which are relevant for recipients.
The Future of Email Deliverability
The trend towards sophisticated response measurement will continue and the future is likely to bring more and more areas for email deliverability experts to pay attention to. Constantly reviewing advanced analytics will become a standard requirement and proper segmentation an absolute must, while closely listening to the feedback of recipients to meet their expectations. Those senders who invest in a true dialogue with their audiences will be most successful.
Just as ISPs make use of ‘Big Data’, senders will benefit immensely by joining this development too. All leading solution providers can help facilitate optimal segmentation based your various data sources, and personalise content based on the activity and interests of your subscribers. The tools are available – all that eMarketers needs is a good partner to help them build and execute the right strategy.
For more information on quality email marketing and engagement metrics, you might like to read some of the following: