Select a page

15 Pivotal Elements in SEO History #IP15

In an industry now roughly two decades old, it’s very difficult to isolate a generally agreed-up list of the most pivotal moments, ideas, people, and concepts. As a result, to honor IP’s 15th anniversary, what follows is my own list of influential topics and a brief note about how and why each one is important. I’ve probably missed some biggies or included some events or ideas whose validity you’ll question. As always, feel free to support or refute my choices in the comments section.

1. Crawling Search Engines. Today, “search engines” are generally synonymous with crawlers, but that hasn’t always been the case. Prior to the development of really top-grade crawlers, discovering new sites was much more reliant on directories and, to an unfortunate degree, big-media “site of the day”-type posts. With the development of crawlers such as WebCrawler, Lycos, and Inktomi, engines now had much more appeal due to the ability to scale their index size automatically and with less reliance on human editors.

2. Yahoo!. Incorporated in 1995, Yahoo’s popularity rose with the internet population’s need for a qualified way of finding content in an increasingly anarchic internet (I was going to say “anarchic web,” but Yahoo’s directory was also full of non-HTTP resources such as Gopher, WAIS, Usenet, and so on.) While it went through numerous changes about whom it powered, who powered it, and its change from a directory to a crawler-based search engine, it was the epitome of successful, pre-bubble search portals.

Yahoo's home page circa 1994

Yahoo’s home page, circa 1994. Photo credit.

3. Google. You can’t have an SEO article that excludes Google as a pivotal piece of history. Founded in 1998, it quickly became the darling of tech-savvy searchers due to its clean interface and, more important, the relevance of its results. Today, Google is synonymous with search, and it has stayed pretty true to its model of simple relevance, even though it’s hit and missed on various ventures outside the traditional SERP.

4. Danny Sullivan. In 1996, Danny published “A Webmaster’s Guide to Search Engines,” which later became the iconic Search Engine Watch. In 1999, he put SEO on the trade show map with the first Search Engine Strategies conference. These events helped facilitate engine-webmaster communications better than any other avenue at the time. At his core, Danny is a journalist and an objective one. Any legitimacy the SEO industry has (don’t go there) first began with Danny Sullivan.

5. PageRank and The Google Toolbar. Far beyond simply allowing users to search from the toolbar, the Google Toolbar featured the famous green PageRank indicator, the most visible (although overly superficial) way of discerning Google’s “opinion” of the URL your browser was currently visiting. This resulted in further PageRank hysteria, even though the toolbar PageRank (tbPR)  values were only a snapshot. Due to updates that occurred very sporadically (about once a quarter or so), tbPR values were almost always obsolete and therefore rarely accurate, but unfortunately, it didn’t stop many sites from using PageRank as a marketing KPI.

6. Adwords. Google wasn’t the first search engine to carry advertisements, but the birth of Adwords did answer the crucial question of how Google was, in fact, going to make any money. What started out as a CPM model soon turned to auction-based placement. Google (and other engines) have come under fire throughout the years for the poor delineation between paid and free results, and they continue testing ways of fulfilling the FTC’s mandate of differentiating between paid and natural results for users, all the while trying to convince users that the differences aren’t all that great.

7. Google “Florida” Update. I could write a Top 15 article only about Google’s algorithm updates, and I still wouldn’t have enough slots for all of the important ones. But “Florida” was the algo update that made everyone aware that algo updates were a real thing that could affect results on a widespread scale. “The outcry from affected site owners has been unprecedented,” reported Danny Sullivan, and this was one of the first major algo shifts that led webmasters to understand that Google doesn’t owe anyone a living. Google’s Panda and Hummingbird updates are the latest and some of the most famous updates, but they’re only the latest in a long line of shakeups.

8. Google Dance and Everflux. Prior to 2003, Google results were more or less the same for approximately three weeks out of the month. During the fourth week, results started to shift and by the end of that week of “Google Dance,” the newly ranked sites emerged for any given query. I remember vividly trying to make recommendations that could be implemented soon enough to be crawled before a given month’s Google Dance, because if your updated content wasn’t crawled by a certain date, you’d miss that month’s cutoff and have to wait another month to benefit from it.

9. Paid Inclusion. The middle ground between full organic (a crawl-based, algo-driven meritocracy) and full paid placement (such as Adwords or banners) was “paid inclusion.” The primary provider of paid inclusion was MSN, achieved through the help of its Inktomi crawling partner. Paid inclusion guaranteed that URLs would be crawled quickly and frequently (at the cost of about $39 for the first URL and $25 for each subsequent URL), but it didn’t guarantee any placement whatsoever. Rapid indexing and frequent refreshes made sense prior to Everflux, but with Google’s market share growing alongside its ability to crawl sites more frequently and keep its indexed refreshed, MSN phased out paid placement around 2004.

10. Nofollow. Designed to combat the manipulation of links for the purposes of SEO gain, the “nofollow” rel attribute dealt a blow to those who bought, sold, and dropped links arbitrarily on sites that would accept them. (Eventually, this would lead to a slight decrease in people with names such as “Long Island Exterminator”.) Because of the robots meta tag command of the same name, there was a great deal of initial confusion about what, exactly, this command did (that is, whether it crawled but did not bestow credit, didn’t crawl at all, etc.), but once the effects were understood, sites began to adopt it en masse. One of the biggest ripples occurred when Wikipedia began adding the tag to outgoing links in 2007.

11. BMW Penalty. In 2006, the German BMW site ( got famous for some low-level JavaScript cloaking. JS-enabled visitors saw the site normally, while JS-disabled visitors (guten Tag, Googlebot) saw a page stuffed full of “Gebrauchtwagen BMW” (“used BMW cars”) terms. Google dropped from the index — for about 36 hours. This was a watershed moment, because it told SEOs two things: First, Google takes its terms of service seriously. Second, if you’re a big brand, you probably have a distinct advantage when it comes to letting Google know you’ve cleaned up your site.

BMW's page, with and without JS enabled, 2006

BMW’s page, with and without JS enabled, 2006

See also: Ode to a German Auto

12. XML Sitemaps. was (is) a joint effort among major search engines that allows webmasters to create XML-based lists of URLs for rapid, efficient crawling purposes. This was one of the first major joint efforts of engines, which have generally come about due to gaps in what engines can accomplish on their own infrastructure side (other examples are the expanded robots protocol, honoring the nofollow tag, and Efforts like this are important because they represent an outreach to webmasters that are relatively rare and that generally benefit the sites taking part and following the recommendations closely.

13. Google Webmaster Central. Later named “Google Webmaster Tools,” Google Webmaster Central was what Google Sitemaps turned into when its toolset and community grew. Developed by Vanessa Fox during her tenure at Google, the site has grown into a full-blown diagnostic center and is now used for key analytics data due to the “not provided” issue. Google has not always been first to market with the various reports and tools that it offers (Yahoo offered parameter-handling tools well before Google did, for example), but no other engine comes close to offering the breadth of information that GWT does.

14. Personalized Search. In early 2007, Google changed the game by delivering search results that varied based on your personal search history. This didn’t necessarily nullify rank checking, but it left you very unsure about what types of results the general population was seeing. While personalized results still exist today (and engines have experimented with using all sorts of factors to personalize them, including social connections, Google +1s, etc.), this change never had the apocalyptic effect on SEO that many predicted it would. See also: ClickZ.

15. Canonical Tag. One of the single most effective tools for duplication mitigation, the canonical tag is the ax to prior parameter-handling tools’ scalpel. The philosophy behind the canonical tag — engines offering ways for webmasters to help engines interpret large content chunks — eventually gave way to additional special tags such as pagination’s rel=”next” and rel=”prev” and the official usage of existing tags such as HREFLANG.

16. Bonus: Non-HTML Search Results. While the “Universal Search” movement from 2007 was a critical moment for SERP evolution, the indexing of non-HTML assets goes much further back. Google indexed PDF files starting in 2001, and Google Image Search launched the same year. While it’s difficult to remember, there was a time when your traditional HTML pages were the only asset type that you needed to optimize for search.

17. Bonus: Keyword (Not Provided). In late 2011, Google moved its searches to the secure (https) platform. What was then predicted to be of minimal impact eventually turned into drastically declining insight on referring keyword data for organic searches. Referring keyword data, in case you’re unfamiliar, tells us exactly what search term the visitor used to arrive at the site. Consequently, while we now know where users land on the site and what they do, we’re less and less sure whether they arrived through branded or non-branded terms, short or long queries, and so on. Google Webmaster Tools offers some solace through its growing Search Queries report, but it’s still nowhere near what we used to have. 

Growth of "not provided" terms from 2011-present.

Growth of “not provided” terms from 2011-present.

As I said before, I’ve probably missed some, and I’m eager to hear your thoughts. Happy 15th Anniversary to my IP colleagues.

Opt In Image
Ready to Work With Us?

Are you ready to have results that you've only dreamed of? Need someone on your side who can explain the ins and outs of organic search and social media marketing? Ready to have your brand be extended, amplified and protected?

Contact our sales team about working with IP and find out what options exist for your business today.

Erik Dafforn
Erik joined Intrapromote full-time in 2002 after coming on as a contractor in 1999. Yeah, he's been here a while. Currently, as President, he divides his time between overseeing the SEO department and managing organic SEO campaigns. He's also one of Intrapromote's Chief Big Idea Guys. Prior to working at Intrapromote, Erik worked in publishing as a development editor in the programming imprints of IDG Books and Prentice-Hall Computer Publishing so if anyone needs a keyboard shortcut, they ask Erik. Among Erik's professional bragging rights include the fact that he led the team that was awarded Honda's prestigious "Premier Partner" vendor award for SEO Services. He also contributed on the recently published Search Engine Optimization Secrets with Danny Dover. Erik rivals Aaron Sorkin in his candid wit (although he will completely disagree with that statement) and has been a tremendous asset to Intrapromote since day one. He is incredibly intelligent and when you read any of his posts, you'll be able to sit back and say "Wow, I've just been Dafforned."
Erik Dafforn
Erik Dafforn
Erik Dafforn


Leave a reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>