In an industry now roughly two decades old, it’s very difficult to isolate a generally agreed-up list of the most pivotal moments, ideas, people, and concepts. As a result, to honor IP’s 15th anniversary, what follows is my own list of influential topics and a brief note about how and why each one is important. I’ve probably missed some biggies or included some events or ideas whose validity you’ll question. As always, feel free to support or refute my choices in the comments section.
1. Crawling Search Engines. Today, “search engines” are generally synonymous with crawlers, but that hasn’t always been the case. Prior to the development of really top-grade crawlers, discovering new sites was much more reliant on directories and, to an unfortunate degree, big-media “site of the day”-type posts. With the development of crawlers such as WebCrawler, Lycos, and Inktomi, engines now had much more appeal due to the ability to scale their index size automatically and with less reliance on human editors.
2. Yahoo!. Incorporated in 1995, Yahoo’s popularity rose with the internet population’s need for a qualified way of finding content in an increasingly anarchic internet (I was going to say “anarchic web,” but Yahoo’s directory was also full of non-HTTP resources such as Gopher, WAIS, Usenet, and so on.) While it went through numerous changes about whom it powered, who powered it, and its change from a directory to a crawler-based search engine, it was the epitome of successful, pre-bubble search portals.
3. Google. You can’t have an SEO article that excludes Google as a pivotal piece of history. Founded in 1998, it quickly became the darling of tech-savvy searchers due to its clean interface and, more important, the relevance of its results. Today, Google is synonymous with search, and it has stayed pretty true to its model of simple relevance, even though it’s hit and missed on various ventures outside the traditional SERP.
4. Danny Sullivan. In 1996, Danny published “A Webmaster’s Guide to Search Engines,” which later became the iconic Search Engine Watch. In 1999, he put SEO on the trade show map with the first Search Engine Strategies conference. These events helped facilitate engine-webmaster communications better than any other avenue at the time. At his core, Danny is a journalist and an objective one. Any legitimacy the SEO industry has (don’t go there) first began with Danny Sullivan.
5. PageRank and The Google Toolbar. Far beyond simply allowing users to search from the toolbar, the Google Toolbar featured the famous green PageRank indicator, the most visible (although overly superficial) way of discerning Google’s “opinion” of the URL your browser was currently visiting. This resulted in further PageRank hysteria, even though the toolbar PageRank (tbPR) values were only a snapshot. Due to updates that occurred very sporadically (about once a quarter or so), tbPR values were almost always obsolete and therefore rarely accurate, but unfortunately, it didn’t stop many sites from using PageRank as a marketing KPI.
6. Adwords. Google wasn’t the first search engine to carry advertisements, but the birth of Adwords did answer the crucial question of how Google was, in fact, going to make any money. What started out as a CPM model soon turned to auction-based placement. Google (and other engines) have come under fire throughout the years for the poor delineation between paid and free results, and they continue testing ways of fulfilling the FTC’s mandate of differentiating between paid and natural results for users, all the while trying to convince users that the differences aren’t all that great.
7. Google “Florida” Update. I could write a Top 15 article only about Google’s algorithm updates, and I still wouldn’t have enough slots for all of the important ones. But “Florida” was the algo update that made everyone aware that algo updates were a real thing that could affect results on a widespread scale. “The outcry from affected site owners has been unprecedented,” reported Danny Sullivan, and this was one of the first major algo shifts that led webmasters to understand that Google doesn’t owe anyone a living. Google’s Panda and Hummingbird updates are the latest and some of the most famous updates, but they’re only the latest in a long line of shakeups.
8. Google Dance and Everflux. Prior to 2003, Google results were more or less the same for approximately three weeks out of the month. During the fourth week, results started to shift and by the end of that week of “Google Dance,” the newly ranked sites emerged for any given query. I remember vividly trying to make recommendations that could be implemented soon enough to be crawled before a given month’s Google Dance, because if your updated content wasn’t crawled by a certain date, you’d miss that month’s cutoff and have to wait another month to benefit from it.
9. Paid Inclusion. The middle ground between full organic (a crawl-based, algo-driven meritocracy) and full paid placement (such as Adwords or banners) was “paid inclusion.” The primary provider of paid inclusion was MSN, achieved through the help of its Inktomi crawling partner. Paid inclusion guaranteed that URLs would be crawled quickly and frequently (at the cost of about $39 for the first URL and $25 for each subsequent URL), but it didn’t guarantee any placement whatsoever. Rapid indexing and frequent refreshes made sense prior to Everflux, but with Google’s market share growing alongside its ability to crawl sites more frequently and keep its indexed refreshed, MSN phased out paid placement around 2004.
10. Nofollow. Designed to combat the manipulation of links for the purposes of SEO gain, the “nofollow” rel attribute dealt a blow to those who bought, sold, and dropped links arbitrarily on sites that would accept them. (Eventually, this would lead to a slight decrease in people with names such as “Long Island Exterminator”.) Because of the robots meta tag command of the same name, there was a great deal of initial confusion about what, exactly, this command did (that is, whether it crawled but did not bestow credit, didn’t crawl at all, etc.), but once the effects were understood, sites began to adopt it en masse. One of the biggest ripples occurred when Wikipedia began adding the tag to outgoing links in 2007.
See also: Ode to a German Auto
12. XML Sitemaps. Sitemaps.org was (is) a joint effort among major search engines that allows webmasters to create XML-based lists of URLs for rapid, efficient crawling purposes. This was one of the first major joint efforts of engines, which have generally come about due to gaps in what engines can accomplish on their own infrastructure side (other examples are the expanded robots protocol, honoring the nofollow tag, and Schema.org). Efforts like this are important because they represent an outreach to webmasters that are relatively rare and that generally benefit the sites taking part and following the recommendations closely.
13. Google Webmaster Central. Later named “Google Webmaster Tools,” Google Webmaster Central was what Google Sitemaps turned into when its toolset and community grew. Developed by Vanessa Fox during her tenure at Google, the site has grown into a full-blown diagnostic center and is now used for key analytics data due to the “not provided” issue. Google has not always been first to market with the various reports and tools that it offers (Yahoo offered parameter-handling tools well before Google did, for example), but no other engine comes close to offering the breadth of information that GWT does.
14. Personalized Search. In early 2007, Google changed the game by delivering search results that varied based on your personal search history. This didn’t necessarily nullify rank checking, but it left you very unsure about what types of results the general population was seeing. While personalized results still exist today (and engines have experimented with using all sorts of factors to personalize them, including social connections, Google +1s, etc.), this change never had the apocalyptic effect on SEO that many predicted it would. See also: ClickZ.
15. Canonical Tag. One of the single most effective tools for duplication mitigation, the canonical tag is the ax to prior parameter-handling tools’ scalpel. The philosophy behind the canonical tag — engines offering ways for webmasters to help engines interpret large content chunks — eventually gave way to additional special tags such as pagination’s rel=”next” and rel=”prev” and the official usage of existing tags such as HREFLANG.
16. Bonus: Non-HTML Search Results. While the “Universal Search” movement from 2007 was a critical moment for SERP evolution, the indexing of non-HTML assets goes much further back. Google indexed PDF files starting in 2001, and Google Image Search launched the same year. While it’s difficult to remember, there was a time when your traditional HTML pages were the only asset type that you needed to optimize for search.
17. Bonus: Keyword (Not Provided). In late 2011, Google moved its searches to the secure (https) platform. What was then predicted to be of minimal impact eventually turned into drastically declining insight on referring keyword data for organic searches. Referring keyword data, in case you’re unfamiliar, tells us exactly what search term the visitor used to arrive at the site. Consequently, while we now know where users land on the site and what they do, we’re less and less sure whether they arrived through branded or non-branded terms, short or long queries, and so on. Google Webmaster Tools offers some solace through its growing Search Queries report, but it’s still nowhere near what we used to have.
As I said before, I’ve probably missed some, and I’m eager to hear your thoughts. Happy 15th Anniversary to my IP colleagues.