How SEO Works: Crawling, Indexing and Ranking

Search engine optimization (SEO) is an important part of any online marketing effort. By moving up in search rankings, you can bring more visitors to your website without paying for ads — potentially growing revenue in a powerful way. But how does SEO work?

This is an enormous, sometimes confusing topic, but we are going to boil it down to give you a clear understanding of what SEO is, how it works, and why it’s so important when you’re talking about digital marketing and web design. Let’s get started!

What is a Search Engine?

Okay, so this is admittedly starting a bit basic, but we’d like to lay the groundwork for the more advanced concepts to come. A search engine, in its simplest form, is a website that helps you find other websites. By entering words into the search bar – those words are known as “keywords” in search parlance – you tell the search engine what you want to find. The engine then goes to work by reviewing its massive catalog of sites and pages to find results that it feels are relevant to your query.

Of course, you are already familiar with the technology as a user. You probably run tens – if not hundreds – of web searches every day. But you may not have stopped to think about how search works from a technical perspective. How do search engines like Google sort through the massive internet and find what you need so reliably? Thinking about search engines from that perspective can help you understand how search engine optimization works, and how you should approach it for your organization.

Learn to Crawl Before You Can Walk

Much like a small child learning to move, a search engine begins its process by crawling. In this case, however, the crawling is digital, and it refers to the process of the search engine visiting every website and web page that it can find across the internet. As you might imagine, crawling the web requires a tremendous amount of resources and computing power, which is one of many reasons why there are so few search engines in the market today.

You don’t need to understand the nitty-gritty details of how search engines crawl the web, but here is a basic outline:

• The search engine sends out its crawlers to explore the web and capture data from each page that it finds
• These crawlers use links to branch out all over the internet and find as many pages as possible to index (more on indexing later)
• This crawling process is constantly underway, as the internet is perpetually changing

If you own a website, you can get a sneak peek at the crawling process by looking at your server logs. In addition to the traffic that landed on your site from regular visitors, you will also find search engine crawlers coming to your pages to capture data and document the presence of your site in their records.

Crawling Creates a Huge Index

Search engines aren’t bothering to crawl the entire discoverable internet just for fun. This crawling activity has a specific purpose in mind – to create an index of web pages that can be delivered as results for any possible query. No matter what the search happens to be, the search engine needs to return a relevant set of results, and all those results will be pulled from the index that is compiled during the crawling process.

As you might imagine, it’s not good enough for a search engine to simply create a basic index of web pages, organized by alphabet or some other rudimentary category. Rather, the index is highly advanced, making it possible to deliver search results in just a fraction of a second, in most cases. And, for popular searches, the number of pages returned is often in the millions. Search engine indexes are indeed incredible pieces of technology that enable the modern internet to act as the powerful, life-changing tool that it has become.

It’s All About Rankings

So far, we have a search engine that has crawled the web to discover every page it can find, and it has collected the information from those pages into an enormous index. That’s great – but it still doesn’t quite do the job that we expect a search engine to do. Lastly, that index needs to be used to produce a relevant list of results for each search that is performed. The order that is used to display the search results is typically called the “rankings” and securing the top spot in those rankings for as many searches as possible is the ultimate goal of SEO.

At this point, we are going to tell you exactly how search engines like Google decide which web pages are moved to the top of the rankings. Okay, not really. The algorithm used by Google is a proprietary piece of technology and would likely sell for trillions of dollars if it were made available. Experts in the field of SEO believe that they know some of the important ranking factors, and Google itself does shine some light on what helps a website improve its rankings from time to time. However, you will not be able to study the precise details of how the rankings are established, so you’ll be left to educated guesswork on this piece of the puzzle.

Optimizing for Search

We’ve now spent a few hundred words taking a high-level view of how search engines operate. You probably aren’t ready to start your own search engine company with the information we provided, but you should have a general outline of what’s going on behind the scenes. Now, let’s get back to the point of how all this impacts you and your business.

Search engine optimization is a complicated topic, but its goal is simple—to give the search engines what they want. The “optimization” in SEO refers to the process of tuning up your site and other efforts to impress the search engines and move up the rankings. In nearly every case, a web page that is higher in the rankings will get more traffic than one further down the list, so getting as high up in the results as possible is an important—and lucrative—goal.

How important is it to dial in your SEO efforts? While the numbers will vary from study to study, you can assume that the web page ranking in the top spot will get roughly a quarter of the clicks after a search is performed. At the bottom of the first page of results, the page in the #10 spot will get just a tenth of those clicks. Let’s put some numbers to this example to highlight the value of SEO:

• Imagine that your targeted search term is searched 5000 times per month, on average
• If you rank in the top spot for that search term, you should get approximately 1,250 clicks during the month
• A page ranking in the 10th spot for that search would get only around 125 clicks

In this example, that’s a difference of 1,125 visitors to your site over the course of a month. What could your business accomplish with more than a thousand extra visits to your website? If your conversion rate is around 10%, you might wind up with more than 100 extra customers based only on moving from the bottom of page one to the top.

This example is a simplification of the real world, of course, but you get the idea. The top spot on Google is a big deal, and your business stands to benefit in a huge way by putting a detailed, effective search engine optimization strategy into practice.

Crafting a Plan

As mentioned above, no one knows for certain how the search rankings are determined. What we do know, however, is that there are various signals that play a role in how pages are ranked. So, as a business that is interested in getting more traffic from search, you can aim to improve in certain areas with the goal of moving up. Some of the places to focus your attention are as follows:

• Creating relevant content. It is often said that “content is king” in SEO, and that certainly seems to be true. When you create content for your website—say, in the form of a blog post—you will have the opportunity to include keywords that are relevant to your business. The examples here are endless, but let’s look at one quickly to highlight the concept. Let’s say you run a website for an HVAC company in Seattle, WA. Your company serves not only Seattle itself, but also nearby cities like Bellevue, Everett, and Renton. So, you could create web pages that explain your services in each of those other locations. Then, when someone who needs HVAC service searches for something like “HVAC company in Bellevue”, you’ll have a chance to appear. If the word “Bellevue” is nowhere to be found on your site, it will be hard to rank. By creating pages for each of the locations you serve, you’ll be telling the search engines exactly what you do and who you serve.
• Acquiring backlinks. In SEO-speak, a “backlink” is simply a link that points to your website from another website. Search engines tend to use these links as a vote of confidence for your site–it’s like another site saying “yeah, that site is worth checking out”. If you can accumulate a long list of backlinks from quality sites, that will be seen favorably in search and your rankings may improve.
• Improving other metrics. There are plenty of other things you can do when working on SEO beyond creating content and trying to get links. Other important areas include improving the speed of your site, improving the design so users stay longer, and making sure the pages are served over a secure (HTTPS) connection. None of these points alone is going to cause your site to dramatically shoot up the rankings, but they can make an impact when taken in total.

Getting Professional Help

Search engine optimization is a competitive game. With so much at stake, there are people in virtually every industry fighting hard for those top spots. It can be hard for an individual business owner to make a dent in their SEO efforts when trying to do it alone. That’s why it’s such a common approach to work with a marketing agency that offers SEO services.

By teaming up with experienced professionals, small businesses can create strategies that will hopefully help them gain an edge over the competition. Whether you want to go all-in on SEO or just test the waters to see what’s possible, the best plan is to get started right away.