How Search Engines Work
Before you can improve your ranking, you need to understand the complex systems that decide who gets the top spot.
Key Terms — Day 2
Search Engines Do Not Know Everything
It might seem like Google just knows about every website on the internet, but it does not. Search engines have to actively go out and find pages, understand what is on them, and then decide how useful they are when someone searches.
This process happens in three stages: Crawling, Indexing, and Ranking. Understanding these stages is important because SEO is essentially about making each stage go as well as possible for your website.
Stage 1: Crawling
Search engines use automated programmes called bots, also known as crawlers or spiders. These bots are constantly moving across the internet, jumping from one link to another, finding new pages and revisiting old ones.
- Bots start from pages they already know about and follow every link they find
- If no other website links to your page, bots might never find it
- This is one reason why backlinks matter — they create pathways for bots to reach your site
Stage 2: Indexing
Once a bot finds a page, the search engine reads its content and stores it in a massive database called the index. A page that is indexed can appear in search results. A page that is not indexed cannot appear no matter how good it is.
Reasons a page might not get indexed:
- Very little or thin content
- Duplicate content that already exists elsewhere
- Technical errors that stop bots from reading the page
- The page has been blocked in the site's settings
Stage 3: Ranking
When someone types a search query, the search engine looks through its index and picks the pages it thinks are most relevant and useful. It then orders them from best to worst — this is ranking.
Some of the most important ranking factors:
- How relevant the page's content is to the search query
- How many quality websites link to the page (backlinks)
- How fast the page loads
- How long people spend on the page before leaving
- Whether the page is easy to read on a mobile phone
Video Recap Slides
Watch This First
12:04 DurationWatch the full video on Crawling, Indexing, and Ranking before attempting the quiz.
Day 2 Challenge
Answer all 10 questions based on the video. Type your answer in each box then click Verify Knowledge.
1. What is the first stage search engines use to find web pages called?
2. What are the automated programmes search engines use to scan the web called?
3. What is it called when a search engine stores a page's content in its database?
4. What is the third and final stage called, where pages are ordered in search results?
5. How many stages does the video say search engines go through?
6. What helps crawlers find your website if no one has linked to it yet?
7. Give one reason why a page might NOT get indexed by Google.
8. What does a search engine look for when deciding which pages to rank highest?
9. What file can you submit to Google to help it find all your pages faster?
10. Who ultimately decides which result they click on after ranking happens?