---
metadata:
  - name: generator
    content: Diplodoc Platform v5.66.1
alternate:
  - https://yandex.com.tr/support/webmaster/en/yandex-indexing/site-indexing.md
  - https://yandex.com.tr/support/webmaster/tr/yandex-indexing/site-indexing.md
  - href: https://yandex.com.tr/support/webmaster/en/yandex-indexing/site-indexing.md
    type: text/markdown
    title: Markdown version
  - href: https://yandex.com.tr/support/webmaster/en/llms.txt?revision=r22014549
    rel: describedby
---
> **Documentation Index:** Fetch the complete configuration index at https://yandex.com.tr/support/webmaster/en/llms.txt?revision=r22014549


# How does Yandex search work?

To start displaying your site in search results, Yandex must find out about its existence using the robots.

A _robot_ is a system that crawls site pages and loads them into its database. Yandex has [lots of robots](https://yandex.com.tr/support/webmaster/en/robot-workings/check-yandex-robots.md). Saving pages to the database and their further processing using algorithms is called _indexing_. Based on the uploaded data, search results are generated. They are regularly updated and [may affect the site ranking](https://yandex.com.tr/support/webmaster/en/yandex-indexing/rank.md#site-position).

There are several stages before a site appears in search results:

<div class="container-galery">
  <div class="doc-c-imagemaps">
      <div class="doc-c-imagemaps__image">
          <img src="https://yastatic.net/s3/doc-binary/freeze/Bw-DzK_5EpkQ4PFWXXpqFJm16Ps.png">
          <a class="doc-c-imagemaps__shape doc-c-imagemaps__shape_simple-circle" title="Этап 1. Обход сайта" href="#explore" style="top: 2.9703%; left: 6.17647%; height: 94.0594%; width: 27.9412%;"></a>
          <a class="doc-c-imagemaps__shape doc-c-imagemaps__shape_simple-circle" title="Этап 2. Загрузка и обработка данных (индексирование)" href="#download" style="top: 15.8416%; left: 38.5294%; height: 71.2871%; width: 21.1765%;"></a>
          <a class="doc-c-imagemaps__shape doc-c-imagemaps__shape_simple-circle" title="Этап 3. Формирование базы страниц, которые могут участвовать в поиске" href="#data" style="top: 24.7525%; left: 61.0294%; height: 49.505%; width: 14.7059%;"></a>
          <a class="doc-c-imagemaps__shape doc-c-imagemaps__shape_simple-circle" title="Этап 4. Формирование результатов поиска" href="#search" style="top: 13.3663%; left: 77.7941%; height: 74.2574%; width: 22.0588%;"></a>
      </div>
  </div> 
</div>

<!-- ![](../_assets/indexing1.png) -->

[Stage 1. Crawling the site](#explore)

[Stage 2. Loading and processing (indexing) the data](#download)

[Stage 3. Creating a database of the pages that can be included in search results](#data)

[Stage 4. Generating search results](#search)


## Stage 1. Crawling the site {#explore}

The robot determines which sites to crawl and how often, as well as how many pages to crawl on each of them.

When crawling them, the robot takes into account the list of already known pages, which is based on the following data:
- [Internal](*внутренние) and [external](*внешние) links.
- Links specified in the [Sitemap file](https://yandex.com.tr/support/webmaster/en/controlling-robot/sitemap.md).
- [Yandex Metrica data](https://yandex.com.tr/support/webmaster/en/indexing-options/link-metrica.md).
- Directives in the [robots.txt file](https://yandex.com.tr/support/webmaster/en/controlling-robot/robots-txt.md).
- The site page size — pages larger than 10 MB are not indexed.

Robots continually monitor the appearance of new links, content updates on previously downloaded pages, and page availability. They do this as long as:
- The link is placed on your own or third-party site.
- The page is not prohibited for indexing in the `robots.txt` file.

When the robot tries to load a site page, it receives a response from the server with the HTTP status code:


#|
||
**HTTP status code**
|
**Note**
||
||
200 OK
|
The robot will crawl the page.
||
||
[3XX](https://yandex.com.tr/support/webmaster/en/error-dictionary/http-codes.md#redirect-3xx)
|
The robot needs to crawl the page that is the redirect target. [Learn more about handling redirects](https://yandex.com.tr/support/webmaster/en/robot-workings/managing-redirects.md).
||
||
[4XX](https://yandex.com.tr/support/webmaster/en/error-dictionary/http-codes.md#client-4xx) and [5XX](https://yandex.com.tr/support/webmaster/en/error-dictionary/http-codes.md#server-5xx)
|
A page with this code won't be included in the search. If it was before the robot crawled it, then it will be removed from the search.

To prevent the page from falling out of the search, configure the server so that it responds with the [429](https://yandex.com.tr/support/webmaster/en/error-dictionary/http-codes.md#429) code. The robot will access the page and check the response code. This can be useful if the site page looks incorrect due to problems with the CMS. After the error is fixed, change the server response.


{% note info %}

If the page responds with the 429 code for a long time, it will indicate that the server experiences difficulties with the load. This can reduce the [site crawl rate](https://yandex.com.tr/support/webmaster/en/service/crawl-rate.md).

{% endnote %}
||
|#


{% cut "Useful tools" %}


- [Troubleshooting](https://yandex.com.tr/support/webmaster/en/service/site-diagnostics.md) — Helps check the quality of a site and fix errors, if any.
- [Crawl statistics](https://yandex.com.tr/support/webmaster/en/service/site-indexing.md) — Shows which pages the robot crawled and how often it accesses the site.
- [How to reindex a site](https://yandex.com.tr/support/webmaster/en/robot-workings/site-reindex.md) — Allows you to report a new page on the site or an update of a page already included in the search.
- [Region](https://yandex.com.tr/support/webmaster/en/site-geography/site-region.md) — Helps the robot to correctly determine the region of the site and display it for [location-dependent queries](*geo-dep-queries).
- [Server response check](https://yandex.com.tr/support/webmaster/en/indexing-options/server-response.md) — Indicates whether the page to be indexed is accessible to the robot.


{% endcut %}


**HTTP/2 version support**

 
:   The Yandex indexing bot supports [HTTP/2](https://en.wikipedia.org/wiki/HTTP/2). The HTTP/2 protocol speeds up page loading, including on mobile devices, which makes it easier for users to interact with the website. On top of that, this protocol reduces server load and saves traffic. The HTTP/2 protocol doesn't directly affect the frequency of crawling a website and doesn't change the website's position in Yandex search results.
    
    If you use HTTP/1.1, the robot will continue indexing your website. These two protocols are compatible, so there will be no conflicts with your server settings.

## Stage 2. Loading and processing (indexing) the data {#download}

The robot determines the content of the page and saves it to its database. To do this, it analyzes the page content, for example:
- The contents of the [description](https://yandex.com.tr/support/webmaster/en/indexing-options/description.md) meta tag, the title element, and the [Schema.org](https://yandex.com.tr/support/webmaster/en/schema-org/what-is-schema-org.md) markup, which can be used to generate a page snippet.
- The [noindex](https://yandex.com.tr/support/webmaster/en/controlling-robot/metatags.md#meta-robots) directive in the `robots` meta tag. If it's found, the page won't be included in the search results.
- The [rel="canonical"](https://yandex.com.tr/support/webmaster/en/robot-workings/canonical.md) attribute indicating the address that you consider a priority for displaying in the search results for a group of pages with the same content.
- Text, images, and videos. If the robot determines that the content of several pages matches, it may treat them as [duplicates](https://yandex.com.tr/support/webmaster/en/robot-workings/double.md).


{% cut "Recommendations" %}


- [How to work with page content](https://yandex.com.tr/support/gorsel/tr/indexing.html)
- [How to work with images](https://yandex.com.tr/support/gorsel/tr/indexing.html)
- [How to work with video content](https://yandex.com/support/video/partners/markup.html)
- [Markup of page language versions](https://yandex.com.tr/support/webmaster/en/yandex-indexing/locale-pages.md)


{% endcut %}

{% cut "Useful tools" %}


- [Troubleshooting](https://yandex.com.tr/support/webmaster/en/service/site-diagnostics.md) — Helps check the quality of a site and fix errors, if any.
- [Crawl statistics](https://yandex.com.tr/support/webmaster/en/service/site-indexing.md) — Shows which pages the robot crawled and how often it accesses the site.
- [How to reindex a site](https://yandex.com.tr/support/webmaster/en/robot-workings/site-reindex.md) — Allows you to report a new page on the site or an update of a page already included in the search.


{% endcut %}


## Stage 3. Creating a database of the pages that can be included in search results {#data}

Based on the information collected by the robot, the algorithms determine the pages that can be included in the search results. The algorithms take into account a variety of ranking and indexing factors that are used to make the final decision. For example, the database won't include pages with indexing disabled or [duplicate pages](https://yandex.com.tr/support/webmaster/en/robot-workings/double.md).

A page may contain the original, structured text but the algorithm won't add it to the database, as it's highly unlikely that the page gets into the range of view in the search results. For example, due to lack of demand from users or high competition in this topic.


{% cut "Useful tools" %}


- [Searchable pages](https://yandex.com.tr/support/webmaster/en/service/searchable.md) — Helps you track the status of site pages, for example, HTTP response status codes or duplicate pages.
- [Security and violations](https://yandex.com.tr/support/webmaster/en/service/security-threats.md) — Provides information about violations and infected files.

To find out if a site subdomain appears in the search results, [subscribe to notifications](https://yandex.com.tr/support/webmaster/en/service/notifications.md).


{% endcut %}


## Stage 4. Generating search results {#search}

The algorithm determines the quality of a page, namely:
- To what extent the page content matches the search query (i.e., whether it's relevant).
- Whether the page content is clear and useful to the user.
- Whether the page is convenient (how text is structured, paragraphs and headers of different levels are arranged, and so on).

If the page quality is high enough, it's more likely that it will be included in the search results. As a result, not all site pages appear in Yandex search. They may also [disappear from the search results](https://yandex.com.tr/support/webmaster/en/site-indexing/excluded-pages.md).

[How to improve site ranking in search results](https://yandex.com.tr/support/webmaster/en/yandex-indexing/rank.md)


{% cut "Useful tools" %}


- [Pages in search](https://yandex.com.tr/support/webmaster/en/service/searchable.md) — Allows you to find out which site pages are included or excluded from the search results. You can also track [pages that are most important to you](https://yandex.com.tr/support/webmaster/en/service/tracking-url.md).
- [Query statistics](https://yandex.com.tr/support/webmaster/en/service/statistics.md) — Helps you track the number of impressions of your site and clicks on the snippet.
- [All queries and groups](https://yandex.com.tr/support/webmaster/en/service/search-queries.md) — Shows the search queries for which your site is displayed in the search results.
- [Troubleshooting](https://yandex.com.tr/support/webmaster/en/service/site-diagnostics.md) — Provides information about the pages that have no [description](https://yandex.com.tr/support/webmaster/en/indexing-options/description.md) meta tag or title element.
- [Sitelinks](https://yandex.com.tr/support/webmaster/en/search-results/quick-links-result.md) — Helps you check if there're sitelinks in the snippet and configure them.


{% endcut %}


## Questions and answers {#qanda}

{% cut "The page description in the snippet differs from the content in the description meta tag" %}

The page description in the search results is based on the text that is most relevant to the search query. This can be the content of the Description meta tag or the text placed on the page. For more details, see the [Snippet](https://yandex.com.tr/support/webmaster/en/search-results/site-description.md) section.

{% endcut %}

{% cut "Search results show links to internal site frames" %}

Before loading the page in the browser console, check if the parent frame with navigation is open. If not, open it.

{% endcut %}

{% cut "My server doesn't provide last-modified" %}


Your site will still be indexed even if your server doesn't provide last-modified document dates. However, you should keep in mind the following:

- The date won't be displayed in the search results next to your site pages.
    
- The robot won't know if a site page has been updated since it was last indexed. Modified pages will be indexed less often, as the number of pages that the robot gets from a site each time is limited.
    


{% endcut %}

{% cut "How does encoding affect indexing?" %}

The type of encoding used on the site doesn't affect the site indexing. If your server doesn't pass the encoding in the header, the Yandex robot will identify the encoding itself.

{% endcut %}

{% cut "Can I manage reindexing frequency with the Revisit-After directive?" %}

No. The Yandex robot ignores it.

{% endcut %}

{% cut "Does Yandex index a site on a foreign domain?" %}

Yes. Sites containing pages in Russian, Ukrainian, and Belarusian are indexed automatically. Sites in English, German, and French are indexed if they might be interesting to users.

{% endcut %}

{% cut "How does a large number of URL parameters and the URL length affect indexing?" %}


A large number of parameters and nested directories in the URL, or overly long URLs, may interfere with the site indexing.

The URL can be up to 1024 characters.


{% endcut %}

{% cut "Does the robot index GZIP archives?" %}

Yes, the robot indexes archives in GZIP format (GNU ZIP compression).

{% endcut %}

{% cut "Does the robot index anchor URLs (#)?" %}


The Yandex robot doesn't index anchor URLs of pages, except for AJAX pages (with the #! character). For example, the `http://example.com/page/#title` page won't get into the robot database. It will index the `http://example.com/page/` page (URL before the # character).


{% endcut %}

{% cut "How does the robot index paginated pages?" %}

The robot ignores the `rel` attribute with the `prev` and `next` values. This means that pagination pages can be indexed and included in search without any restrictions.

{% endcut %}

## Learn more

- [How do I make my site appear in search results?](https://yandex.com.tr/support/webmaster/en/robot-workings/robot.md)
- [Why is it taking so long for pages to appear in search results?](https://yandex.com.tr/support/webmaster/en/robot-workings/robot.md#not-in-search)
- [Why are pages excluded from the search?](https://yandex.com.tr/support/webmaster/en/site-indexing/excluded-pages.md)
- [How many site pages are included in search?](https://yandex.com.tr/support/webmaster/en/site-indexing/page-in-search.md)


<div class="cut-button">

{% cut "Contact support" %}

If the pages don't appear in the search results for a long time or were excluded from them, provide examples of such pages in the form.




<div style="padding: 15px;
      margin: 10px 0;
      background: #FFFFFF;
      border-radius: 10px;
      border: 1px solid var(--yc-color-line-generic);">
  <iframe style="background: #FFFFFF;"
        height="700"
        width="100%"
        frameborder="0"
        src="https://forms.yandex.com.tr/surveys/2426/?&iframe=1&lang=tr">
  </iframe>
</div>



{% endcut %}

</div>


<!-- source: en/_includes/cut-button-style.md -->

<!-- endsource: en/_includes/cut-button-style.md -->

<!-- source: en/_includes/footer.md -->
<div width="100%" align="center" class="main-footer"> 
<br><br><br>
You can also go to 
<br><br>





[![](../_assets/logo/logo-pic.svg)](https://webmaster.yandex.com.tr/) [![](../_assets/logo/logo-en.svg)](https://webmaster.yandex.com.tr/)


</div>


<!-- source: en/_includes/styles/footer-style.md -->

<!-- endsource: en/_includes/styles/footer-style.md -->
<!-- endsource: en/_includes/footer.md -->




[*внутренние]: A link that connects pages on a single site. Pages can be located in different directories of the same domain or on subdomains.

[*внешние]: A link located on another site.

[*geo-dep-queries]: A query that is related to a specific region. For example, [taxi] or [buy iphone]. [Learn more](http://company.yandex.com/technologies/regions.xml)