Magento 2 robots txt: Configuring it is essential for SEO in store

If you are reading the post, it means either you are a Magento 2 store owner or are responsible for the SEO of the store. You must be aware of the importance of robots’ txt files. It is a text file that instructs the web robots, i.e., search engines which pages to crawl and which not!

In the words of Google, “A robots.txt file tells search engine crawlers which pages or files the crawler can or can’t request from your site.” This tiny file is a part of EVERY website, and Magento 2 stores are no exception. Fortunately, the default Magento 2 supports configuring a robot’s txt file. And it will be shown in this article.

Why is Magento 2 robots txt essential for SEO?

Controls how the search engine spiders see and interact with the pages of the Magento 2 store.
Its improper use can affect the rankings of your store negatively.>
Robots.txt file is the foundation of how the search engine works.
Restrict duplicate content pages from appearing in SERPs.
It helps to avoid overloading your store with requests from Google’s crawler.

SMTP (Simple Mail Transfer Protocol): The benefits and how to configure in Magento 2

Configure Magento 2 robots txt File

1. First of all, log in to your Magento 2 Admin Panel.

2. Click Content and under Design, choose Configuration.

3. From the left side, Edit the Global Design Configuration.

4. Expand the Search Engine Robots section.

5. Now you need to do the following:

a) Set the Default Robots option to one of the following:

INDEX, FOLLOW: Instructs search engine crawlers to index the store and check back later for changes.

NOINDEX, FOLLOW: Instructs search engine crawlers to avoid indexing the store but check back later for changes.

INDEX, NOFOLLOW: Instructs search engine crawlers to index the store once but do not check back for changes.

NOINDEX, NOFOLLOW: Instructs search engine crawlers to avoid indexing the store and do not check back for changes.

b) In the Edit Custom Instruction Of Robots.txt File option, you can enter the custom instructions if needed. For example, while developing your Magento 2 eCommerce store, you can disallow access to all folders by entering custom instructions.

c) Reset To Default button will remove your custom instructions and reset the Magento 2 robots.txt file to the system’s default.

6. Once you’re done with the configuration, hit the Save Configuration button to apply the changes.

Custom Instructions For Magento 2 Robots.txt

Like any other web application or e-commerce platform, you can add custom instructions to the robots.txt file in Magento. Here are some examples of “Disallow” to consider for the Magento 2 robots txt file.

Allow Full Access

```
User-agent:*
```
```
Disallow:
```

Disallow Access to All Folders

```
User-agent:*
```
```
Disallow: /
```

Default Instructions

```
Disallow: /lib/
```
```
Disallow: /*.php$
```
```
Disallow: /pkginfo/
```
```
Disallow: /report/
```
```
Disallow: /var/
```
```
Disallow: /catalog/
```
```
Disallow: /customer/
```
```
Disallow: /sendfriend/
```
```
Disallow: /review/
```
```
Disallow: /*SID=
```

Restrict User Account & Checkout Pages

```
Disallow: /checkout/
```
```
Disallow: /onestepcheckout/
```
```
Disallow: /customer/
```
```
Disallow: /customer/account/
```
```
Disallow: /customer/account/login/
```

Disallow Catalog Search Pages

```
Disallow: /catalogsearch/
```
```
Disallow: /catalog/product_compare/
```
```
Disallow: /catalog/category/view/
```
```
Disallow: /catalog/product/view/
```

Disallow URL Filter Searches

```
Disallow: /*?dir*
```
```
Disallow: /*?dir=desc
```
```
Disallow: /*?dir=asc
```
```
Disallow: /*?limit=all
```
```
Disallow: /*?mode*
```

Restrict CMS Directories

```
Disallow: /app/
```
```
Disallow: /bin/
```
```
Disallow: /dev/
```
```
Disallow: /lib/
```
```
Disallow: /phpserver/
```
```
Disallow: /pub/
```

Disallow Duplicate Content

Disallow: /tag/
Disallow: /review/

Conclusion

The Magento 2 robots txt file is essential when communicating with web crawlers. This standard defines how to inform the search engine bots about the pages of your Magento 2 store that should be excluded or opened for crawling. That’s why the robots txt file is significant for the correct store indexation and its overall search visibility. As Magento 2 provides a mechanism for creating a robots txt file, there is no need to create one manually. All you need to do is add some configuration in Magento itself, and a Magento 2 robots txt will be generated periodically. We hope that you found this quick guide on how to configure it. If you have any questions, feel free to connect with us. We with Magento Website Development service, are willing to help you anytime.

Magento 2 robots txt: Configuring it is essential for SEO in Magento 2 store

Why is Magento 2 robots txt essential for SEO?

Configure Magento 2 robots txt File