Just how do you set up robots.txt to permit crawling of the website with the exception of a couple of directory sites?

What is the most effective first or basic arrangement for the robots.txt to permit internet search engine to experience the website, yet possibly limit a couple of folders?

Exists a basic arrangement that should constantly be made use of?

0
webdev
2019-05-07 03:00:23
Source Share
Answers: 2

Google Webmaster devices has actually a Section called "Crawler accessibility"

This area permits you really conveniently to create your robots.txt

For instance to permit every little thing other than blog site a folder called examination your robot.txt would certainly look something like 

User-agent: *
Disallow: /Test
Allow: /
0
2019-05-09 09:22:19
Source

Here's every little thing you require to find out about the robots.txt file

0
2019-05-09 09:21:24
Source

Related questions