What is robots.txt?

Go down

Re: What is robots.txt?

Post by bangalorewebguru on Thu Aug 10, 2017 9:09 pm

This text file which sits in the root of your website's folder communicates a certain number of guidelines to search engine crawlers. For instance, if your robots.txt file has this line in it; User-agent: * Disallow: / it's basically telling every crawler on the web to take a hike and not index ANY of your site's content.

bangalorewebguru
platinum
platinum

Posts : 1185
Earned : 4828
Credit : 4
Join date : 2013-10-15
Age : 24
Location : Bangalore

View user profile

Back to top Go down

Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum