Use of robot.txt ?

View previous topic View next topic Go down

Use of robot.txt ?

Post by zinavo on Tue Apr 11, 2017 7:40 pm

Use of robot.txt ?

zinavo
platinum
platinum

Posts : 105
Earned : 318
Credit : 3
Join date : 2016-10-11

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by zorbee on Wed Apr 19, 2017 3:08 am

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

zorbee
platinum
platinum

Posts : 126
Earned : 261
Credit : 0
Join date : 2016-08-31
Age : 29
Location : Delhi

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by bangalorewebguru on Wed May 03, 2017 9:08 pm

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

bangalorewebguru
platinum
platinum

Posts : 1115
Earned : 4481
Credit : 4
Join date : 2013-10-15
Age : 29
Location : Bangalore

View user profile http://www.bangalorewebguru.co.in/

Back to top Go down

Re: Use of robot.txt ?

Post by janpaul7890 on Thu May 04, 2017 8:44 pm

Robots.txt is a simple text file webmaster creates to instruct search engine crawler how to crawl and indexed of their web pages.

janpaul7890
platinum
platinum

Posts : 72
Earned : 144
Credit : 0
Join date : 2016-08-10

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by hariandro on Thu May 25, 2017 5:54 pm

Robots.txt - This text file which sits in the root of your website's folder communicates a certain number of guidelines to search engine crawlers. For instance, if your robots.txt file has this line in it; User-agent: * Disallow: / it's basically telling every crawler on the web to take a hike and not index ANY of your site's content.

hariandro
platinum
platinum

Posts : 145
Earned : 371
Credit : 0
Join date : 2016-12-22
Location : chennai

View user profile http://www.samsmarine.org

Back to top Go down

Re: Use of robot.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum