Use of robot.txt ?

View previous topic View next topic Go down

Use of robot.txt ?

Post by bangalorewebguru on Thu Jun 08, 2017 5:15 pm

Use of robot.txt ?

bangalorewebguru
platinum
platinum

Posts : 1115
Earned : 4481
Credit : 4
Join date : 2013-10-15
Age : 29
Location : Bangalore

View user profile http://www.bangalorewebguru.co.in/

Back to top Go down

Re: Use of robot.txt ?

Post by Designpluz on Tue Jun 20, 2017 7:31 pm

it will instruct search engine which pages should be accessed by google bot & which pages has to be restricted for indexing
avatar
Designpluz
platinum
platinum

Posts : 879
Earned : 3873
Credit : 0
Join date : 2017-01-04

View user profile http://www.designpluz.com.au/

Back to top Go down

Re: Use of robot.txt ?

Post by Sankara Vignesh on Wed Jun 21, 2017 8:00 pm

Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Sankara Vignesh
platinum
platinum

Posts : 117
Earned : 348
Credit : 0
Join date : 2015-07-28

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum