Affiliates
Latest topics
» What is long tail keyword
Yesterday at 6:58 pm by kalmakuran

» What is Crawling
Yesterday at 6:46 pm by kalmakuran

» What is Page Title
Yesterday at 6:20 pm by nidhijain1221

» 10 Most Famous Historical Monuments of India...?
Yesterday at 2:28 pm by angilpari1122

» Is YouTube important for marketing?
Wed Feb 21, 2018 5:08 pm by Sankara Vignesh

» How important are keywords in content marketing?
Wed Feb 21, 2018 5:03 pm by Sankara Vignesh

» What are paid results
Wed Feb 21, 2018 4:57 pm by Sankara Vignesh

» What are backlinks
Wed Feb 21, 2018 4:53 pm by Sankara Vignesh

» Name of a few search engine
Wed Feb 21, 2018 4:50 pm by Sankara Vignesh


Use of robot.txt ?

Go down

Use of robot.txt ?

Post by bangalorewebguru on Thu Jun 08, 2017 5:15 pm

Use of robot.txt ?

bangalorewebguru
platinum
platinum

Posts : 1185
Earned : 4828
Credit : 4
Join date : 2013-10-15
Age : 24
Location : Bangalore

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by Designpluz on Tue Jun 20, 2017 7:31 pm

it will instruct search engine which pages should be accessed by google bot & which pages has to be restricted for indexing
avatar
Designpluz
platinum
platinum

Posts : 1224
Earned : 5598
Credit : 0
Join date : 2017-01-04

View user profile http://www.designpluz.com.au/

Back to top Go down

Re: Use of robot.txt ?

Post by Sankara Vignesh on Wed Jun 21, 2017 8:00 pm

Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Sankara Vignesh
platinum
platinum

Posts : 142
Earned : 428
Credit : 0
Join date : 2015-07-28

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum