Affiliates
Latest topics
» What is organic result in seo...???
Yesterday at 6:00 pm by komaljain0012

» What is Indexing in seo..???
Yesterday at 5:54 pm by komaljain0012

» What is organic result in seo ...????
Tue Apr 24, 2018 10:20 pm by sameergupta12

» What is Web Hosting..???
Tue Apr 24, 2018 10:17 pm by sameergupta12

» What is organic result in seo...???
Mon Apr 23, 2018 7:07 pm by kalmakuran

» What is a Web server..????
Mon Apr 23, 2018 7:02 pm by kalmakuran

» Most popular search engines?
Sat Apr 21, 2018 7:43 pm by angilpari1122

» which year was Google founded..?
Fri Apr 20, 2018 10:20 pm by rajshreedixit01

» common SEO mistakes..?
Sat Apr 14, 2018 9:25 pm by angilpari1122


Use of robot.txt ?

Go down

Use of robot.txt ?

Post by zinavo on Tue Apr 11, 2017 7:40 pm

Use of robot.txt ?

zinavo
platinum
platinum

Posts : 105
Earned : 318
Credit : 3
Join date : 2016-10-11

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by zorbee on Wed Apr 19, 2017 3:08 am

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

zorbee
platinum
platinum

Posts : 131
Earned : 271
Credit : 0
Join date : 2016-08-31
Age : 29
Location : Delhi

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by bangalorewebguru on Wed May 03, 2017 9:08 pm

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

bangalorewebguru
platinum
platinum

Posts : 1199
Earned : 4877
Credit : 4
Join date : 2013-10-15
Age : 25
Location : Bangalore

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by janpaul7890 on Thu May 04, 2017 8:44 pm

Robots.txt is a simple text file webmaster creates to instruct search engine crawler how to crawl and indexed of their web pages.

janpaul7890
platinum
platinum

Posts : 72
Earned : 144
Credit : 0
Join date : 2016-08-10

View user profile

Back to top Go down

Re: Use of robot.txt ?

Post by hariandro on Thu May 25, 2017 5:54 pm

Robots.txt - This text file which sits in the root of your website's folder communicates a certain number of guidelines to search engine crawlers. For instance, if your robots.txt file has this line in it; User-agent: * Disallow: / it's basically telling every crawler on the web to take a hike and not index ANY of your site's content.

hariandro
platinum
platinum

Posts : 145
Earned : 371
Credit : 0
Join date : 2016-12-22
Location : chennai

View user profile http://www.samsmarine.org

Back to top Go down

Re: Use of robot.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum