Affiliates
Latest topics
» Oldest Temple in India..?
Yesterday at 2:27 pm by angilpari1122

» which is the oldest religion..?
Fri Nov 17, 2017 9:42 pm by vishakhajain

» freezing rain and sleet..?
Fri Nov 17, 2017 8:16 pm by rajshreedixit01

» Best product to export from India..?
Fri Nov 17, 2017 3:43 pm by ranisharma1531

» Referral traffic
Thu Nov 16, 2017 8:57 pm by sameergupta12

» RankBrain in SEO
Thu Nov 16, 2017 8:48 pm by sameergupta12

» How to save India from pollution..?
Thu Nov 16, 2017 4:05 pm by nishajain1421

» major religions in India..?
Thu Nov 16, 2017 2:17 pm by ShrutiDesai

» Xml Sitemap
Wed Nov 15, 2017 9:10 pm by nidhijain1221


What is the use of Robots.txt ?

View previous topic View next topic Go down

What is the use of Robots.txt ?

Post by jerrysmith on Wed Oct 26, 2016 9:40 pm

Hello friends,


I want to know that What is the use of Robots.txt ? tell me ..

jerrysmith
diamond member
diamond member

Posts : 49
Earned : 236
Credit : 0
Join date : 2015-10-12

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by zinavo on Thu Oct 27, 2016 7:03 pm

The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. This is done primarily in order to save the crawler’s time: bots crawl on a budget .

zinavo
platinum
platinum

Posts : 105
Earned : 318
Credit : 3
Join date : 2016-10-11

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by zorbee on Thu Nov 03, 2016 4:36 am

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

zorbee
platinum
platinum

Posts : 126
Earned : 261
Credit : 0
Join date : 2016-08-31
Age : 29
Location : Delhi

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by bangalorewebguru on Mon Dec 12, 2016 4:47 pm

Robots code is most useful step in On page seo.Which is use to disallow to crawl some of the main pages in the website which are all we are added in the robots file.

bangalorewebguru
platinum
platinum

Posts : 1115
Earned : 4481
Credit : 4
Join date : 2013-10-15
Age : 29
Location : Bangalore

View user profile http://www.bangalorewebguru.co.in/

Back to top Go down

Re: What is the use of Robots.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum