What is the use of Robots.txt ?

View previous topic View next topic Go down

What is the use of Robots.txt ?

Post by jerrysmith on Wed Oct 26, 2016 9:40 pm

Hello friends,


I want to know that What is the use of Robots.txt ? tell me ..

_________________
Get My Lost Lover Back in india | How To Get My Ex Girlfriend Back | husband Wife Dispute Problem Solution | Love And Relationship Problems Solutions

jerrysmith
diamond member
diamond member

Posts : 49
Earned : 236
Credit : 0
Join date : 2015-10-12

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by zinavo on Thu Oct 27, 2016 7:03 pm

The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. This is done primarily in order to save the crawler’s time: bots crawl on a budget .

_________________
Web Designing Company Bangalore|Website Designing Bangalore|
Web Designing Companies Bangalore|
Website Designing Company Bangalore

zinavo
platinum
platinum

Posts : 105
Earned : 318
Credit : 3
Join date : 2016-10-11

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by zorbee on Thu Nov 03, 2016 4:36 am

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

_________________
Salesforce App Development | Salesforce Consultants | Salesforce Implementation Partners

zorbee
platinum
platinum

Posts : 121
Earned : 251
Credit : 0
Join date : 2016-08-31
Age : 28
Location : Delhi

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by bangalorewebguru on Mon Dec 12, 2016 4:47 pm

Robots code is most useful step in On page seo.Which is use to disallow to crawl some of the main pages in the website which are all we are added in the robots file.

_________________
Web Designing Company Bangalore | Website Designing Company Bangalore | Website Design Company Bangalore | Web Design Companies Bangalore

bangalorewebguru
platinum
platinum

Posts : 1114
Earned : 4476
Credit : 4
Join date : 2013-10-15
Age : 29
Location : Bangalore

View user profile http://www.bangalorewebguru.co.in/

Back to top Go down

Re: What is the use of Robots.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum