Affiliates
Latest topics
» What is organic result in seo...???
Yesterday at 7:07 pm by kalmakuran

» What is a Web server..????
Yesterday at 7:02 pm by kalmakuran

» Most popular search engines?
Sat Apr 21, 2018 7:43 pm by angilpari1122

» which year was Google founded..?
Fri Apr 20, 2018 10:20 pm by rajshreedixit01

» common SEO mistakes..?
Sat Apr 14, 2018 9:25 pm by angilpari1122

» What is the meaning of competitive analysis
Fri Apr 13, 2018 2:37 pm by komaljain0012

» What is internal linking
Thu Apr 12, 2018 3:31 pm by nidhijain1221

» What is On Page SEO
Thu Apr 12, 2018 3:26 pm by nidhijain1221

» Oldest people to Climb Mount Everest...?
Tue Apr 10, 2018 6:26 pm by kavitajain9782


What is the use of Robots.txt ?

Go down

What is the use of Robots.txt ?

Post by jerrysmith on Wed Oct 26, 2016 9:40 pm

Hello friends,


I want to know that What is the use of Robots.txt ? tell me ..

jerrysmith
diamond member
diamond member

Posts : 49
Earned : 236
Credit : 0
Join date : 2015-10-12

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by zinavo on Thu Oct 27, 2016 7:03 pm

The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. This is done primarily in order to save the crawler’s time: bots crawl on a budget .

zinavo
platinum
platinum

Posts : 105
Earned : 318
Credit : 3
Join date : 2016-10-11

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by zorbee on Thu Nov 03, 2016 4:36 am

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

zorbee
platinum
platinum

Posts : 131
Earned : 271
Credit : 0
Join date : 2016-08-31
Age : 29
Location : Delhi

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by bangalorewebguru on Mon Dec 12, 2016 4:47 pm

Robots code is most useful step in On page seo.Which is use to disallow to crawl some of the main pages in the website which are all we are added in the robots file.

bangalorewebguru
platinum
platinum

Posts : 1199
Earned : 4877
Credit : 4
Join date : 2013-10-15
Age : 25
Location : Bangalore

View user profile

Back to top Go down

Re: What is the use of Robots.txt ?

Post by Sponsored content


Sponsored content


Back to top Go down

Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum