Friday, March 15, 2013

How to generate robots.txt for blogspot blogger

13 comments
Google now lets you edit your robot.txt file to blogger. What are Robots.txt files, robots files are plain text files which instruct the search engines what to not index.

Follow the steps :

1) Log-in to your blogger account.
2) Go to your blog.
3) Now go to "Settings>Search Preferences"
4) Under the "Crawlers and indexing" you can find "Custom robots.txt"
5) Click "Edit" link.

Now you can add different Crawler to instructions. This post I added different useful crawl robots.txt codes. First we identify how this code works.

User-agent: *
Disallow: /search
Allow: /

Above code,

User-agent: - Mention crawler name. For instance, Google bot.
Disallow: - Specify which pages should not crawl.
Allow: - Which pages crawl.
/ (slash):- indicate your home page.

Setup instructions for all robots

If you use above code, it will cause duplicate content issues and your site rank will be reduced. So that, you can use the following code. It will allow index and crawl entire blog but, not allow label and search pages.

User-agent: *
Allow: /

Block label and search pages crawling.
If you use above code, it will cause duplicate content issues and your site rank will be reduced. So that, you can use the following code. It will allow entire blog but, not allow crawl label and search pages.
User-agent: *
Disallow: /search
Allow: /


Block certain page (s).

Some reasons, you many need to hide your selected page or pages from the search engines. At that time you can use the following code.
User-agent: *
Disallow: /p/page-one.html
Allow: /

If you need to block more-than one page add their URL one by one on Disallow section like below.

User-agent: *
Disallow: /p/page-one.html
Disallow: /p/page-two.html
Allow: /

Allow all but block specific crawler.

If you want to block single crawler, you can add the following code.

User-agent: <bot name>
Disallow: /

User-agent: Googlebot-News
Disallow:

Instruction on how to setup crawler of adsense :

To improve your Google AdSense performances; you can specify how AdSense bot crawl your site. Actually there is no need to block anything.

User-agent: Mediapartners-Google
Disallow:

Block Images indexing:

If you don't like to see your blog post’s images on the Google search result, you can remove them by using the following code.

User-agent: Googlebot-Image
Disallow: /

If you need to block any-other bot crawl your site, use following code. However you need to add selected bot name in “User-agent:” section.

User-agent: <required crawler name>
Disallow:

User-agent: *
Disallow: /

You can find more crawlers and their user agent information in here and in here.

Add sitemap:

Apart from above crawl instructions, you can add a sitemap. Normally blogger default sitemap provides 26 posts. So that you can add correct sitemap using "Custom robots.txt". This is an example of how to add a sitemap.

If you need to improve your blog's search engine visibility and crawl entire blog other than labels and search pages using the following code as your robots.txt.

User-agent: Googlebot-Image
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: <paste  your blog sitemap here>

Default Sitemap for Blogspot Blog if you don't know how to setup :

Sitemap: /feeds/posts/default?orderby=updated
Sitemap: /atom.xml
Sitemap: /atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: /atom.xml?redirect=false&start-index=501&max-results=500

User-agent:  *
Disallow: /search*
Disallow: /feeds/*
Disallow: *?*
Allow: /search/label/*
Allow: /

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://thebloggingtricks.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Replace "thebloggingtricks.blogspot.com" with your website.

13 comments:

  1. please visit my blog completepcguide.blogspot.com

    ReplyDelete
  2. Thanks for informing about Robotstxt. We are giving more updates with...
    web application development | local seo australia

    ReplyDelete
  3. Good work…unique site and interesting too… keep it up…looking forward for more updates.Good luck to all of you and thanks so much for your hard-work.
    Transcription Services Bangalore, Closed Captioning Services in Bangalore

    ReplyDelete
  4. I am very happy when read this blog post because blog post written in good manner and write on good topic.
    Thanks for sharing valuable information.
    Web Design Company Bangalore,
    Digital Marketing Company

    ReplyDelete
  5. Awesome blog. It was very informative. I would like to appreciate you. Keep updated like this best sap simple finance online training institute in hyderabad

    ReplyDelete
  6. Dapatkan Berita Teraktual Dan Terupdate Seputar Dunia Sepakbola Hanya di www.beritabola6.com

    Yang Mencakup :
    -Berita Sepakbola Indonesia hingga Mancanegara
    -Live Skor
    -Live Streaming
    -Bursa Taruhan Sepakbola
    -Jadwal Pertandingan
    -Klasemen

    Untuk Info Lebih Lanjut Silakan Kunjungi Website kami di www.beritabola6.com

    ReplyDelete
  7. Dapatkan Berita Teraktual Dan Terupdate Seputar Dunia Sepakbola Hanya di www.beritabola6.com

    Yang Mencakup :
    -Berita Sepakbola Indonesia hingga Mancanegara
    -Live Skor
    -Live Streaming
    -Bursa Taruhan Sepakbola
    -Jadwal Pertandingan
    -Klasemen

    Untuk Info Lebih Lanjut Silakan Kunjungi Website kami di www.beritabola6.com

    ReplyDelete
  8. Dapatkan Berita Teraktual Dan Terupdate Seputar Dunia Sepakbola Hanya di www.beritabola6.com

    Yang Mencakup :
    -Berita Sepakbola Indonesia hingga Mancanegara
    -Live Skor
    -Live Streaming
    -Bursa Taruhan Sepakbola
    -Jadwal Pertandingan
    -Klasemen

    Untuk Info Lebih Lanjut Silakan Kunjungi Website kami di www.beritabola6.com

    ReplyDelete
  9. Dapatkan Berita Teraktual Dan Terupdate Seputar Dunia Sepakbola Hanya di www.beritabola6.com

    Yang Mencakup :
    -Berita Sepakbola Indonesia hingga Mancanegara
    -Live Skor
    -Live Streaming
    -Bursa Taruhan Sepakbola
    -Jadwal Pertandingan
    -Klasemen

    Untuk Info Lebih Lanjut Silakan Kunjungi Website kami di www.beritabola6.com

    ReplyDelete
  10. Dapatkan Berita Teraktual Dan Terupdate Seputar Dunia Sepakbola Hanya di www.beritabola6.com

    Yang Mencakup :
    -Berita Sepakbola Indonesia hingga Mancanegara
    -Live Skor
    -Live Streaming
    -Bursa Taruhan Sepakbola
    -Jadwal Pertandingan
    -Klasemen

    Untuk Info Lebih Lanjut Silakan Kunjungi Website kami di www.beritabola6.com

    ReplyDelete
  11. This site helps to clear your all query. rdvv ba 2nd year result 2021
    davv ba 2nd year result 2021 This is really worth reading. nice informative article.

    ReplyDelete
  12. Excellent blog since I have visited is really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative. We are also providing the best services click on below links to visit our website.
    Oracle Fusion HCM Training
    Workday Training
    Okta Training
    Palo Alto Training
    Adobe Analytics Training

    ReplyDelete