Saturday, 2 March 2013
What is Robots.txt File and How to Edit This?
Robots.txt file is very important for a blog or website for best SEO. This file consists of some instructions for search engines. Each search engine uses its own automatic process to scan the internet which is called "robots". The name of famous robots are crawlers, spiders and wanderers. A good robot always follow the instructions within the file robots.txt. This file is located in the main directory of each site e.g. our site robot.txt file is located in the http://www.bloggerzhelp.com/robots.txt. Similarly, each website has its own robots.txt file within /robots.txt. If a website do not have this file then search engine will ignore that site always or in case this file is configured incorrectly. So to better SEO check the existence of this file and verify the configuration of this.
Before fetching any instruction from your site, robot always read the instructions from this file and follow that which path is allowed and disallowed for it. If there is no robots.txt file, then it is considered by robot that the web owner does not want to provide any specific instructions and leave that site.
- All blogger blogs have this file by default and you can see that the "Custom robots.txt" option is disabled by default. Remember that, for better search engine optimization, always keep disable custom robots.txt file as Blogger is providing you the best robots.txt file. You can check this from Settings>Search Preferences under the field of "Crawlers and Indexing".
For those who has self hosted website must create this file and upload it in the "/" directory. There are some instruction for editing this file.
Open notepad and write this:
# robots.txt for http://www.example.com
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: [your sitemap URL Here]
Replace your domain name instead of example and paste the URL for your sitemap instead of
[your sitemap URL Here]
Now there are some extra tips to create a better robots.txt file:
If you want to allow all robots to visit all files then use this:
User-agent: *
Disallow:
If you want all robots to stay out of your site then use this: (This is not good)
User-agent: * Disallow: /
If you want all robots to restrict from specific directories then use this:
User-agent: * Disallow: /directory1/ Disallow: /foldername/ Disallow: /tmp/
If you want a specif robot not to enter your site then:
User-agent: BadBot (replace 'BadBot' with the actual user-agent of the bot) Disallow: /
Now update your robots.txt file for better Search Engine Optimization.
Best Wishes!
SHARE THIS POST
Author: Muhammad Faheem Usama
Subscribe to:
Post Comments
(
Atom
)
0 comments :