I have experimented and learned so much on robots.txt file and read so many article over net and implemented my robots.txt file by following rules and regulations suggested by google. I have clearly shown my success in uploading robots.txt file and got it downloaded by google  1 hour back.

Today (27/06/2009) afternoon around 12:00 PM my Crawler access page was showing as below

Check to see that your robots.txt is working as expected. (Any changes you make to the robots.txt content below will not be saved.)

Text of http://www.punarvasi.com/robots.txt

User-agent: *
Disallow:

Sitemap: http://punarvasi.com/sitemap.xml.gz

Then I realised that my robots.txt file is stopping Google bot in crawling my site because robots.txt file is having User-agent: * Disallow: line. So immediately what I did is I have generated ne file using Google webmaster tools crawlers’ access page and uploaded to my site’s root. After uploading this fresh file with allow command in robots.txt file also same status of above information like above. I have checked now the status is changed to the format just like below. Now the time is 10:36 PM

Check to see that your robots.txt is working as expected. (Any changes you make to the robots.txt content below will not be saved.)

 
Text of http://www.punarvasi.com/robots.txt

User-Agent: *
Allow: /

So I learned that Google will download the robots.txt file and contents of the file will be shown in the text box which is located after
“Text of http://www.punarvasi.com/robots.txt” line. Google has taken almost 10 hours time to update robots.txt file in crawler access page

If you need explanation about robots.txt file interactively, please send a message to I will show you this online

Tags: downloaded robots.txt

This entry was posted on June 27, 2009 at 10:24 PM. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Posted in Blog by Raj No Comments Yet