Help with robots.txt

[Chris H]

May have mentioned a problem with googlebot leaching all the bandwidth off one of me sites last month.

Set me robots.txt up as follows

User-agent: *
Disallow: /administrator/
Disallow: etc, etc etc.....

User-agent: Slurp
Crawl-delay: 20

User-agent: Googlebot
Crawl-delay: 20

This month I was expecting better things but have now found that less than halfway through the month googlebot has eaten 92% of me 6Gb bandwidth 

Looking at me sitemaps report it tells me that the crawl-dealy is 'Rule ignored by Googlebot'.

Any ideas? Don't really want to shut out google and would like to know why it's ignoring me robots.txt.

I'm going to have to scrap the crawl delay and just disallow googlebot for the next couple of weeks.

[slaughteredlamb]

Please bare in mind that I know less than nothing about web design and how Google works but I thought that Google and indeed all search engines now ignored the robots.txt file so putting anything in there is just a waste of time.

If you just want to exclude everything inside your administrator folder can't you just password protect it which would stop the googlebot and make your site that little bit more secure?

Like I say, I know nothing about this. 

[Chris H]

Disallowing /admin etc isn't the issue. As far as I iknow those folders aren't indexed as I haven't seen any content on the search engines.

It's just that googlebot is visiting so often that it's using all my bandwidth.

Here's a list of bots and bandwidth.

Googlebot 4.20 GB
Inktomi Slurp  380.46 MB
Unknown robot (identified by 'robot') 210.94 MB
MSNBot 194.94 MB

[Mike@TheWhippinpost]

That is extreme.

I'd first check the IP addy from your logs to make sure it is actually Google, then maybe try a [site:yoursite.com] to see what pages it has - It could be there's something about the site menu-construction that is sending it into an ascending loop from index.php?id=x, for instance.

Maybe a search function is getting triggered?

You got Xenu Linksleuth? Try throwing it at your site.

[Chris H]

Cheers Mike.

Google lists over 3000 entries for the site. Not surprising as it's been running since Mambo 4.5 and uses joom!fish to translate into about 3 other languages.

However, Google is listing everything, even the kitchen sink, all graphics, everything that I've set to disallow in robots.txt. Does Google ignore robots.txt?

I do have this in the head:

<meta name="robots" content="index, follow" />

And also this:

<meta name="revisit-after" value="2 days">

which may have been an effort at some time to slow the beastie down.

[Chris H]

Looking back over the year I have noticed that one or two months last year it went close to the bandwidth limit. Other months it used maybe 900Mb for all spiders.

[Mr Anderson]

http://www.google.com/support/webmasters/bin/answer.py?answer=33571&topic=8460

[Chris H]

http://www.google.com/support/webmasters/bin/answer.py?answer=33571&topic=8460

Good thinking. Will do that tonight.

[Mike@TheWhippinpost]

No it doesn't ignore robots.txt but your meta tags might be over-riding them.

Translation will be a heavy hit.

In addition to the above, check links sitewide (as I said) including www and non-www variations. Printer pages. Error pages.

Oh yeah, does your server issue a "if-not-modified" 304?

I'm working in the dark here so I dunno.

[Mr Anderson]

No it doesn't ignore robots.txt

But does ignore crawl delay. Google expects you to contact them to ask them to change the crawl frequency.

[Chris H]

No it doesn't ignore robots.txt

But does ignore crawl delay. Google expects you to contact them to ask them to change the crawl frequency.

I did set crawl delay to slow at the beginning of the month via the sitemapper pages.

[civ]

No it doesn't ignore robots.txt

But does ignore crawl delay. Google expects you to contact them to ask them to change the crawl frequency.

I did set crawl delay to slow at the beginning of the month via the sitemapper pages.

Might take a while to take effect? I guess only G can tell you for sure.

[Chris H]

Setting the crawl delay in sitemapper tools has slowed G down.

Now getting through about 2Gb per 10 days.

An improvement I suppose.

Had to throw some money at the host for extra bandwidth to get the site back. Wonder if Google will reimburse me?