Under attack... by Bing.

[Anti-Trend]

Apparently MSNBOT no longer pays any attention to robots.txt. Sniper had set the connection limit for msnbot to once every 2 minutes. Instead, it was making hundreds of connections per second, which severely bogged down the server. I've dropped their IP block for the time being, but if you experience any strange latency issues with the forums, it could be another clumsy attempt by Microsoft to crawl the site.

 

[RHochstenbach]

I'm indeed noticing performance issues on HWF. I'll try to find the IP range of the Bing crawlers. But how are you blocking their range? Through the firewall or just through the forums software?

edit: Bing uses the 65.55.* range of addresses. Best is to block the entire 65.55.0.0/16 CIDR range.

 

[Sniper]

Thanks I've just blocked that range :)

Were using a firewall, rather than the forum software...

 

[Anti-Trend]

I'm indeed noticing performance issues on HWF. I'll try to find the IP range of the Bing crawlers. But how are you blocking their range? Through the firewall or just through the forums software?

edit: Bing uses the 65.55.* range of addresses. Best is to block the entire 65.55.0.0/16 CIDR range.

Yeah, it's more like 65.52.0.0/14 though. I dropped it on the firewall for a few hours last night, but since they're at it again it looks like a permanent block is called for.

It'd be different if we had a flat website, but most things hosted here are php doing database queries on the back end. I've never seen a search engine behave so badly before now.

 

[RHochstenbach]

I've never seen a search engine behave so badly before now.

With Microsoft you can expect anything. I think if you'd contact them, they'll try to sell Windows Server 2008 to you :D