Under attack... by Bing.

Discussion in 'HWF Info & Issues' started by Anti-Trend, Nov 10, 2009.

  1. Anti-Trend

    Anti-Trend Nonconformist Geek

    Likes Received:
    118
    Trophy Points:
    63
    Apparently MSNBOT no longer pays any attention to robots.txt. Sniper had set the connection limit for msnbot to once every 2 minutes. Instead, it was making hundreds of connections per second, which severely bogged down the server. I've dropped their IP block for the time being, but if you experience any strange latency issues with the forums, it could be another clumsy attempt by Microsoft to crawl the site.
     
  2. RHochstenbach

    RHochstenbach Administrator

    Likes Received:
    26
    Trophy Points:
    48
    I'm indeed noticing performance issues on HWF. I'll try to find the IP range of the Bing crawlers. But how are you blocking their range? Through the firewall or just through the forums software?

    edit: Bing uses the 65.55.* range of addresses. Best is to block the entire 65.55.0.0/16 CIDR range.
     
  3. Sniper

    Sniper Administrator Staff Member

    Likes Received:
    59
    Trophy Points:
    63
    Thanks I've just blocked that range :)

    Were using a firewall, rather than the forum software...
     
  4. Anti-Trend

    Anti-Trend Nonconformist Geek

    Likes Received:
    118
    Trophy Points:
    63
    Yeah, it's more like 65.52.0.0/14 though. I dropped it on the firewall for a few hours last night, but since they're at it again it looks like a permanent block is called for.

    It'd be different if we had a flat website, but most things hosted here are php doing database queries on the back end. I've never seen a search engine behave so badly before now.
     
  5. RHochstenbach

    RHochstenbach Administrator

    Likes Received:
    26
    Trophy Points:
    48
    With Microsoft you can expect anything. I think if you'd contact them, they'll try to sell Windows Server 2008 to you :D
     

Share This Page