WPMU Plugin: WP Global Robots.txt
robot.txt get more important each day in my opinion. If you take a look at what some of these bots are looking for it is a total waste of server resources.
I’ve had some of these repeatidly look for pages which were on a system we removed well over a year ago – I mean, come on a YEAR AGO. And if you don’t find the file…please – stop looking. Bah, whatever – you are blocked! :)