I like the idea of using a rewrite rule instead of robots.txt, though that still assumes the crawlers are actually their own useragent rather than emulating a regular user.