Does anyone know of a reliable open source library for parsing robots.txt files in Java? I found a couple of open source projects at Sourceforge but neither had released anything. This is part of the standard library in Python and available in CPAN for Perl, so there must be something. Perhaps hidden off inside some other project like Lucene or Nutch? If you know of one, please drop me a line. Otherwise, I may have to resort to writing it myself.
Update: suggested using various code search tools. As I half suspected, the Apache Project's Nutch does include a class to do this. I'll have to see if I can extract this without pulling in the entire Nutch codebase.