Open source crawler used by the Internet Archive. Looks likely to be very useful, going to investigate further but certainly I can see many uses for a good, well written (and well behaving) web page crawler/archiver. Especially as a tool to help with my other AI research (i.e. didn't really want to write a crawler myself, but I do need a large archive of websites/pages for much of what my AI research leads to).
1/07/2004 12:15:00 PM
0 commentslinks to this post