Fix SF bug #482171: webchecker dies on file: URLs w/o robots.txt
The cause seems to be that when a file URL doesn't exist, urllib.urlopen() raises OSError instead of IOError. Simply add this to the except clause. Not elegant, but effective. :-)
Showing
Please register or sign in to comment