In this commit we exand the Robots.txt parser.
Now, `Allow` rules are parsed, they take precedence
over `Disallow` ones and the detection is case
insensitive.