If I use the `--parserobots` argument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives. You can try it on `https://google.com` to see the behaviour.
If I use the
--parserobotsargument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives.You can try it on
https://google.comto see the behaviour.