Should check_robots_txt be True by default?

[ttoommmmiiee]

Hi All,

Curious about the decision to make the check_robots_txt equal False by default. Seems like making the crawler compliant by default would help instil a better culture of web crawling. I can understand that there may be instances where you want to override this but in general I imagine that the community would support encouraging a culture of respect for data use preferences.

Curious to hear arguments otherwise. Thanks!