Obeying robots.txt seems to work only for some links

[danrg]

There seems to be a bug in Rcrawlerp.R, line 429:
domain<-strsplit(gsub("http://|https://|www\\.", "", Website), "/")[[c(1, 1)]] if (Obeyrobots) { rules<-RobotParser(Website,Useragent) urlbotfiler<-rules[[2]] urlbotfiler<-gsub("^\\/", paste("http://www.",domain,"/", sep = ""), urlbotfiler , perl=TRUE) urlbotfiler<-gsub("\\*", ".*", urlbotfiler , perl=TRUE) } else {urlbotfiler=" "}

This suggests only links starting with 'http://www' are going to obey robots.txt.

Are PRs accepted?