Disallow search engines on the old website

[sdarwin]

in order to encourage search engines to display boost content from the new website.

[gregjkal]

As long as we're OK with leaving any already-crawled original.boost.org pages, this looks good to me.

If we want to remove already-crawled pages (I don't know if that's a real problem), this won't necessarily help, and it may be better to add noindex meta tags or X-Robots-Tag: noindex headers in responses.

[sdarwin]

"Google will remove a site from the index if blocked by robots.txt, but it will take time. Search engines are notoriously slow."

While this is a basic fix, if it works, then a delay of even a month shouldn't be a big deal, since I don't believe there's some major problem that Google is responding to most queries with "original". I have tested a few queries, and none of them include "original" near the top.

[sdarwin]

cc. @pdimov