Remove dlang.org/library/ from search results?

[Vladimir Panteleev via Digitalmars-d]

On Friday, 19 September 2014 at 04:35:04 UTC, Andrei Alexandrescu 
wrote:
> On 9/18/14, 4:10 PM, Vladimir Panteleev wrote:
>> The DDox documentation generator (and DMD's JSON output, which 
>> it
>> heavily relies on) still seems to have a far way to go, as can 
>> be seen
>> from the project's numerous open reported issues[1] and 
>> multiple
>> problems with pages such as [2].
>>
>> I've been getting the DDox versions of Phobos documentation at 
>> the top
>> of my Google search results, which is a little annoying. We 
>> could remove
>> the pages from search engines via robots.txt until DDox is 
>> ready.
>>
>> Should they be removed from Google for now? Or does anyone 
>> find them
>> useful?
>>
>>   [1]: https://github.com/rejectedsoftware/ddox/issues
>>   [2]: http://dlang.org/library/std/process/execute.html
>
> Is there a way to disable indexing by google yet keep them? -- 
> Andrei

Yes, listing them in robots.txt, which was what I suggested. I'm 
not saying we should actually remove them from the website :)

We already have /phobos-prerelease/ and /library-prerelease/ in 
robots.txt:
http://dlang.org/robots.txt