help design a query interface for DStress' "bad apple" database

Thomas Kuehne thomas-dloop at kuehne.cn
Sun Dec 3 14:13:54 PST 2006


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kirk McDonald schrieb am 2006-12-03:
> Thomas Kuehne wrote:
>> The problem are misbehaving crawling bots repeatedly downloading some
>> parts of the site if the files have a common mime type like text/plain
>> instead of a uncommon on like text/x-dsrc and thereby sucking GBs.
>> I'm currently preparing to move to a hoster with more fine grained controls.
>> Let's wait and see if the bots can be kept below a certain limit, if
>> so I'll enable the "plain view" feature.
>> 
>
> No robots.txt? Or is that what you meant by "misbehaving"?
> http://en.wikipedia.org/wiki/Robots.txt

Ignoring robots.txt and re-requesting the same files over and over
again. I've since then disabled directory listings and use the
robots.txt only to identify smart robots (those that handle redirects of
robots.txt correctly).

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFFc1i9LK5blCcjpWoRAuTiAKCVLX5EwW6GSgaIlqCYCeWJ4bnNsQCgorHx
tv9ieEA4O7pgHbOgpWI9E5Q=
=tnfp
-----END PGP SIGNATURE-----



More information about the Digitalmars-d mailing list