help design a query interface for DStress' "bad apple" database
thomas-dloop at kuehne.cn
Sun Dec 3 14:13:54 PST 2006
-----BEGIN PGP SIGNED MESSAGE-----
Kirk McDonald schrieb am 2006-12-03:
> Thomas Kuehne wrote:
>> The problem are misbehaving crawling bots repeatedly downloading some
>> parts of the site if the files have a common mime type like text/plain
>> instead of a uncommon on like text/x-dsrc and thereby sucking GBs.
>> I'm currently preparing to move to a hoster with more fine grained controls.
>> Let's wait and see if the bots can be kept below a certain limit, if
>> so I'll enable the "plain view" feature.
> No robots.txt? Or is that what you meant by "misbehaving"?
Ignoring robots.txt and re-requesting the same files over and over
again. I've since then disabled directory listings and use the
robots.txt only to identify smart robots (those that handle redirects of
-----BEGIN PGP SIGNATURE-----
-----END PGP SIGNATURE-----
More information about the Digitalmars-d