Re: Preventing Data Mining -- WEB400

When reading you message in GMAIL, Google showed me an ad for the following ...

http://exterminator.bigwebmaster.com/exterminator-features.html
It is a perl script for Apache and a method to trap "leechers" who
steal images, and uses the Apache banned ip features.

It might be adaptable usable for your purposes, or it might just show
you have to update teh banned IP list in Apache automatically.

On 12/12/05, Tom Jedrzejewicz <tomjedrz@xxxxxxxxx> wrote:
> On 12/12/05, Wayne McAlpine <wayne.mcalpine@xxxxxxxxxxxxxxxxx> wrote:
> > We make available access to a large database using CGI programs.  More
> > and more frequently, I am seeing attempts to download the entire data
> > base of millions of records by hitting the server with a script to
> > sequentially access every record.
> >
> > I'm trying to come up with a method of preventing this kind of wholesale
> > download.  Many web sites now use a challenge system consisting of an
> > authorization word in graphics format that the user must key in before
> > being granted access.  The problem is that this solution imposes a
> > burden on our legitimate users that shouldn't be necessary, so I'm
> > looking for a solution that doesn't involve any additional steps for our
> > users.
> >
> > The first few times this happened, I changed the Apache server
> > configuration to deny access to the offending IP address, but it's
> > impossible to monitor log files to keep up with the ever-changing IP
> > addresses where these attacks originate.
> >
> > Anyone have any thoughts on how to approach this problem?
>
> One obvious solution is to force authentication, but this requires
> administration.  You could setup some kind of "create your own user"
> system, but then the offenders will
>
> I have heard of anti-spam mail servers that, when they sense that a
> server is sending huge volumes, they slow down their responses to that
> server so as to essentially disable it.  You could implement something
> similar, effectively "throttling" the offending servers.  Count
> queries by IP address in your CGI program.  Then, put a delay in the
> job, holding the connection open and slowing the their server.  Set
> the delay so that it won't matter to the standard user; if the typical
> user makes 10 requests in a delay, then for example, set the delay to
> 1 sec * (requests - 10).  Or after a certain number of tries in a
> given time period return a "Server busy, please try again" message.
>
> You could also get mean and return bogus data to the offending
> systems, although that might not be legal from a governmental
> organization.
>
> This is similar to a Denial of Service attack, so you might also
> search for "Apache" and "DoS" on Google; there might be some ideas.
>
> Good luck.
>
> --
> Tom Jedrzejewicz
> tomjedrz@xxxxxxxxx
>


--
Tom Jedrzejewicz
tomjedrz@xxxxxxxxx