Re: Preventing Data Mining

On 12/12/05, Wayne McAlpine <wayne.mcalpine@xxxxxxxxxxxxxxxxx> wrote:
> We make available access to a large database using CGI programs.  More
> and more frequently, I am seeing attempts to download the entire data
> base of millions of records by hitting the server with a script to
> sequentially access every record.
>
> I'm trying to come up with a method of preventing this kind of wholesale
> download.  Many web sites now use a challenge system consisting of an
> authorization word in graphics format that the user must key in before
> being granted access.  The problem is that this solution imposes a
> burden on our legitimate users that shouldn't be necessary, so I'm
> looking for a solution that doesn't involve any additional steps for our
> users.
>
> The first few times this happened, I changed the Apache server
> configuration to deny access to the offending IP address, but it's
> impossible to monitor log files to keep up with the ever-changing IP
> addresses where these attacks originate.
>
> Anyone have any thoughts on how to approach this problem?

One obvious solution is to force authentication, but this requires
administration.  You could setup some kind of "create your own user"
system, but then the offenders will

I have heard of anti-spam mail servers that, when they sense that a
server is sending huge volumes, they slow down their responses to that
server so as to essentially disable it.  You could implement something
similar, effectively "throttling" the offending servers.  Count
queries by IP address in your CGI program.  Then, put a delay in the
job, holding the connection open and slowing the their server.  Set
the delay so that it won't matter to the standard user; if the typical
user makes 10 requests in a delay, then for example, set the delay to
1 sec * (requests - 10).  Or after a certain number of tries in a
given time period return a "Server busy, please try again" message.

You could also get mean and return bogus data to the offending
systems, although that might not be legal from a governmental
organization.

This is similar to a Denial of Service attack, so you might also
search for "Apache" and "DoS" on Google; there might be some ideas.

Good luck.

--
Tom Jedrzejewicz
tomjedrz@xxxxxxxxx