Re: Preventing Data Mining

["Raul A. Jager W." <raul@xxxxxxxxxx>]

You can keep a "log" in your application, or just a table indexed by 
client IP, counting the times your server was accesed from each IP in 
the last 3 minutes (or the interval you choose) and rejecting the 
request or adding an increasing ammout of "sleep" time.
____________________________________________________________________________________________________
Wayne McAlpine wrote:

> We make available access to a large database using CGI programs.  More 
> and more frequently, I am seeing attempts to download the entire data 
> base of millions of records by hitting the server with a script to 
> sequentially access every record.
>
> I'm trying to come up with a method of preventing this kind of 
> wholesale download.  Many web sites now use a challenge system 
> consisting of an authorization word in graphics format that the user 
> must key in before being granted access.  The problem is that this 
> solution imposes a burden on our legitimate users that shouldn't be 
> necessary, so I'm looking for a solution that doesn't involve any 
> additional steps for our users.
>
> The first few times this happened, I changed the Apache server 
> configuration to deny access to the offending IP address, but it's 
> impossible to monitor log files to keep up with the ever-changing IP 
> addresses where these attacks originate.
>
> Anyone have any thoughts on how to approach this problem?