More Fine-Grained Rate Shaping
The examples above all apply a global limit. All requests are put into the same rate queue.
In many circumstances, you may need to apply more fine-grained rate-shape limits. For example, imagine a login page; we wish to limit how frequently each individual user can attempt to log in to just 2 attempts per minute.
With a global limit, the effect would be to restrict accesses to the login page to 2 per minute in total. We wish to impose independent per-user limits instead, so that each user is restricted to 2 logins per minute.
It would be possible to create a unique rate shaping class for each user, and select the appropriate class using TrafficScript, but there is a much better solution.
The rate.use() function can take an optional “key” which identifies a new instance of the rate class. This key can be used to create multiple, independent rate classes that share the same limits, but enforce them independently for each individual key:
$url = http.getPath();
if( string.endsWith( $url, "login.cgi" ) ) {
$user = http.getFormParam( "username" );
rate.use( "login limit", $user );
}The rate shaping limits are applied independently to each different value of $user. As each new user accesses the system, they are limited to 2 requests per minute, independently of all other users who share the “login limit” rate shaping class.
Rate-Shaping Web Spiders
Occasionally, a poorly written Web spider can overwhelm a Web site with requests, using excessive bandwidth and reducing the level of service that legitimate site users receive.
The following TrafficScript rule uses a rate class named “spiders” to limit individual spiders to 1 request per second. The TrafficScript identifies spiders as users who do not prefix their User-Agent header with either “Mozilla/” or “Opera/”. It then applies limits to each individual spider instance, identified by the user agent and source IP address:
$ua = http.getHeader( "User-Agent" );
if( ! string.startsWith( $ua, "Mozilla/" ) &&
! string.startsWith( $ua, "Opera/" ) ) {
$ip = request.getRemoteIP();
rate.use( "spiders", $ua.$ip );
}A more sophisticated rule might only restrict users to 1 request per second if they had previously exceeded a rate of, perhaps 10 per second over a period of 5 seconds, in order to dynamically detect spiders. It could work as follows:
- Maintain a list of known offenders (user agent and IP address) using the data.set() TrafficScript function. This list would initially be empty.
- When a request is received, check the list of known offenders, and use the 1-per-second rate if the request is to be throttled.
- Otherwise, use the 10-per-second rate, but before you apply the rate limit, check the backlog. If it is too large (greater than 50 perhaps), then the user has sent too many requests; insert them into the list of offenders.