Anti Spider Wizard
Purpose
You can start this application-specific wizard on the Wizards tab when you have selected an application in the navigation area.
The Anti Spider Wizard is used simply to block undesirable programs automatically accessing a web application.
The check is carried out via the User Agent field in the HTTP header. You can specify a list of valid and invalid character strings that are permitted to occur in this HTTP header. In addition, vWAF generates a robots.txt file, which is observed by standardized search engines.
User Agents that are not allowed are denied by vWAF with HTTP error code 403 (Forbidden).
ATTENTION This feature cannot offer completely full protection due to the nature of the HTTP protocol. A skilled attacker can simulate the behavior of a legitimate user by modifying the HTTP header accordingly. The Anti Spider Wizard is therefore suitable primarily for protecting against slightly less sophisticated attacks or to prevent undesirable access attempts by known spiders.
For more information regarding Wizards, see Using Wizards to Configure Applications.
Attributes
Attribute | Meaning |
|---|---|
Valid User Agents | Here you specify a whitelist of the User Agents that vWAF is to permit access to your web application. By default, the most common browsers are already entered here. Regular Expressions can be used. For details on priority and internal processing, see How Blacklists, Whitelists, and Graylists Are Processed . |
Invalid User Agents | Here you specify a blacklist of the User Agents that vWAF is to deny access to your web application. By default, several known agents are already entered here. Regular Expressions can be used. Tip: Check the Log Files regularly for additional User Agents you may want to enter here as well. For details on priority and internal processing, see How Blacklists, Whitelists, and Graylists Are Processed. |
Examples for specifying the User Agents
The following example permits everything with the exception of curl:
Valid User Agents: ^.*$
Invalid User Agents: curl
The following example permits Internet Explorer only: ^Mozilla/4\.0 (compatible: MSIE \d\.d; Windows.*$
Valid User Agents:
Invalid User Agents: (empty)
Handlers configured by the Anti Phishing Wizard
The Anti Spider Wizard configures the following handlers: