HelpStartConceptsSpider
The spider is a tool that is used to automatically discover new resources (URLs) on a particular Site. It begins with a list of URLs to visit, called the seeds, which depends on how the Spider is started. The Spider then visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit and the process continues recursively as long as new resources are found.
The Spider can configured and started using the Spider dialogue.
During the processing of an URL, the Spider makes a request to fetch the resource and then parses the response, identifying hyperlinks. It currently has the following behavior when processing types of responses:
Processes the specific tags, identifying links to new resources:
- Base - Proper handling
- A, Link, Area - 'href' attribute
- Frame, IFrame, Script, Img - 'src' attribute
- Meta - 'http-equiv' for 'location' and 'refresh'
- Form - proper handling of Forms with both GET and POST method. The fields values are generated validly, including HTML 5.0 input types.
- Comments - Valid tags found in comments are also analyzed, if specified in the Options Spider screen
If set in the Options Spider screen, it also analyzes the 'Robots.txt' file and tries to identify new resources using the specified rules. It has to be mentioned that the Spider does not follow the rules specified in the 'Robots.txt' file.
OData content using the Atom format is currently supported. All included links (relative or absolute) are processed.
Text responses are parsed scanning for the URL pattern
Currently, the Spider does not process this type of resources.
- When checking if an URL was already visited, the behaviour regarding how parameters are handled can be configured on the Spider Options screen.
- When checking if an URL was already visited, there are a few common parameters which are ignored: jsessionid, phpsessid, aspsessionid, utm_*
- The Spider's behaviour regarding cookies depends on how the spider is started and which options are enabled. For more details refer to the Spider Options screen.
The spider is configured using the Spider Options screen.
| UI Overview | for an overview of the user interface | |
| Features | provided by ZAP | |
| Spider Options screen | for an overview of the Spider Options |
-
ZAP User Guide
- Introduction
-
Getting Started
- Configuring proxies
-
Features
- Active Scan
- Add-ons
- Alerts
- Anti CSRF Tokens
- API
- Authentication
- Break Points
- Callbacks
- Contexts
- Data Driven Content
- Filters
- Globally Excluded URLs
- HTTP Sessions
- Man-in-the-middle Proxy
- Modes
- Notes
- Passive Scan
- Scan Policies
- Scope
- Session Management
- Spider
- Statistics
- Structural Modifiers
- Structural Parameters
- Tags
- Users
- Scanner Rules
- A Simple Penetration Test
-
The User Interface
- Overview
- The Top Level Menu
- The Top Level Toolbar
- The Tabs
-
The Dialogs
- Active Scan
- Add Alert
- Add Break Point
- Add Note
- Encode/Decode/Hash
- Filter
- Find
- History Filter
- Manual Request Editor
- Manage Add-ons
- Manage Tags
-
Options
- Active Scan
- Active Scan Input Vectors
- Alerts
- Anti CSRF Tokens
- API
- Breakpoints
- Callback Address
- Certificate
- Check for Updates
- Connection
- Database
- Display
- Dynamic SSL Certificates
- Extensions
- Global Exclude URL
- HTTP Sessions
- JVM
- Keyboard
- Language
- Local Proxies
- Passive Scan Rules
- Passive Scan Tags
- Passive Scanner
- Rule Configuration
- Scripts
- Search
- Spider
- Statistics
- Persist Session
- Resend
- Scan Policy Manager
- Scan Progress
- Session
- Spider
- The Footer
- Command Line
- Add Ons
- Releases
- Paros Proxy
- Credits
