The headers provider.
The proxy provider.
The storage of data. defined how to storage your data.
The Usage-Agent provider.
The max concurrency of request. Default 1.
Run in headless mode. Default: false.
The interval between after request done. let it take a rest. Default: 0.
Whether to continue the last crawl. Default: false.
The retry times of each request when fail. Default: 0.
The timeout of each request. Default: 60 * 1000 ms.
The auth provider.