Provider | @axetroy/crawler

Hierarchy

Provider

Index

Properties

Methods

Properties

Optional defaultHeaders

defaultHeaders: IncomingHttpHeaders

The default headers for send request.

name

name: string

The name of provider

urls

urls: Url[]

The urls that was originally crawled

Methods

Optional beforeRequest

beforeRequest(config: AxiosRequestConfig): Promise<void>

- Defined in work/crawler/crawler/src/provider.ts:43
When before request. You can change the config before send request.

Parameters
- config: AxiosRequestConfig
Returns Promise<void>

Optional beforeStart

beforeStart(): Promise<void>

- Defined in work/crawler/crawler/src/provider.ts:38
Before start provider. something we want to do. eg. connect to database.

Returns Promise<void>

Optional beforeStartUrl

beforeStartUrl(startUrls: Url[]): Promise<Url[]>

- Defined in work/crawler/crawler/src/provider.ts:33
Before start crawl the starts up. we can return a new start urls. You can change urls in this function.

Parameters
- startUrls: Url[]
Returns Promise<Url[]>

parse

parse(response: Response): Promise<any[]>

- Defined in work/crawler/crawler/src/provider.ts:27
How to parse the response and extract data

Parameters
- response: Response
  
  The request response
Returns Promise<any[]>