processor.base Module

Base classes for processors.

class wpull.processor.base.BaseProcessor[source]

Bases: object

Base class for processors.

Processors contain the logic for processing requests.

close()[source]

Run any clean up actions.

process(item_session: wpull.pipeline.session.ItemSession)[source]

Process an URL Item.

Parameters:item_session – The URL item.

This function handles the logic for processing a single URL item.

It must call one of engine.URLItem.set_status() or engine.URLItem.skip().

Coroutine.

class wpull.processor.base.BaseProcessorSession[source]

Bases: object

Base class for processor sessions.

wpull.processor.base.REMOTE_ERRORS = (<class 'wpull.errors.ServerError'>, <class 'wpull.errors.ProtocolError'>, <class 'wpull.errors.SSLVerificationError'>, <class 'wpull.errors.NetworkError'>)

List of error classes that are errors that occur with a server.