Automation for the UK Web Archive

Introduction

This project includes the Python Luigi tasks used to automate the management of crawls and content at the UK Web Archive.

The overall, higher-level system documentation is here: <https://github.com/ukwa/ukwa-documentation>, whereas this documentation site provides more fine-grained detail about individual tasks and processes.

API Reference

tasks Luigi task definitions for UK Web Archive processes.
tasks.access Tasks relating to providing access to the UK Web Archive
tasks.ingest Tasks relating to ingesting content into the UK Web Archive
tasks.backup Internal tasks making backups of UKWA systems.
lib Shared code and classes to support UKWA tasks.

Indices and tables