Usage@Ingest batches

• Usecase: management of batches and solving of incidents that occured during the ingest process

The dashboard shows a list of baches and allows sorting using the attributes Producer, Created, Updated, State. Next to the batches with the state Suspended the table contains buttons allowing to Resume or Cancel the batch. Similarly, in the batches with the state Processing the dashboard allows the user to Suspend or Cancel the batch using the respective buttons.

Action	Description
Suspend	suspends execution of batch, already started ingestion workflows continue until the end, no new ingest workflows will be started
Resume	resumes previously suspended batch, ingest workflows for unprocessed SIP packages are started automatically
Cancel	cancels execution of batch, all ingest workflows with the state PROCESSING are terminated

An instance of Batch can transition between the states:

• Processing – execution of batch has started and some of the SIP packages belonging the batch are in the state Processing

• Suspended – batch has been suspended on request of the user, the batch can be resumed again

• Processed – ingest workflows of all SIP packages of the batch have successfully finished

• Processed with failures – ingest workflows of all SIP packages of the batch have finished, some have finished unsuccessfully

• Canceled – execution of batch has been canceled on request of the user or because of multiple failures of SIP ingest workflows, batch is canceled automatically in the moment when more than half of SIPs belonging the batch have finished with failure and there is a SIP package to be processed, the batch cannot be resumed again

Batch detail

For the detail of a specific batch click the batch in the dashboard. This page contains additional information about the batch:

• Workflow configuration
• transfer area path
• list of associated ingest workflows
• list of related incidents

Attribute external ID at ingest workflow is assigned internally by system, but is to be used as an external identifier of entities. System internally uses for joining of entities in database attribute id of type UUID, this is not shown in front end. The reason for this solution is systems robustness, otherwise it would be difficult to migrate data from other instance of ARCLib without solving collision of identifiers.

Action	Description
Solve incident	user selects checkboxes for incidents that he wants to resolve with a new workflow configuration, from the drop down list chooses the operation Solve and in the text area Workflow configuration provides the new configuration. The new configuration rewrites the old one completely i.e. not only rewrites configuration for the task which invoked the incident but also for all subsequent tasks. Typical replacement scenario is therefore copy the old configuration and just add/rewrite config for the task which invoked the incident.
Cancel incident	user selects checkboxes for incident that he wants to cancel, from the drop down list chooses the operation Cancel and in the text area Reason provides a reason for the cancelation

Ingest workflow detail

For the detail of an ingest workflow click the ingest workflow in the table with the list of ingest workflows. This detail contains comprehensive information about the ingest workflow process and most importantly, it allows debugging of failed ingest workflows.

First lets define the used terminology:

• Event - An event which has occurred during the ingest. Many times there is just one event per BPM task reflecting that the task was executed.
• Issue - Event which has to be solved by JSON configuration. The configuration may for example contain rule stating that if the particular issue occurs, the ingest should ignore it and continue. There might be multiple issues of the same type, e.g. 10 missing files may results in 10 issues.
• Incident - Issue which was not solved by JSON configuration, i.e. the configuration didn't contain any rule how to handle the issue. If there are multiple issues of the same type, they are usually grouped in one incident. Incident pauses the ingest workflow process until the incident is resolved or cancelled. If the incident is solved, the process continues from the current position and another incident may occur later in the process.

Section Ingest Workflow contains general information about the workflow. If the workflow has failed there should be a failure type, message and in some cases a error stacktrace. The failure type is one of the following:

• INVALID_CHECKSUM - the checksum provided by the producer does not match the checksum of the SIP
• AUTHORIAL_PACKAGE_LOCKED - the package to be updated is locked by other ongoing update
• BATCH_CANCELLATION - the batch has been canceled
• BPM_ERROR - an issue has occurred and there was a rule in the JSON configuration which states that if the particular issue occurs, the ingest workflow should fail
• INCIDENT_CANCELLATION - an incident occurred and than user canceled it through GUI
• INTERNAL_ERROR - other unrecognized error

Section Events contains all related ingest events and issues together with name of tool which fired the event. If the flag successfully processed is set to false it means that the event is an issue and it has resulted in incident. By clicking on the event you can see its details. There might be additional linkage with preservation database, e.g. link to related format or in the case of issue a link to an entry in the issue dictionary.

Section Process variables contains all process variables used to pass data between BPM tasks.

Incident detail

For the detail of an incident click the incident in the table with the list of incidents. In the detail you can see:

• Created - time when the incident occurred
• Message - message with the failure description
• Stack trace - full exception stack trace
• Workflow configuration - workflow configuration used when the incident occurred
• Responsible person - the person responsible for solving the incident, that is the person who either started the ingest through Ingest section or created the related ingest routine