Skip to content

data batch not sent to Storage API if smaller than PACKAGE_SIZE_API_SEND

Description
Currently, if a harvester sends a batch smaller than the configured PACKAGE_SIZE_API_SEND, the batch is not forwarded to the Storage API.

This leads to the situation that the last package of a harvesting run is never stored in the database.

This is especially problematic when PACKAGE_SIZE_API_SEND is larger than the total harvested data. For example, in small harvests (e.g., ROR), none of the harvested data is ever persisted.

Steps to reproduce

  1. Configure PACKAGE_SIZE_API_SEND to a value larger than the expected harvest size.
  2. Run a harvester with fewer records than PACKAGE_SIZE_API_SEND.
  3. Observe that no data is sent to the Storage API, and the database remains empty.