data batch not sent to Storage API if smaller than PACKAGE_SIZE_API_SEND
Description
Currently, if a harvester sends a batch smaller than the configured PACKAGE_SIZE_API_SEND
, the batch is not forwarded to the Storage API.
This leads to the situation that the last package of a harvesting run is never stored in the database.
This is especially problematic when PACKAGE_SIZE_API_SEND
is larger than the total harvested data. For example, in small harvests (e.g., ROR), none of the harvested data is ever persisted.
Steps to reproduce
- Configure
PACKAGE_SIZE_API_SEND
to a value larger than the expected harvest size. - Run a harvester with fewer records than
PACKAGE_SIZE_API_SEND
. - Observe that no data is sent to the Storage API, and the database remains empty.