We use data pipelines for raw data harvesting, re-harvesting sources, mapping the data semantically and integrating it into our graph DB.

All these steps should be managed by an appropriate pipeline manager (e.g. Apache AIRflow). The Logs need to be extracted purposefull (i.e. to investigate bugs, and to extract values for reporting.

This includes orchestration with deployment components and testing.