Improved provenance tracking mechanism for uplifting

In !32 (merged) I introduced an implementation for a generic mechanism to track provenance of computations for different datatypes, providing a simple toy example for tracking int computations (see here)

Main Features:

• the provenance trace is pydantic-enabled and supports (de-)serialization
• the decorator does not fix the location of the provenance-tracked argument, it is controlled by the usage site
• the decorator takes care of cloning the input value and adding the computation result to the prov tracking context
• decorated functions can be used normally just as before if tracking is not needed
• tracking is accessed using a new .with_prov method attached to wrapped objects with the decorator
• tracking functions are compositional (one can nest prov-tracked computations)
• patch creation can be disabled (e.g. for performance) tracking-enabled computation chain by setting a flag, no code needs to change

Further Generalization Ideas:

Should the need arise, this could be extended further into a more powerful provenance tracking mechanism.

• each tracked entity could get a UUID (generated on creation, restored from deserialization)
• tracked entities could keep reference to other tracked entities that were passed into a decorated function
• the decorator would allow to pass multiple tracked entities and add an entry to a subset of them, cross referencing each other's id (unwrapped arguments could be either ignored or be auto-wrapped with a one-use entity)

But this might be overengineering at this point and before doing it maybe existing lightweight Python provenance tracking solutions should be evaluated (such as the FZJ alpaca tool?).

TODO:

• implement a subclass for rdflib.Graph (based on the current rdfpatch.py implementation, should be straight-forward)
• substitute the old decorator in the codebase with the new one and adapt respective usage locations
• add tests for the new decorator (generic stuff and graph-specific)

added enhancement label

assigned to @j.broeder

changed the description

changed the description

added Backlog label

unassigned @j.broeder

This was envisioned, because the current decorate 'generate_patch' is bound to graphs and it changes the return signature of the decorated function, i.e. it adds the patch as an output, to be then used in further code.

Of course this could be fixed in requiring that functions always have to return this, with and without the decorator switched on.

Also the whole implementation of provenance tracking and how to do it depends one has to consider the following questions:

1. Why and for what do we want to track the provenance? Do we just want to understand what happened, or do we want such detailed provenance that one can reproduce from the provenance the data? (Which was the idea of the patches, i.e no matter the operation each graph operation could be redone and undone by applying the resulting rdf level patch on the graph. Also there was some hope that one could also use these patches on the file level to update/patch the data already in the large graph. The later turned out to be much more difficult.)

2. Is there a library that one can already use for the needed purpose and needs?

3. How do you ensure that provenance is right and not broken?

• from user is responsible to data pipeline dev is responsible to ensure this, to designed system makes it impossible to break it, i.e by forcing this to be done in only a specific way. This may conflict with point 4. Also this is the idea of decentralized versus central provenance tracking and how granularity can be adjusted by a developer.

4. How much additional work and knowledge is required for the developer to understand the tracking system and how to use it right? Do not under estimate this point. i.e just one decorator to add, to requirements on the functions and how they have to be implemented and report.