DataAuditLog - append first digits from the hash to the saved data folder/zip
Description
Currently the concept is that after processing a method is available in the CRNSDataHub which is along the lines of:
data_hub.archive(/path/, type="folder") # or zip
After this is done a call is done which closes down and parses the DataAuditLog:
DataAuditLog.archive_and_delete_log(site_name="Test_site")
A goal is that the data that has been saved is somehow appended with the first 6 digits of the hash in the DataAuditLog to differentiate between processing methods. This should be implemented in a future version.
Expected Behavior
When the DataAuditLog is closed down, it somehow changes the name of the zip/folder where the data is stored. So in the above case it would be:
/Path/to/data/Test_site_6f8sb7
if some other choices in the pipeline change the processing methods it might then be:
/Path/to/data/Test_site_f9s74d
Current Behavior
DataAuditLog does not interact with the save system and vice versa.
Proposed Solution
One idea is that the archive method on the data_hub telegraphs the name of the folder to the DataAuditLog. This can be done using @log_key_steps('path')
in the archive method. The method to close down the DataAuditLog would then need to get this info, delete it from the log, create the YAML, hash the YAML, append the 6 digits from the YAML to the folder by changing the dir name.
Acceptance Criteria
-
Ensure the DataAuditLog remains none unique to a site (delete save path on parsing) -
Append the hash digits to the saved data