Security concept
The program needs a security concept.
For now, all jobs are run in sibling directories without any encapsulation/containerization/... This means that a job, running concurrently with other jobs, can easily access their data. This is a problem when dealing with jobs that contain sensitive information that must not be made public.
A possible solution might be a pool of cluster users that run incoming jobs in a round-robin manner. Their working directories can be secured using UNIX/LDAP permissions.