airflow scheduler get's stuck
I was running 1k TotalSegmentator inferences. After about 100 successful runs (and 3 fails), the airflow scheduler was not scheduling any gpu jobs anymore. Logs are attached.
logs-from-scheduler-in-airflow-scheduler-574795d475-479lb.log