Airflow logs aren't available while another attempt is running.

Jonas/Hanno reported an issue with the airflow logs. "sometimes just an ip sometimes something like this"

*** Could not read served logs: [Errno 111] Connection refused
*** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.

The only Issue I can reproduce is that airflow logs of earlier attempts are not available in the UI while the pod of a newer attempt is running.

10.1.66.8

bzw:

Jonas proposed to explicitly configure a log-port in the scheduler pod with:

ports: 
- name: logs-port 
  containerPort: 8793

Jonas reported that this fixed the original issue, at my system this change does not have any effect. The logs are served from this scheduler port, independently from the port being explicitly opened in the deployment.

Excerpt from airflow ui without the port configured:

10.1.66.8
*** Found logs served from host http://10.1.66.8:8793/log/dag_id=delete-series-from-platform/run_id=delete-series-from-platform-240227150046322954/task_id=get-input-data/attempt=1.log

With the port configured I get exactly the same message (redeploying airflow leads to kubernetes assigning a different ip to the scheduler pod adress but w/e).

When I redeploy the airflow pod I get the following error:

10.1.66.8
*** Could not read served logs: timed out

because airflow doesn't update the scheduler pod ip immediately but clearing and rerunning the pod gets logs from the correct pod.