Merge branch '96-use-keycloak-for-athentication-in-development' into 'main'

Resolve "Use keycloak for authentication in development"

Closes #96

See merge request !153

.env.example

@@ -240,3 +240,8 @@ CRON_SMS_API_TOKEN=00000000000000000000
 
 CRON_CV_ACCESS_TYPE=${CV_ACCESS_TYPE}
 CRON_CV_API_URL=${CV_URL}api/v1/
+
+############################################################
+# keycloak
+############################################################
+KEYCLOAK_PORT=8081
\ No newline at end of file

.gitlab/ci/end-to-end.gitlab-ci.yml

@@ -31,6 +31,7 @@ visualization-e2e:
     - "[ \"$TEAMS_UID\" == \"$GROUP_UID\" ] || exit 1"
   tags:
     - "dind"
+    - "hifis"
 
 avro-thing-creation-e2e:
   image: "docker:${DOCKER_IMAGE_TAG}"

README.md

@@ -253,3 +253,16 @@ TSM_DIRECTORY (can be any dir)
 ├── tsm-extractor
 └── tsm-ufz-tsmdl
 ```
+
+# Keycloak as identity provider in development environment
+
+## Admin console
+
+- http://localhost:8081/admin/master/console/#/demo/ 
+  - or: http://keycloak:8081/admin/master/console/#/demo/ (if you updated /etc/hosts)
+  - or: http://keycloak:KEYCLOAK_PORT/admin/master/console/#/demo/ || http://localhost:KEYCLOAK_PORT/admin/master/console/#/demo/ (if you changed the port to something other than 8081) 
+- Credentials:
+  - User: `admin`
+  - Password: `admin`
+
+See [here](./keycloak/README.md) for further information regarding configuration and setup.
\ No newline at end of file

docker-compose-dev.example.yml

@@ -57,3 +57,45 @@ services:
   tsmdl:
     volumes:
       - "${DEV_TSMDL_SRC_DIR:-../tsm-ufz-tsmdl/app}:/app"
+
+  keycloak:
+    image: "quay.io/keycloak/keycloak:24.0.3"
+    ports:
+      - "${KEYCLOAK_PORT}:${KEYCLOAK_PORT}"
+    user: "${KEYCLOAK_UID:-1000}:${KEYCLOAK_GID:-1000}"
+    environment:
+      KEYCLOAK_ADMIN: "admin"
+      KEYCLOAK_ADMIN_PASSWORD: "admin"
+      KC_DB: postgres
+      KC_DB_URL_DATABASE: keycloak
+      KC_DB_SCHEMA: public
+      KC_DB_URL: jdbc:postgresql://keycloak-postgres/keycloak
+      KC_DB_USERNAME: keycloak
+      KC_DB_PASSWORD: password
+    volumes:
+      - "./keycloak/keycloak-init.json:/opt/keycloak/data/import/keycloak-init.json"
+    command:
+      - start-dev
+      - --import-realm
+      - --http-port=${KEYCLOAK_PORT}
+    depends_on:
+      keycloak-postgres:
+        condition: service_healthy
+
+  keycloak-postgres:
+    image: "postgres:14-alpine"
+    environment:
+      POSTGRES_DB: keycloak
+      POSTGRES_USER: keycloak
+      POSTGRES_PASSWORD: password
+    volumes:
+      - "keycloak-data:/var/lib/postgresql/data/"
+    healthcheck:
+      test: [ "CMD-SHELL", "pg_isready -U keycloak" ]
+      interval: "${POSTGRES_HEALTHCHECK_INTERVAL}"
+      timeout: 5s
+      retries: 5
+
+volumes:
+  keycloak-data:
+

keycloak/README.md

+# Keycloak
+
+## Purpose
+The purpose of this keycloak is to avoid the dependency to the [Helmholtz AAI](https://hifis.net/aai/) and to mock its behavior.
+
+## Documents
+
+- [Configuration.md](./keycloak/docs/Configration.md)
+  - Contains all information how the Keycloak was setup
+- [Important Information.md](./keycloak/docs/Important Information.md)
+  - Contains information you need to connect an application to a client and where to find that information
+- [Specification for timeIO.md](./keycloak/docs/Specification for timeIO.md)
+  - Contains information specific for the time.IO client:
+    - Created groups
+    - Created users
+    - __Which environment variables you need to set__
\ No newline at end of file

keycloak/docs/Configuration.md

+# Configuration 
+
+## How the Keycloak was setup to mimic the Helmholtz AAI
+
+### Prerequisite
+
+- Update your local `/etc/hosts` to include:
+  - `127.0.0.1       keycloak`
+
+### Start the docker compose setup
+
+- Start docker compose setup
+
+  - `docker compose up -d`
+
+- visit: http://keycloak:`KEYCLOAK_PORT`
+  - login using 
+    - username: `admin` 
+    - password: `admin`
+  - __Note__ these are set by the environment variables `KEYCLOAK_ADMIN` and `KEYCLOAK_ADMIN_PASSWORD`
+
+### Create a realm
+
+  - click the drop-down-menu on the left side (right under the header) where currently `Keycloak` is selected
+    - click the button `Create realm`
+  - name it `demo`
+  - click `create`
+![1001](./../media/1001.png)
+![1002](./../media/1002.png)
+
+### Create a client communicating over backend channel
+  - go to `clients` in left menu
+  - click `create client`
+  - fill in the form (see pictures)
+  - __Note__ It is important to set the `Client authentication` __on__ for the client to enable the authentication over the backend channel
+
+![1003](./../media/1003.png)
+![1004](./../media/1004.png)
+![1005](./../media/1005.png)
+![1006](./../media/1006.png)
+
+### Create client scopes
+  - these scopes will be needed to mock some properties of the helmholtz aai
+  - you will need the scopes `openid`,  `eduperson_entitlement`,  `eduperson_principle_name`, `eduperson_unique_id`
+  - go to `client scopes` in left menu
+####   `openid` scope:
+- __Note__ This scope is needed (and must also be send by the client) to provide access to the `userinfo` endpoint
+- click `create client scope`
+  - name: openid
+  - Type: Default
+
+![1007](./../media/1007.png)
+
+#### `eduperson_principal_name` scope
+- click `create client scope`
+  - name: `eduperson_principal_name`
+  - Type: default
+- add mapper to `eduperson_principal_name` scope
+  - __After save__: Go to the `Mapper` tab of the saved scope
+  - click `Configure a new mapper`
+  - Select `user property`
+  - Fill in the form:
+    - Name: `Mapper eduperson_principal_name`
+    - Property: `username`
+    - token claim name: `eduperson_principal_name`
+
+![1008](./../media/1008.png)
+![1009](./../media/1009.png)
+
+####  `eduperson_entitlement` scope
+- click `create client scope`
+  - name: `eduperson_entitlement`
+  - Type: default
+- add mapper to `eduperson_entitlement` scope
+  - __After save__: Go to the `Mapper` tab of the saved scope
+  - click `Configure a new mapper`
+  - Select `Group Membership`
+  - Fill in the form:
+    - Name: `Mapper eduperson_entitlement`
+    - token claim name: `eduperson_entitlement`
+    - full group path: `off`
+
+![1010](./../media/1010.png)
+![1011](./../media/1011.png)
+
+#### `eduperson_unique_id` scope
+- click `create client scope`
+  - name: `eduperson_unique_id`
+  - Type: default
+- add mapper to `eduperson_unique_id` scope
+  - __After save__: Go to the `Mapper` tab of the saved scope
+  - click `Configure a new mapper`
+  - Select `user property`
+  - Fill in the form:
+    - Name: `Mapper eduperson_unique_id`
+    - Property: `id`
+    - token claim name: `eduperson_unique_id`
+    ![1012](./../media/1012.png)
+
+### Add the scopes to your client
+
+  - go to `clients` in left menu
+  - select you client, e.g. `timeIO-client`
+  - go to `client scopes` tab
+  - click `add client scope`
+  - select the previously created ones
+    - select type: `default`
+
+![1015](./../media/1015.png)
+![1016](./../media/1016.png)
+
+### Create a group
+
+  - you can adopt the procedure to create more
+  - __Note__ 
+    - The original Helmholtz Virtual Organization groups look something like this: `urn:geant:helmholtz.de:group:VO-Name:grouup-name#login.helmholtz.de`, so the name of the groups must follow the naming schema
+  - go to `Groups` in left menu
+  - click `create group`
+  - fill in the form:
+    - Name: `a:a:a:group:VO:Group1#`
+
+### Create a user
+  - you can adopt the procedure to create more
+  - go to `Users` in left menu
+  - click `Create new user`
+  - fill in the form:
+    - Email verified: `yes`
+      - __Note__ This is import to set to `yes` otherwise the user has to confirm its email adress
+    - username: `user1`
+    - email: `user1@provider.org`
+    - first name: `User1`
+    - last name `One`
+    - Groups: 
+      - click `join groups`
+      - selected the previous created group
+      - click `join`
+    - click `create`
+
+![1013](./../media/1013.png)
+
+  #### Add user credentials
+- __After create__: Go to `credentials` tab
+  - click `Set password`
+  - fill in the form:
+    - Password: `password`
+    - Password confirmation: `password`
+    - Temporary: `Off`
+      - __Note__: If you set this to `on` the user has to update its password on first login. With off you can use the dummy password everytime
+
+![1014](./../media/1014.png)
+
+### Update Authentication flow
+- go to `Authentication` in left menu
+- click on `browser`
+- Set `Requirement` for `Cookie` to `Disabled
+
+__Reason for doing this__: 
+
+If the cookie is disabled, you have to enter a password everytime you login. This is important, because otherwise it would not be possible to change the user. And it's acutal the same behavior as the aai.
+
+![1017](./../media/1017.png)
+![1018](./../media/1018.png)
+
+### Update SSO Session Max
+- go to `Realm Settings` in left menu
+- click on tab `Sessions`
+- Set `SSO Session Max` to `10` `Seconds`
+
+__Reason for doing this__: 
+There was a problem, when you logged in with one user in the sms and then  logged out and tried to login with another user:
+
+```
+You are already authenticated as different user 'user1' in this session. Please sign out first.
+```
+This change fixes this.
+
+![1019](./../media/1019.png)
+
+### Export the settings to an file
+  - this file can be used to start the keycloak server and fill it with some initial data    
+    - prerequisite - docker compose work:
+      - first create a ` keycloak-init.json` in you local docker compose setup
+      - mount it to the keycloak service using volumes
+      - ```yaml
+            volumes:
+              - "./keycloak/keycloak-init.json:/opt/keycloak/data/import/keycloak-init.json"
+        ```
+    - Run the following command to export the file
+  ~~~bash
+  docker compose exec keycloak /opt/keycloak/bin/kc.sh export --file /opt/keycloak/data/import/keycloak-init.json --users same_file --realm demo
+  ~~~
+  - to use that file on start use the following service command in the `docker-compose.yml`
+  - ```yaml
+        command:
+          - start-dev
+          - --import-realm     
+    ```
+
+- __Important/Troubleshooting export file__
+
+  - If you export the json file it could be possible that you could have some problems, when you start the keycloak using it
+
+  - See: https://howtodoinjava.com/devops/keycloak-script-upload-is-disabled/
+
+  - To solve the problem:
+
+    - > So, to solve the “Script upload is disabled” error **clean the realm JSON file by removing the ‘`authorizationSettings`‘ node** altogether. After cleaning the realm file, the import will run successfully and the server will start.

keycloak/docs/Important Information.md

+# Important Information
+
+With important information we mean theses kind of information you'll need to connect you application with the client to keycloak.
+
+- Make sure you have selected the right realm `demo`
+- The Client ID
+  - Go to `clients` in left menu
+  - Select the client you want, e.g. `timeIO-client`
+  - there you find the `Client ID`
+- The Client Secret
+  - Go to `clients` in left menu
+  - Select the client you want, e.g. `timeIO-client`
+  - go to `Credentials` tab
+  - there you find the `Client Secret`
+- Well Known Url
+  - Go to `realm settings` in left menu
+  - in the current `general` tab you find the `Endpoints`
+  - Select the `OpenID Endpoint Configuration`
+  - e.g. `http://keycloak:KEYCLOAK_PORT/realms/demo/.well-known/openid-configuration`

keycloak/docs/Specification for timeIO.md

+# Specification for timeIO configuration
+
+## Environment Variables
+
+```
+DJANGO_HELMHOLTZ_CLIENT_ID=timeIO-client
+DJANGO_HELMHOLTZ_CLIENT_SECRET=pUolEnz1Ve5djx9oUpw1KBgB0YgIlUOQ
+DJANGO_HELMHOLTZ_AAI_CONF_URL=http://keycloak:KEYCLOAK_PORT/realms/demo/.well-known/openid-configuration
+
+ALLOWED_VOS=UFZ-Timeseries-Management, VO
+```
+ __IMPORTANT NOTE FOR ORCHESTRATION:__
+- `DJANGO_HELMHOLTZ_AAI_CONF_URL=http://keycloak:KEYCLOAK_PORT/realms/demo/.well-known/openid-configuration`
+  - this is the internal portal of the container, so adapt to your `KEYCLOAK_PORT`
+## Created groups:
+
+  - `a:a:a:group:VO:Group1#`
+  - `a:a:a:group:VO:Group2#`
+  - `a:a:a:group:VO2:Group1#` (can be used as a not allowed VO)
+
+- __Note__:
+
+  - Please note that the `VO` name must be added to the `ALLOWED_VOS` environment variable of the `tsm-frontend` to make the group selectable
+
+## Created user:
+  - __Note__:  All users have the password `password`
+
+
+| Username | Groups                                             | Purpose                                    |
+| -------- | -------------------------------------------------- | ------------------------------------------ |
+| `user1`  | `a:a:a:group:VO:Group1#`, `a:a:a:group:VO:Group2#` | a user in two valid groups                 |
+| `user2`  | `a:a:a:group:VO:Group1#`                           | a user in one valid group, same as `user1` |
+| `user3`  | `a:a:a:group:VO2:Group1#`                          | a user in a not valid virtual organization |
+| `user4`  | -                                                  | a user not in any group                    |
+