Get started with ZID/TSM
The orchestration repository is the place where all the little ZID/TSM components are combined to a running system. This is achieved by putting the docker images of the ZID/TSM components together in a docker compose file so you can start them with a single command and without deeper knowledge of every single part of it.
Used ZID/TSM components in that repo:
1. Create environment/config file from example:
cp .env.example .env
The settings from the example are ok for local testing and development. Postgres, Minio and MQTT services are exposed on localhost, so you can access them with clients from your machine.
When using this in (semi-) production (e.g. on a server) some settings, especially the passwords should be changed! Also keep in mind, that you will need encryption when exposing the services: For example, minio HTTP ports will need a TLS proxy (i.e. nginx) with certificates (i.e. issued by DFN PKI).
2. Run all the services and have fun
docker-compose up -d
It will take some seconds until everything is up.
3. Create a thing
A thing in ZID/TSM/STA sense is an entity that is producing time series data in one or more data streams. In ZID/TSM we follow the approach, that an end user is able to create a new thing and all its settings for its infrastructure like database credentials or parser properties. When somebody enters or changes settings of a thing these changes are populated to action services by MQTT events.
As long as ZID/TSM doesn't have a graphical end user frontend we have to produce events by ourselves. We directly use the MQTT container for that:
cat thing-event-msg.json | docker-compose exec -T mqtt-broker sh -c "mosquitto_pub -t thing_creation -u \$MQTT_USER -P \$MQTT_PASSWORD -s"
The dispatcher action services will create
- a new minio user and bucket:
- a new postgres database role and schema:
- postgresql://myfirstproject_6185a5b8462711ec910a125e5a40a845:d0ZZ9d3QSDZ6tXIZTnKRY1uVLKIc05GmQh8SA36M@postgres/postgres
- and a thing entity with (hopefully) all the necessary properties
in the new
thing
table
4. Upload data
Now you can go to the fresh new bucket in the
minio console
and upload a csv
file.
The dispatcher action service called run-process-new-file-service gets notified by a MQTT event produced by minio and will forward the file resource and the necessary settings to the scheduler. The scheduler starts the extractor wo will parse the data and write it to the things database.
5. Clean up
To temporary stop the containers and services use docker-compose stop
.
When you're ready or destroyed your setup while playing around you can reset everything by kicking away the containers and removing all data:
docker-compose down --timeout 0 -v --remove-orphans && ./remove-all-data.sh
All data is lost with this. Be careful!
Using SFTP and FTP for uploads
The minio object storage provides SFTP and FTP services. Its automatically equipped with self signed TLS certs for FTP and a generated SSH host key. Please change the TLs certificates to some officially signed, you can take the same certs that are used by the proxy.
You can directly use the minio accounts, but it would be better to use minio service accounts for authentication.
lftp
Testing FTP service with With the previously generated account and the default settings:
lftp -p 40021 thedoors-057d8bba-40b3-11ec-a337-125e5a40a849@localhost -e "set ssl:ca-file ./data/minio/certs/minio-ftp.crt"
In development mode with self signed certificate we have to define the CA.
sftp
Testing SFTP service with With the previously generated account and the default settings:
sftp -P 40022 thedoors-057d8bba-40b3-11ec-a337-125e5a40a849@localhost
Further thoughts and hints
Configuring and operating Mosquitto MQTT broker
General
When using in production it is recommended to use the mosquitto.tls.conf
template (change
MOSQUITTO_CONFIG
in your .env
file) to enable encrypted connections by tls.
When started the first time it generates a password database
(data/mosquitto/passwd/mosquitto.passwd
) with the credentials from the environment. Later
changes of the password and user in env do not have any effect to the password file but will
break the health check of the service. To change passwords or add users use the
mosquitto_passwd
command from inside the container:
docker-compose run --rm mqtt-broker mosquitto_passwd -b /mosquitto-auth/mosquitto.passwd "user" "password"
With interactive password input:
docker-compose run --rm mqtt-broker mosquitto_passwd /mosquitto-auth/mosquitto.passwd "user"
Example for adding a new user and an acl to publish data
- Start the mqtt-broker service with
docker-compose up mqtt-broker
at least once the create the initialmosquitto.passwd
andmosquitto.acl
files. - Call
docker-compose exec mqtt-broker bash -c $'echo `echo -n "MY_NEW_MQTT_USER:" && /mosquitto/pw -p "MY_NEW_MQTT_PASSWORD"` >> /mosquitto-auth/mosquitto.passwd'
to add the new user with its password - Restart the mqtt-broker service
docker-compose restart mqtt-broker
- From now on you should be able to publish to the new users topic namespace:
echo "very nice data!" | docker-compose exec -T mqtt-broker sh -c "mosquitto_pub -t mqtt_ingest/MY_NEW_MQTT_USER/beautiful/sensor/1 -u MY_NEW_MQTT_USER -P MY_NEW_MQTT_PASSWORD -s"
docker-compose logs --follow mqtt-cat
mosquitto_ctrl
mosquitto_ctrl seems to be a new API to configure the mosquitto server on runtime without to reload it when things change.
Mosquitto auth plugins
For dynamic acls from database: https://gist.github.com/TheAshwanik/7ed2a3032ca16841bcaa
Minio
-
Yes, we really need four volumes, otherwise object lock will not work.
-
Find the current event ARN to configure bucket notifications:
mc admin info myminio/ --json | jq .info.sqsARN
Naming conventions
Human readable ID for projects and things: Use UUID as suffix and sanitized name to fill it from the left until it is 63 chars long.
import re
def slug(self):
return re.sub(
'[^a-z0-9_]+',
'',
'{shortname}_{uuid}'.format(shortname=self.name[0:30].lower(), uuid=self.uuid)
)
# Or with minus chars at all but less space for the name
def slug_with_minus(self):
return re.sub(
'[^a-z0-9\-]+',
'',
'{shortname}-{uuid}'.format(shortname=self.name[0:26].lower(), uuid=self.uuid)
)
Enable TLS Security for Postgres database
For secure connections over the network you need transport security like
https
. To achieve this you need certificates of a public key
infrastructure (PKI) like the
DFN PKI.
Once you have a private key and a public certificate you can enable security by
- changing
POSTGRES_TLS_CERT_PATH
to the path of your certificate file - changing
POSTGRES_TLS_KEY_PATH
to the path of your private key file - uncommenting the line beginning with
POSTGRES_EXTRA_PARAMS
in the .env
file of your deployment.
Now you're able to access the minio service with https
. The postgres
database will enforce encryption but you need to enable
full-verification
mode in client to also check the identity of the
server.