Structure of TimeIO frontend

Csv Parser

A CSV Parser is responsible to read your CSV-files correctly. Later you will need to select a parser when you create a Thing. So this is the first to start with.

In the following we will have a look at the fields which need to be filled out, when creating a parser:

• Project: The parser will only be available for users within this project
• Timestamp format: The expected format follows https://strftime.org/ (also see interactive https://www.strfti.me/) [with some minor exeptions - see Example 3]
  • Example 1:
    • If your timestamp looks like this 6/13/2024 3:45:00 PM
    • The Timestamp format is : %m/%d/%Y %I:%M:%S %p
  • Example 2:
    • Timestamp as in file : 2025-02-10 00:15:30
    • Timestamp format :%Y-%m-%d %H:%M:%S
  • Example 3:
    • Timestamp as in file : "2025-02-10 00:15:30"
    • Timestamp format : %Y-%m-%d %H:%M:%S
    • Note in this example, that the quotationmarks " do not appear in the timestamp format.
• Column delimiter: e.g. typical csv delimiter: , ; \t
• Timestamp column(s) :
  • Note that the counter starts with zero.
  • That means: If your Timestamp is in column one 1 in your file. You have to write a Zero 0 in this field.
  • It is possible to specify different columns if the date/time format is spread across multiple columns. To do so, specify the column and the corresponding format. Additional columns can be added by pressing the + button (Add another Timestamp column).
• Footlines: Set the number of footlines which have to be excluded from data. Note: Empty trailing lines are excluded by default without the footline-setting.
• Decimal sign: standard is the "Point"
  • So far this can only be changed by admins * However if requested more often this might change in future
  • It might happen sometimes by chance that you have a line which states
    • "Pandas read csv: {"decimal":","}
    • In this case you have the Komma as decimal sign. Please create a new parser to have the standard "Point" as decimal sign

You can check if your time format fits the right format by using this tool: https://www.strfti.me/

MQTT device types

QA/QC Settings

Things

In this area you can manage your Things.

• Existing Things are displayed here and can be searched and edited.
• New Things can be created.
• A Thing describes a data stream that comes from a defined source (e.g. a file, a file server or a specific server with a programming interface).
• Various metadata can be attached to this data stream and supplemented with specific attributes, e.g. for the data format. But this is configuered elswere.

Search Thing

To search for a Thing, you have the option of restricting the display using a filter. The following filter methods are currently available:

1. Filter by ingest type
2. Filter by project

Please note that you will only see Things that belong to projects in which you are registered. The entire project management is administered via Helmholtz-VO. For more information, see here

Create Thing

To create a new Thing press the button add Thing in the upper right corner. A new tab opens. To successfully recreate a thing you need to provide the following information:

Label

Project/Group

• Maximum name length accepted in TimeIO: 60 characters
• The Project Group is coming from a Data Project. See RDM Page for more infos

Description

Ingest type

SFTP Settings

• Filename pattern:
  • You can work with wildcards by using the asterix sign *
  • Example 1:
    • You only want to consider files that start with abc, contain fgh somewhere in the middle and end with .xyz,
    • then the filename pattern would be abc*def*xyz

External SFTP Server Settings

To specify the path on the sFTP Server which should be synced, you can use absolute and relative descriptions.

• Example 1
  • Relative path
    • Fileserver uri: sftp://files.ufz.de
    • User: met-mosep
    • Path: Logger/rawdata
  • Absolute path:
    • Fileserver uri: sftp://files.ufz.de
    • User: met-mosep
    • Path: /met-mosep/Logger/rawdata

If you want to sync the entire home directory you could as well use any of the two options.

• Example 2

  • Relative path
    • Fileserver uri: sftp://files.ufz.de
    • User: met-mosep
    • Path: .
  • Absolute path
    • Fileserver uri: sftp://files.ufz.de
    • User: met-mosep
    • Path: /met-mosep/

Public Key This is the public key generated by TimeIO. You have to place this key on your external SFTP such that TimeIO is allowed to access it.

• On your SFTP (eg. files.ufz.de) open the hidden folder .ssh with the file authorized_keys
• Copy the Public Key from Timeio to a new line and save the file
• You might add a comment at the end of the line, so in the future you know where this key belongs to.
• Example:

Public Key

Public Key with comment

Edit Thing

To edit an existing thing, click on it in the selection list. A new tab will open. For more information, see create Thing.

Delete Thing

To delete an existing thing, you must mark it in the selection list. You can then delete it by selecting the action Delete selected Thing. Please note that you must have the required permissions to do this.