A transferable infrastructure to enable FAIR timeseries data

The field of earth system sciences relies heavily on the collection, analysis, and interpretation of data to understand complex spatiotemporal environment processes and predict future trends. Time series data, which captures measurements or observations at regular intervals over time, plays a crucial role in elucidating patterns, detecting changes, and informing decision-making in various environmental domains. However, the effective storage and management of time series data present significant challenges that necessitate the development of robust and scalable data infrastructures.

A well-designed data infrastructure is crucial to ensure the reliability, accessibility, interoperability, and sustainability of time series data across different domains and scales. It enables efficient data collection through automated sensing technologies, standardized data exchange protocols, and quality control procedures. Moreover, robust time series data infrastructures facilitate the integration of data from diverse sources into distributed data infrastructures on national and/or continental scales and advances the dissemination of data to a wide range of stakeholders including scientists, policymakers, resource managers, and the general public.

The focus lies on enhancing accessibility for end users, operators, system integrators, and maintainers. The time series management system encompasses all essential components. It features a user-centric web-based frontend, a versatile data integration layer, a robust time series database, efficient object storage, real-time quality control, and comprehensive data visualization capabilities. It supports modern and classical data transfer protocols, and ensures compliance with OGC standards for data access. Moreover, our fully integrated and containerized solution offers the convenience of swift and effortless deployment within minutes and allows the seamless integration with existing services such as databases, identity providers, and object storages.

This innovative infrastructure has the potential to significantly enhance environmental data management, fostering more efficient research and facilitating informed decision-making processes.

Features

• File based raw data ingest with common protocols (FTP, SFTP, S3)
• Stream based raw data ingest with MQTT (TLS required)
• HIFIS/Helmholtz AAI Login for end users - all people associated to institutions that are connected to GÉANT Edugain can log in from anywhere
• Grouping data and things by projects that are organized as HIFIS/Helmholtz AAI Virtual Organisation (VO)
• Instant data visualisation of incoming data in Grafana: New Datastreams automatically get a dynamic dashboard with no need but the ability for further configuration and customisation
• Integration layer to link datastreams to sensor metadata in the Sensor Management System (SMS)
• Dynamic OGC STA endpoints with SMS metadata integration per project to access data in a standardized way