fairmq.md

layout: spotlight

# -----------------------------------------------------------------------------
# Properties for spotlights list page
# -----------------------------------------------------------------------------

# The name of the software
name: FairMQ

# The date when the software was added to the spotlights YYYY-MM-DD
date_added: 2022-04-01

# Small preview image shown at the spotlights list page.
# Note: the path is relative to /assets/img/spotlights/
preview_image: fairmq/FAIR_Logo.svg

# One or two sentences describing the software
excerpt: FairMQ is designed to help implement large-scale data processing workflows needed in next-generation particle physics experiments.

# -----------------------------------------------------------------------------
# Properties for individual spotlights page
# -----------------------------------------------------------------------------
# Entries here will be shown in the green box on the right of the screen.

# Jumbotron (optional)
# The path is relative to /assets/img/jumbotrons/
title_image:

# Title at the top, inside the title-content-container
title: FairMQ - A message queuing library and framework

# Add at least one keyword
keywords:
    - framework
    - workflow library

# The Helmholtz research field
hgf_research_field: Matter

# At least one responsible centre
hgf_centers:
    - Helmholtz Centre for Heavy Ion Research (GSI)

# List of other contributing organisations (optional)
contributing_organisations:
    - name: European Organization for Nuclear Research (CERN)

# List of scientific communities
scientific_community:
    - Particle physics

# Impact on community (optional, not implemented yet)
impact_on_community: ESCAPE 2020

# An e-mail address
contact: D.Klein@gsi.de

# Platforms (optional)
# Provide platforms in this format
#   - type: TYPE
#     link_as: LINK
# Valid TYPES are: webpage, telegram, mailing-list, twitter, gitlab, github
# Mailing lists should be added as "mailto:mailinglist@url.de"
# More types can be implemented by modifying /_layouts/spotlight.html
platforms:
    - type: github
      link_as: https://github.com/FairRootGroup/FairMQ

# The software license (optional)
license: LGPL-3.0-only

# Is the software pricey or free? (optional)
costs: free

# What is this software used for in general (e.g. modelling)? (optional, not implemented yet)
software_type:
    - Data Analytics

# The applicaiton type (Desktop, Mobile, Web) (optional, not implemented yet)
application_type:
    - Desktop

# List of programming languages (optional)
programming_languages:
    - C++

# DOI (without URL, just 10.1000/1.0000000 ) (optional)
doi: 10.5281/zenodo.1689985

FairMQ in a nutshell

Next-generation Particle Physics Experiments at GSI/FAIR and CERN are facing unprecedented data processing challenges. Expected data rates require a non-trivial amount of high performance compute (HPC) resources in the order of thousands of CPU/GPU cores per experiment. Online (synchronous) data processing (compression) is crucial to stay within storage capacity limits. The complexity of tasks that need to be performed during the online data processing is significantly higher than ever before. Classically, complex tasks like calibration and track finding run in an offline (asynchronous) environment. Now they have to run online in a high performance and high throughput environment.

The FairMQ C++ library is designed to aid the implementation of such large-scale online data processing workflows by

• providing an asynchronous message passing abstraction that integrates different existing data transport technologies (no need to re-invent the wheel),
• providing a reasonably efficient data transport service (zero-copy, high throughput - TCP, SHMEM and RDMA implementations available),
• being data format agnostic (suitable data formats are usually experiment-specific), and
• providing further basic building blocks such as a simple state machine-based execution framework and a plugin mechanism to integrate with external config/control systems.

FairMQ is not an end-user application, but a library and framework used by software experts to implement higher-level experiment-specific applications.

Screenshot of AliceO2 Debug GUI showing the data processing workflow of a single event processing node.

The screenshot shows a visualization of the data processing workflow on a single Alice event processing node (The "O2 Framework debug GUI" tool in the screenshot is part of the AliceO2 project). Data logically flows along the yellow edges (in this case via the FairMQ shmem data transport) through the various processing stages of which some are implemented as GPU and others as CPU algorithms.

Initially designed with the online data processing in mind, FairMQ has been successfully used to parallelize offline simulation and analysis workloads as well.