Exa-AToW : Architectures and Tools for Large-Scale Workflows

Targeted project

Overview

Scientific cross-facility workflows make it possible to harness multiple facilities across the digital continuum (e.g., data centers, HPC centers) within unified processing sequences. However, providing an operational cross-facility workflow platform faces numerous obstacles, such as cybersecurity, heterogeneous system governance, and abstract representation of the continuum, among others.

This is a classic “collaborative system of systems” problem, in which political and technical considerations must align to deliver the required capabilities. This challenge is not new, and despite strong incentives to provide scientists with simple workflow tools —along with the many available technologies (largely driven by the Cloud community)— it has not yet been fully achieved. The stringent operational constraints of GENCI’s supercomputers make this challenge particularly difficult.

The Exa-AToW project addresses this collaborative system-of-systems challenge through a genuinely collaborative approach, bringing together French supercomputing centers, scientific communities (radio astronomy, earth science, high-energy physics, materials science), and computer science expertise.

Partners & Organisation

Consortium & Workpackages

Partners

Irisa

Université de Rennes

François Bodin, Mark Asch

OSU Pythéas

CNRS

Didier Mallarino

Idris

CNRS

Guillaume Harry

Modeling and Exploration of Materials Laboratory

CEA

Thierry Deutsch

LIG

Inria

Olivier Richard

Workpackages

WP1

Overall architecture and federation governance

Lead University of Rennes

WP2

Cyber-security

Lead Idris

WP3

Support tools for workflows management, monitoring and reproducibility

Lead CEA

WP4

Data logistics and data life cycle

Lead University of Rennes

WP5

Systems tools for resource allocation, federation & deployment

Lead Inria

WP6

Energy and environmental sustainability

Lead OSU Pythéas

WP7

Demonstrators and coordination with the other Numpex PCs

Lead University of Rennes

Results to date

Results & Impact

To date, Exa-AToW has delivered proofs of concept for HPC-application-as-a-service, a digital shadow representing the continuum, associated ontologies, and an environmental impact computation library. These developments have been driven by real-world scientific applications such as DDF (radio astronomy) and openQCD (high-energy physics), among others. Tools such as remotemanager, which enables the use of supercomputers via Jupyter notebooks, have also been released.

While continuing these ongoing efforts, Exa-AToW is now exploring agentic AI technologies to facilitate the design and implementation of cross-facility workflows.