Publications

Version: 1.0 Approved

LDO-ID 2021-10-11-01

Title Distributed Workflows with Jupyter

Abstract

The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow system is evaluated on common scenarios for High Performance Computing (HPC), showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations

Authors Colonnelli Iacopo, Rabellino Sergio, Cantalupo Barbara, Spampinato Concetto, Di Carlo Rosario, Morelli Roberto, Magini Nicolò, Cavazzoni Carlo, Aldinucci Marco

Type Paper for Specialistic Magazine

Media Future Generation Computer System (Elsevier B.V., Volume 128, March 2022)

Web site

https://www.sciencedirect.com/science/article/pii/S0167739X21003976

Anno 2022