Pubblicazioni

Versione: 1.0 Approvato

LDO-ID 2021-10-11-01

Titolo Distributed Workflows with Jupyter

Abstract

The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow system is evaluated on common scenarios for High Performance Computing (HPC), showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations

Autori Colonnelli Iacopo, Rabellino Sergio, Cantalupo Barbara, Spampinato Concetto, Di Carlo Rosario, Morelli Roberto, Magini Nicolò, Cavazzoni Carlo, Aldinucci Marco

Tipologia Paper for Specialistic Magazine

Media Future Generation Computer System (Elsevier B.V., Volume 128, March 2022)

Sito web

https://www.sciencedirect.com/science/article/pii/S0167739X21003976

Anno 2022