Publications

Version: 1.0 Approved

LDO-ID 2021PUB00184

Title Distributed Workflows with Jupyter

Abstract

The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-in-dependent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow system is evaluated on common scenarios for High Performance Computing (HPC), showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.

Authors Aldinucci, Marco; Cantalupo, Barbara; Rabellino, Sergio; Spampinato, Concetto; Di Carlo, Rosario; Magini, Nicolò; Morelli, Roberto; Colonnelli, Iacopo; Cavazzoni, Carlo

Type Paper for Specialistic Magazine

Media "Future Generation Computer System" Journal (Elsevier B.V.)

Web site

https://www.sciencedirect.com/science/article/pii/S0167739X21003976

Anno 2021