Scientific and Technical Literature

Full listing

Publications represent one of the most important forms of sharing knowledge generated within a company, in collaboration with other companies or with public research organizations.
This page presents Leonardo's technical-scientific production, relating to conference proceedings, book chapters, articles in specialized journals.
The list contains references to the work carried out since 2012 by Leonardo researchers, engineers and technicians, and authorized for publication.

For further information, please contact us:

technology_transfer@leonardo.com

Contact us

Version: 1.0 Approved

LDO-ID 2021PUB00184

Title Distributed Workflows with Jupyter

Abstract

The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-in-dependent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow system is evaluated on common scenarios for High Performance Computing (HPC), showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.

Authors Aldinucci, Marco; Cantalupo, Barbara; Rabellino, Sergio; Spampinato, Concetto; Di Carlo, Rosario; Magini, Nicolò; Morelli, Roberto; Colonnelli, Iacopo; Cavazzoni, Carlo

Type Paper for Specialistic Magazine

Media "Future Generation Computer System" Journal (Elsevier B.V.)

Web site

https://www.sciencedirect.com/science/article/pii/S0167739X21003976

Anno 2021

Cancel