Indexat a

Llicència i ús

Citacions

Altmetrics

Grant support

This work has been partly funded by the EU Horizon programme under the grant agreements: 101092644 (NearData), 101092646 (CloudSkin), and 101093110 (EXTRACT). German T. Eizaguirre is recipient of a pre-doctoral FPU grant from the Spanish Ministry of Universities (ref. FPU21/00630). Special thanks to Pau Balanza-Malagelada and Jordi Canosa-Casellas for their support in the experimentation phase.

Anàlisi d'autories institucional

Compartir

Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track)

Publicat a: 15-21 - 2024-01-01 (), DOI: 10.1145/3700824.3701095

Autors: Eizaguirre, German T; Barcelona-Pons, Daniel; Arjona, Aitor; Vernik, Gil; Garcia-Lopez, Pedro; Alexandrov, Theodore

Afiliacions

Resum

The scalability of cloud functions makes them a convenient backend for elastic data analytics pipelines where parallelism changes drastically from one stage to the next. However, cloud functions require intermediate storage systems for communication, which limits the efficiency of stateful operations. Furthermore, cloud functions are expensive, which reduces the cost-effectiveness of pure serverless architectures. We propose a hybrid architecture for data analytics that uses cloud functions for embarrassingly parallel stages and virtual cloud instances for stateful operations under a unified serverless programming framework. Extending Lithops, a serverless programming library, we implement a parallel programming interface that proactively provisions serverless and serverful cloud resources with minimal user intervention. We validate the feasibility of a hybrid architecture, by comparing it to fully serverless and serverful versions of a production-level metabolomics pipeline. We show that mixing cloud functions with virtual instances increases the cost-effectiveness of the execution by up to 188.23% over the serverless implementation, while achieving a speedup of 3.64 compared to the serverful one.

Paraules clau