HL-LHC Computing Review Stage 2, Common Software Projects: Data Science Tools for Analysis - INSPIRE

DataBETA

HL-LHC Computing Review Stage 2, Common Software Projects: Data Science Tools for Analysis

Jim Pivarski
(
- Princeton U.
)
(ed.)
,
Eduardo Rodrigues
(
- Liverpool U.
)
(ed.)
,
Kevin Pedro
(
- Fermilab
)
(ed.)
,
Oksana Shadura
(
- U. Nebraska, Lincoln
)
,
Benjamin Krikler
(
- Bristol U.
)

Feb 4, 2022

26 pages

Contribution to:

LHCC Review of HL-LHC Computing

e-Print:

2202.02194 [physics.data-an]

Report number:

FERMILAB-CONF-22-061-SCD

Experiments:

CERN-HL-LHC

View in:

reference search5 citations

Citations per year

Abstract: (arXiv)

This paper was prepared by the HEP Software Foundation (HSF) PyHEP Working Group as input to the second phase of the LHCC review of High-Luminosity LHC (HL-LHC) computing, which took place in November, 2021. It describes the adoption of Python and data science tools in HEP, discusses the likelihood of future scenarios, and recommendations for action by the HEP community.

activity report
CERN LHC Coll: upgrade
programming

References(58)

Figures(7)

[1]

ROOT - An Object Oriented Data Analysis Framework

Rene Brun
,
Fons Rademakers

[2]

HEP Software Foundation (HSF)

- https://hepsoftwarefoundation.org

[3]

The HEP Software Foundation Workshop

Amber Boehnlein

[4]

A Roadmap for HEP Software and Computing R&D for the 2020s

HEP Software Foundation

Collaboration

•

Johannes Albrecht
(
- Dortmund U.
)

et al.

- Comput.Softw.Big Sci. 3 (2019) 1, 7
•
e-Print:
- 1712.06982
•
DOI:
- 10.1007/s41781-018-0018-8

[5]

Data-Intensive ANAlysis for High Energy Physics (DIANA/HEP)

- http://diana-hep.org/

[6]

Institute for Research and Innovation in Software for High Energy Physics (IRIS-HEP)

- https://iris-hep.org/

[7]

The Scikit HEP Project -- overview and prospects

Eduardo Rodrigues
(
- Liverpool U.
)
,
Benjamin Krikler
(
- Bristol U.
)
,
Chris Burr
(
- CERN
)
,
Dmitri Smirnov
(
- Brookhaven
)
,
Hans Dembinski
(
- Tech. U., Dortmund (main)
)

et al.

- EPJ Web Conf. 245 (2020) 06028
•
e-Print:
- 2007.03577
•
DOI:
- 10.1051/epjconf/202024506028

[8]

Coffea: Columnar Object Framework For Effective Analysis

CMS

Collaboration

•

Nicholas Smith
(
- Fermilab
)

et al.

- EPJ Web Conf. 245 (2020) 06012
•
e-Print:
- 2008.12712
•
DOI:
- 10.1051/epjconf/202024506012

[9]

Nested data structures in array frameworks

- J.Phys.Conf.Ser. 1525 (2020) 1, 012053
•
DOI:
- 10.1088/1742-6596/1525/1/012053

[10]

PyHEPWorkshop

[11]

The evolution of software in high energy physics

Ren Brun
(
- CERN
)

- J.Phys.Conf.Ser. 396 (2012) 052016
•
DOI:
- 10.1088/1742-6596/396/5/052016

[12]

Software Challenges For HL-LHC Data Analysis

Guilherme Amadio
(
- CERN
)
,
Sitong An
(
- CERN and
- Carnegie Mellon U.
)
,
Bertrand Bellenot
(
- CERN
)
,
Jakob Blomer
(
- CERN
)
,
Philippe Canal
(
- Fermilab
)

et al.

e-Print:
- 2004.07675

[13]

Evolution of ROOT package management

Oksana Shadura
(
- Nebraska U.
)
,
Brian Paul Bockelman
(
- Unlisted, US, WI
)
,
Vassil Vassilev
(
- Princeton U.
)

- J.Phys.Conf.Ser. 1525 (2020) 1, 012050
•
e-Print:
- 1906.04622
•
DOI:
- 10.1088/1742-6596/1525/1/012050

[14]

Columnar data analysis with ATLAS analysis formats

ATLAS Software and Computing

Collaboration

•

Nikolai Hartmann
(
- Munich U.
)

et al.

- EPJ Web Conf. 251 (2021) 03001
•
DOI:
- 10.1051/epjconf/202125103001

[15]

Declarative Programming in Escher

J.W. Lloyd

[16]

The FAST-HEP toolset: Using YAML to make tables out of trees

Benjamin Edward Krikler
(
- Bristol U.
)
,
Olivier Davignon
(
- Ecole Polytechnique
)
,
Lukasz Kreczko
(
- Bristol U.
)
,
Jacob Linacre
(
- Rutherford
)

- EPJ Web Conf. 245 (2020) 06016
•
DOI:
- 10.1051/epjconf/202024506016

[17]

Analysis Description Languages for the LHC

et al.

- PoS LHCP2020 (2021) 065
•
e-Print:
- 2011.01950
•
DOI:
- 10.22323/1.382.0065

[18]

CutLang v2: Advances in a Runtime-Interpreted Analysis Description Language for HEP Data

et al.

- Front.Big Data 4 (2021) 659986,
- Front. Big Data 4:659986, 2021
•
e-Print:
- 2101.09031
•
DOI:
- 10.3389/fdata.2021.659986

[19]

FuncADL: Functional Analysis Description Language

- EPJ Web Conf. 251 (2021) 03068
•
e-Print:
- 2103.02432
•
DOI:
- 10.1051/epjconf/202125103068

[20]

RDataFrame: Easy Parallel ROOT Analysis at 100 Threads

Danilo Piparo
(
- CERN
)
,
Philippe Canal
(
- Fermilab
)
,
Enrico Guiraud
(
- CERN and
- Carl von Ossietzky U., Oldenburg (main)
)
,
Xavier Valls Pla
(
- CERN
)
,
Gerardo Ganis
(
- CERN
)

et al.

- EPJ Web Conf. 214 (2019) 06029
•
DOI:
- 10.1051/epjconf/201921406029

[21]

Survey of columnar file formats and the techniques they use

Jim Pivarski

[22]

Shared Memory High Throughput Computing with Apache Arrow™

Geoffrey Lentner

[23]

Apache Parquet

Deepak Vohra

[24]

ALICE Run 3 Analysis Framework

Anton Alkin
(
- CERN
)
,
Giulio Eulisse
(
- CERN
)
,
Jan Fiete Grosse-Oetringhaus
(
- CERN
)
,
Peter Hristov
(
- CERN
)
,
Maja Kabus
(
- Warsaw U. of Tech.
)

- EPJ Web Conf. 251 (2021) 03063
•
DOI:
- 10.1051/epjconf/202125103063

[25]

Uproot: rapidly moving data from ROOT to Numpy and Pandas

Jim Pivarski

1-25 of 58
1
2
3
25 / page