Columnar data processing for HEP analysis

In the last stages of data analysis, physicists are often forced to choose between simplicity and execution speed. In High Energy Physics (HEP), high-level languages like Python are known for ease of use but also very slow execution. However, Python is used in speed-critical data analysis in other fields of science and industry. In those fields, most operations are performed on Numpy arrays in an array programming style; this style can be adopted for HEP by introducing variable-sized, nested data structures. We describe how array programming may be extended for HEP use-cases and an implementation known as awkward-array. We also present integration with ROOT, Apache Arrow, and Parquet, as well as preliminary performance results.

programming
performance
data analysis method
numerical calculations
numerical methods

References(15)

Figures(0)

awkward-array

A guide to NumPy

Travis E

A programming language (John & Sons, Inc., New York,). [Google Scholar]

Kenneth E. Iverson

NEP 13: a mechanism for overriding ufuncs

Apache Arrow

- https://arrow.apache.org

The ROOT Object I/O System

Apache Parquet

Dask: Library for dynamic task scheduling

•
- https://dask.org

SPARSKIT: a basic tool kit for sparse matrix computations

uproot

Performance test 1: [Google Scholar]

- https://cernbox.cern.ch/index.php/s/uhAH75uGCddFlFL

Vectorized proof of concepts

Performance test 2: [Google Scholar]

- https://github.com/jpivarski/jupyter-performance-studies/blob/master/2018-09-10-jagged-reduction.ipynb