Analysing collision data efficiently is essential for HEP. Since the LHC start-up, the time needed between dataset arrival and the delivery of scientific results was greatly reduced. Analysis software significantly improved and approaches to analysis were streamlined: for example, analysis trains were introduced by experiments to combine analysis steps from different users and execute them together, reducing access to the same input data. Nevertheless, such improvements will not be sufficient in the future.
Detectors at future hadron colliders will lead to one to two orders of magnitude data rate increases and a veritable escalation of data complexity. The increase poses an unprecedented challenge to the HEP data processing chain. The question of how to analyse data volumes at the HL-LHC and FCC-hh scale is not yet answered and is crucial for the success of these endeavours. What is clear is that future complexity cannot be handled by requiring more advanced programming skills from researchers. Already their time and focus are dissipated understanding how the analysis should be implemented in code rather than what steps are needed for an optimal study.
The R&D programme on future analysis I/O focuses particularly on
- increasing the data reading rate for modern and future storage devices, such as SSDs, NVRAM, and object stores
- improving programming models to make it easier and faster for researchers to ask the right questions to the data.
