Iliana FAYOLLE                                     Univ. Lille, CNRS, Inria Lille, France

16 November 2026

Antoine GEIMER                                           Univ. Lille, CNRS, Inria Lille, France

Daniel DE ALMEIDA BRAGA         Univ. Rennes, CNRS, IRISA Rennes, France

ClĂ©mentine MAURICE                                  Univ. Lille, CNRS, Inria Lille, France

Practice Makes (Im)Perfect

A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks

System     

Sensitive

data

Side channels: Definition

Leak sources

Computation

1. Collect

2. Deduce

system

 

  Unintended sources of information leakage                          

                   sensitive data

 

 "                                                                                 from a                that may be    exploited to infer                          ."

1 / 12

Covert channels

Side channels

Introduction

System     

Sensitive

data

Leak sources

Computation

1. Collect

Side channels: Definition

Introduction

1 / 12

Outdated Methodologies

Introduction

Are they good proxies to evaluate the attack?

Usual Targets

Fingerprinting

Alexa top 100 websites

Control-Flow (CF)

v1.1.{0, 1}h, and v3.2

GnuPG v1.4.{13, 14}

...

OpenSSL v1.0.{1, 2}e,

Memory Access

WolfSSL v4.2.0, and v5.6.4

OpenSSL v1.0.1{e, f},

...

v0.9.8b, v1.1.0g, and v1.0.2

Flawed Benchmarking Practices

2 / 12

Outdated Methodologies

Outdated Methodologies

Unfair Comparisons

Introduction

Flawed Benchmarking Practices

2 / 12

Outdated Methodologies

Outdated Methodologies

Unfair Comparisons

Reproducibility Issues

Introduction

Flawed Benchmarking Practices

2 / 12

Outdated Methodologies

Outdated Methodologies

Unfair Comparisons

Reproducibility Issues

How should microarchitectural side-channel attacks be meaningfully evaluated and compared?

Introduction

2 / 12

Flawed Benchmarking Practices

Benchmarking Practices

Overview

In what forms do they manifest?

How prevalent are they?

How to avoid them?

Recommendations

Outdated Methodologies

Unfair Comparisons

Reproducibility Issues

Practices

3 / 12

   Number of papers per conference over time.

Selected 83 papers

 Architecture & Security

Top Rank venues

 

From 2014 to 2024

 

More Security papers

Some relevant papers are excluded

More Recent papers

 

       Microarchitectural side- & covert-channel attacks

 

Benchmarking Practices

Papers Selection

4 / 12

Our Philosophy:

Claim First, Criterion Second.

Benchmarking Practices

Survey Methodology

5 / 12

Our Philosophy:

Claim First, Criterion Second.

Benchmarking Practices

Survey Methodology

5 / 12

Our Philosophy:

Claim First, Criterion Second.

Benchmarking Practices

Survey Methodology

5 / 12

Flaws and their impact on evaluation properties.

Benchmark Omissions

Improper Comparisons

Missing Information

Annotation Labels

Flawed

Incomplete

Unclear

Partially Flawed / Underspecified

Ok

NA

Not Flawed

SoK: Benchmarking Flaws in Systems Security.

E. van der Kouwe, G. Heiser, D. Andriesse, H. Bos, C. Giuffrida. EuroS&P'19.

Benchmarking Practices

Taxonomy of Flaws

6 / 12

Distribution of the number of flaws per paper.

~ 5.72 flaws per paper

Distribution of side-channel attack types.

218 Attacks

71 Covert Channels

147 Side Channels

Benchmarking Practices

Survey Results: General Observations

7 / 12

Summary of benchmarking flaws across all evaluated papers.

Benchmarking Practices

Survey Results

8 / 12

Summary of benchmarking flaws across all evaluated papers.

Benchmarking Practices

Survey Results

8 / 12

Summary of benchmarking flaws across all evaluated papers.

Benchmarking Practices

Survey Results

8 / 12

Security

Benchmarking Practices

Architecture vs. Security Practices

Architecture

>> Flaws

Median number of (partial) flaws per paper across all conferences.

9 / 12

Benchmarking Practices

Architecture vs. Security Practices

>> Evaluated Platforms

Security

>> Flaws

Median number tested platforms per paper across all conferences.

Architecture

9 / 12

Covert channels and side-channel attack types across conferences.

Benchmarking Practices

Architecture vs. Security Practices

Security

>> Covert Channels

>> Flaws

>> Diversity of Side Channels

>> Evaluated Platforms

Architecture

9 / 12

>> Complete Artifacts

Benchmarking Practices

Architecture vs. Security Practices

>> Covert Channels

>> Diversity of Side Channels

>> Evaluated Platforms

Security

>> Flaws

Both

Security

Availability of artifacts over time.

Architecture

Architecture

The dashed lines show when artefact evaluation began.

9 / 12

>> (Incomplete) Artifacts

Benchmarking Practices

Practices Evolution over Time

Both

Security

Availability of artifacts over time.

The dashed lines show when artefact evaluation began.

Architecture

10 / 12

>> (Incomplete) Artifacts

>> Side-Channel Evaluations

Benchmarking Practices

Practices Evolution over Time

Evolution of median number of benchmarks per paper over time

across all conferences.

10 / 12

Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated Discovery.

D. Vansteenhuyse, A. Bols, L. Desmet, V. Le Pochat, J. Van Bulck, M. Bognar. USENIX'26.

Reproducibility in Applied Security Conferences: An 11-Year Review on Artifacts and Evaluation Committees.

D. Olszewski, A. Lu, A. Crowder, N. Bennett, S. Layton, S. Hrushikesh V. Bhupathiraju, T. Tucker, S. Kalgutkar, H. Ver Helst, C. Stillman, K. R. B. Butler,

S. Rampazzi, P. Traynor. ACM REP'25.

Lessons Learned from Five Years of Artifact Evaluations at EuroSys.

D. C. D’Elia, T. D. Doudali, C. Giuffrida, M. Matos, M. Payer, S. Pirelli, G. Portokalidis, V. Schiavoni, S. Signorello, A. Vahldiek-Oberwagner. ACM REP'25.

Reviewers

Benchmarking Practices

Recommendations

Share artifacts

Authors

Community

Standardize artifact evaluation

11 / 12

Invisible Probe: Timing Attacks with PCIe Congestion Side-channel.

M. Tan, J. Wan, Z. Zhou, Z. Li. S&P'21.

Reviewers

Benchmarking Practices

Recommendations

Share artifacts

Authors

Community

Standardize artifact evaluation

11 / 12

RELOAD+REFRESH: Abusing Cache Replacement Policies to Perform Stealthy Cache Attacks.

S. Briongos, P. MalagĂłn, J. M. Moya, T. Eisenbarth. USENIX'20.

Smallest distinguishable unit and its observation mapping.

Spatial Resolution

Reviewers

Benchmarking Practices

Recommendations

Share artifacts

Prioritize resolution metrics

Authors

Community

Standardize artifact evaluation

Promote comprehensive benchmarking practices

11 / 12

Cache Template Attacks: Automating Attacks on Inclusive Last-Level Caches.

D. Gruss, R. Spreitzer, S. Mangard. USENIX'15.

Smallest distinguishable unit and its observation mapping.

Spatial Resolution

Reviewers

Benchmarking Practices

Recommendations

Share artifacts

Prioritize resolution metrics

Authors

Community

Standardize artifact evaluation

Promote comprehensive benchmarking practices

11 / 12

Temporal Resolution

Time per sample and maximum sampling rate.

TunneLs for Bootlegging: Fully Reverse-Engineering GPU TLBs for Challenging Isolation Guarantees of NVIDIA MIG.

Z. Zhang, T. N. Allen, F. Yao, X. Gao, R. Ge. CCS'23.

Reviewers

Benchmarking Practices

Recommendations

Share artifacts

Prioritize resolution metrics

Authors

Community

Standardize artifact evaluation

Promote comprehensive benchmarking practices

11 / 12

Temporal Resolution

Time per sample and maximum sampling rate.

POWERT Channels: A Novel Class of Covert Communication Exploiting Power Management Vulnerabilities.

S. K. Khatamifard, L. Wang, A. Das, S. Köse, U. R. Karpuzcu. HPCA'19.

Reviewers

Benchmarking Practices

Recommendations

Share artifacts

Prioritize resolution metrics

Authors

Community

Standardize artifact evaluation

Promote comprehensive benchmarking practices

11 / 12

Reviewers

Demand relevant benchmarks

Focus on core properties

Benchmarking Practices

Recommendations

Share artifacts

Prioritize resolution metrics

Authors

Community

Standardize artifact evaluation

Promote comprehensive benchmarking practices

11 / 12

Share artifacts

Prioritize resolution metrics

Authors

Reviewers

Community

Demand relevant benchmarks

Focus on core properties

Standardize artifact evaluation

Promote comprehensive benchmarking practices

How should microarchitectural side-channel attacks be meaningfully evaluated and compared?

Benchmarking Practices

Takeaway

Practices (resolution metrics...)

Outdated Methodologies

Unfair Comparisons

Reproducibility Issues

Practices

12 / 12

Flaws and their impact on evaluation pro.

Backup

Benchmarking Practices

Backup

Benchmarking Practices

Summary of benchmarking flaws across all evaluated papers.

Backup

Benchmarking Practices

Prevalence of the 19 flaws in the 83 surveyed papers

Fig. 4: Distribution of reported properties in covert-channel evaluations.

Fig. 5: Distribution of side-channel attack types.

59 (Speed)

30 (Speed, Error Rate)

Reported Metrics per Paper

18 (Speed, Error Rate, ...)

218 Attacks

71 Covert Channels

147 Side Channels

Attacked Platforms per Paper

73 CPUs (and 65 Intel)

9 GPUs (and 3 iGPUs)

9 TEEs

Backup

Benchmarking Practices

CCS26

By Iliana Fayolle

CCS26

  • 20