Iliana FAYOLLE Univ. Lille, CNRS, Inria Lille, France
16 November 2026



Antoine GEIMER Univ. Lille, CNRS, Inria Lille, France
Daniel DE ALMEIDA BRAGA Univ. Rennes, CNRS, IRISA Rennes, France
Clémentine MAURICE Univ. Lille, CNRS, Inria Lille, France
Practice Makes (Im)Perfect
A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks




System





Sensitive
data


Side channels: Definition










Leak sources
Computation
1. Collect
2. Deduce
system
Unintended sources of information leakage
sensitive data
" from a that may be exploited to infer ."
1 / 12








Covert channels
Side channels
Introduction




System





Sensitive
data












Leak sources
Computation
1. Collect

Side channels: Definition
Introduction
1 / 12

















Outdated Methodologies

Introduction
Are they good proxies to evaluate the attack?
Usual Targets
Fingerprinting
Alexa top 100 websites
Control-Flow (CF)
v1.1.{0, 1}h, and v3.2
GnuPG v1.4.{13, 14}
...
OpenSSL v1.0.{1, 2}e,
Memory Access
WolfSSL v4.2.0, and v5.6.4
OpenSSL v1.0.1{e, f},
...
v0.9.8b, v1.1.0g, and v1.0.2
Flawed Benchmarking Practices
2 / 12



















Outdated Methodologies
Outdated Methodologies

Unfair Comparisons


Introduction
Flawed Benchmarking Practices
2 / 12

















Outdated Methodologies

Outdated Methodologies

Unfair Comparisons


Reproducibility Issues




Introduction
Flawed Benchmarking Practices


2 / 12

















Outdated Methodologies
Outdated Methodologies

Unfair Comparisons

Reproducibility Issues


How should microarchitectural side-channel attacks be meaningfully evaluated and compared?
Introduction
2 / 12
Flawed Benchmarking Practices
Benchmarking Practices
Overview






In what forms do they manifest?
How prevalent are they?
How to avoid them?
Recommendations

Outdated Methodologies
Unfair Comparisons
Reproducibility Issues
Practices
3 / 12
Number of papers per conference over time.
Selected 83 papers
Architecture & Security
Top Rank venues
From 2014 to 2024
More Security papers
Some relevant papers are excluded
More Recent papers


Microarchitectural side- & covert-channel attacks

Benchmarking Practices
Papers Selection

4 / 12


Our Philosophy:
Claim First, Criterion Second.
Benchmarking Practices
Survey Methodology
5 / 12



Our Philosophy:
Claim First, Criterion Second.
Benchmarking Practices
Survey Methodology
5 / 12




Our Philosophy:
Claim First, Criterion Second.
Benchmarking Practices
Survey Methodology
5 / 12

Flaws and their impact on evaluation properties.
Benchmark Omissions
Improper Comparisons
Missing Information
Annotation Labels
Flawed
Incomplete
Unclear
Partially Flawed / Underspecified
Ok
NA
Not Flawed
SoK: Benchmarking Flaws in Systems Security.
E. van der Kouwe, G. Heiser, D. Andriesse, H. Bos, C. Giuffrida. EuroS&P'19.
Benchmarking Practices
Taxonomy of Flaws
6 / 12
Distribution of the number of flaws per paper.
~ 5.72 flaws per paper

Distribution of side-channel attack types.
218 Attacks
71 Covert Channels
147 Side Channels
Benchmarking Practices
Survey Results: General Observations

7 / 12

Summary of benchmarking flaws across all evaluated papers.
Benchmarking Practices
Survey Results
8 / 12

Summary of benchmarking flaws across all evaluated papers.
Benchmarking Practices
Survey Results
8 / 12

Summary of benchmarking flaws across all evaluated papers.
Benchmarking Practices
Survey Results
8 / 12
Security
Benchmarking Practices
Architecture vs. Security Practices
Architecture
>> Flaws
Median number of (partial) flaws per paper across all conferences.

9 / 12
Benchmarking Practices
Architecture vs. Security Practices
>> Evaluated Platforms
Security
>> Flaws
Median number tested platforms per paper across all conferences.
Architecture

9 / 12
Covert channels and side-channel attack types across conferences.

Benchmarking Practices
Architecture vs. Security Practices
Security
>> Covert Channels
>> Flaws
>> Diversity of Side Channels
>> Evaluated Platforms
Architecture
9 / 12
>> Complete Artifacts
Benchmarking Practices
Architecture vs. Security Practices
>> Covert Channels
>> Diversity of Side Channels
>> Evaluated Platforms
Security
>> Flaws
Both
Security
Availability of artifacts over time.
Architecture



Architecture
The dashed lines show when artefact evaluation began.
9 / 12
>> (Incomplete) Artifacts
Benchmarking Practices
Practices Evolution over Time
Both
Security
Availability of artifacts over time.
The dashed lines show when artefact evaluation began.
Architecture



10 / 12
>> (Incomplete) Artifacts
>> Side-Channel Evaluations
Benchmarking Practices
Practices Evolution over Time
Evolution of median number of benchmarks per paper over time
across all conferences.

10 / 12
Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated Discovery.
D. Vansteenhuyse, A. Bols, L. Desmet, V. Le Pochat, J. Van Bulck, M. Bognar. USENIX'26.
Reproducibility in Applied Security Conferences: An 11-Year Review on Artifacts and Evaluation Committees.
D. Olszewski, A. Lu, A. Crowder, N. Bennett, S. Layton, S. Hrushikesh V. Bhupathiraju, T. Tucker, S. Kalgutkar, H. Ver Helst, C. Stillman, K. R. B. Butler,
S. Rampazzi, P. Traynor. ACM REP'25.
Lessons Learned from Five Years of Artifact Evaluations at EuroSys.
D. C. D’Elia, T. D. Doudali, C. Giuffrida, M. Matos, M. Payer, S. Pirelli, G. Portokalidis, V. Schiavoni, S. Signorello, A. Vahldiek-Oberwagner. ACM REP'25.
Reviewers

Benchmarking Practices
Recommendations
Share artifacts
Authors


Community
Standardize artifact evaluation
11 / 12

Invisible Probe: Timing Attacks with PCIe Congestion Side-channel.
M. Tan, J. Wan, Z. Zhou, Z. Li. S&P'21.
Reviewers

Benchmarking Practices
Recommendations
Share artifacts
Authors


Community
Standardize artifact evaluation
11 / 12

RELOAD+REFRESH: Abusing Cache Replacement Policies to Perform Stealthy Cache Attacks.
S. Briongos, P. MalagĂłn, J. M. Moya, T. Eisenbarth. USENIX'20.
Smallest distinguishable unit and its observation mapping.
Spatial Resolution
Reviewers

Benchmarking Practices
Recommendations
Share artifacts
Prioritize resolution metrics
Authors


Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Cache Template Attacks: Automating Attacks on Inclusive Last-Level Caches.
D. Gruss, R. Spreitzer, S. Mangard. USENIX'15.

Smallest distinguishable unit and its observation mapping.
Spatial Resolution
Reviewers

Benchmarking Practices
Recommendations
Share artifacts
Prioritize resolution metrics
Authors


Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Temporal Resolution
Time per sample and maximum sampling rate.
TunneLs for Bootlegging: Fully Reverse-Engineering GPU TLBs for Challenging Isolation Guarantees of NVIDIA MIG.
Z. Zhang, T. N. Allen, F. Yao, X. Gao, R. Ge. CCS'23.

Reviewers

Benchmarking Practices
Recommendations
Share artifacts
Prioritize resolution metrics
Authors


Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Temporal Resolution
Time per sample and maximum sampling rate.
POWERT Channels: A Novel Class of Covert Communication Exploiting Power Management Vulnerabilities.
S. K. Khatamifard, L. Wang, A. Das, S. Köse, U. R. Karpuzcu. HPCA'19.

Reviewers

Benchmarking Practices
Recommendations
Share artifacts
Prioritize resolution metrics
Authors


Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Reviewers

Demand relevant benchmarks
Focus on core properties
Benchmarking Practices
Recommendations
Share artifacts
Prioritize resolution metrics
Authors


Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Share artifacts
Prioritize resolution metrics
Authors


Reviewers

Community
Demand relevant benchmarks
Focus on core properties
Standardize artifact evaluation
Promote comprehensive benchmarking practices
How should microarchitectural side-channel attacks be meaningfully evaluated and compared?
Benchmarking Practices
Takeaway


Practices (resolution metrics...)

Outdated Methodologies
Unfair Comparisons
Reproducibility Issues
Practices




12 / 12

Flaws and their impact on evaluation pro.
Backup
Benchmarking Practices
Backup
Benchmarking Practices
Summary of benchmarking flaws across all evaluated papers.

Backup
Benchmarking Practices
Prevalence of the 19 flaws in the 83 surveyed papers



Fig. 4: Distribution of reported properties in covert-channel evaluations.
Fig. 5: Distribution of side-channel attack types.
59 (Speed)
30 (Speed, Error Rate)
Reported Metrics per Paper
18 (Speed, Error Rate, ...)
218 Attacks
71 Covert Channels
147 Side Channels
Attacked Platforms per Paper
73 CPUs (and 65 Intel)
9 GPUs (and 3 iGPUs)
9 TEEs
Backup
Benchmarking Practices
CCS26
By Iliana Fayolle
CCS26
- 20