Iliana FAYOLLE Univ. Lille, CNRS, Inria Lille, France
16 November 2026
Antoine GEIMER Univ. Lille, CNRS, Inria Lille, France
Daniel DE ALMEIDA BRAGA Univ. Rennes, CNRS, IRISA Rennes, France
Clémentine MAURICE Univ. Lille, CNRS, Inria Lille, France
A Look Back at Benchmarking Practices for Microarchitectural Side-Channel Attacks
System
Sensitive
data
Leak sources
Computation
1. Collect
2. Deduce
system
Unintended sources of information leakage
sensitive data
" from a that may be exploited to infer ."
1 / 12
Covert channels
Side channels
System
Sensitive
data
Leak sources
Computation
1. Collect
1 / 12
Outdated Methodologies
Are they good proxies to evaluate the attack?
Usual Targets
Fingerprinting
Alexa top 100 websites
Control-Flow (CF)
v1.1.{0, 1}h, and v3.2
GnuPG v1.4.{13, 14}
...
OpenSSL v1.0.{1, 2}e,
Memory Access
WolfSSL v4.2.0, and v5.6.4
OpenSSL v1.0.1{e, f},
...
v0.9.8b, v1.1.0g, and v1.0.2
2 / 12
Outdated Methodologies
Outdated Methodologies
Unfair Comparisons
2 / 12
Outdated Methodologies
Outdated Methodologies
Unfair Comparisons
Reproducibility Issues
2 / 12
Outdated Methodologies
Outdated Methodologies
Unfair Comparisons
Reproducibility Issues
How should microarchitectural side-channel attacks be meaningfully evaluated and compared?
2 / 12
In what forms do they manifest?
How prevalent are they?
How to avoid them?
Recommendations
Outdated Methodologies
Unfair Comparisons
Reproducibility Issues
Practices
3 / 12
Number of papers per conference over time.
Selected 83 papers
Architecture & Security
Top Rank venues
From 2014 to 2024
More Security papers
Some relevant papers are excluded
More Recent papers
Microarchitectural side- & covert-channel attacks
4 / 12
Our Philosophy:
Claim First, Criterion Second.
5 / 12
Our Philosophy:
Claim First, Criterion Second.
5 / 12
Our Philosophy:
Claim First, Criterion Second.
5 / 12
Flaws and their impact on evaluation properties.
Benchmark Omissions
Improper Comparisons
Missing Information
Annotation Labels
Flawed
Incomplete
Unclear
Partially Flawed / Underspecified
Ok
NA
Not Flawed
SoK: Benchmarking Flaws in Systems Security.
E. van der Kouwe, G. Heiser, D. Andriesse, H. Bos, C. Giuffrida. EuroS&P'19.
6 / 12
Distribution of the number of flaws per paper.
~ 5.72 flaws per paper
Distribution of side-channel attack types.
218 Attacks
71 Covert Channels
147 Side Channels
7 / 12
Summary of benchmarking flaws across all evaluated papers.
8 / 12
Summary of benchmarking flaws across all evaluated papers.
8 / 12
Summary of benchmarking flaws across all evaluated papers.
8 / 12
Security
Architecture
>> Flaws
Median number of (partial) flaws per paper across all conferences.
9 / 12
>> Evaluated Platforms
Security
>> Flaws
Median number tested platforms per paper across all conferences.
Architecture
9 / 12
Covert channels and side-channel attack types across conferences.
Security
>> Covert Channels
>> Flaws
>> Diversity of Side Channels
>> Evaluated Platforms
Architecture
9 / 12
>> Complete Artifacts
>> Covert Channels
>> Diversity of Side Channels
>> Evaluated Platforms
Security
>> Flaws
Both
Security
Availability of artifacts over time.
Architecture
Architecture
The dashed lines show when artefact evaluation began.
9 / 12
>> (Incomplete) Artifacts
Both
Security
Availability of artifacts over time.
The dashed lines show when artefact evaluation began.
Architecture
10 / 12
>> (Incomplete) Artifacts
>> Side-Channel Evaluations
Evolution of median number of benchmarks per paper over time
across all conferences.
10 / 12
Not All Those Who Share Are Lost: Analyzing 25 Years of Cybersecurity Artifact Sharing Practices Through Automated Discovery.
D. Vansteenhuyse, A. Bols, L. Desmet, V. Le Pochat, J. Van Bulck, M. Bognar. USENIX'26.
Reproducibility in Applied Security Conferences: An 11-Year Review on Artifacts and Evaluation Committees.
D. Olszewski, A. Lu, A. Crowder, N. Bennett, S. Layton, S. Hrushikesh V. Bhupathiraju, T. Tucker, S. Kalgutkar, H. Ver Helst, C. Stillman, K. R. B. Butler,
S. Rampazzi, P. Traynor. ACM REP'25.
Lessons Learned from Five Years of Artifact Evaluations at EuroSys.
D. C. D’Elia, T. D. Doudali, C. Giuffrida, M. Matos, M. Payer, S. Pirelli, G. Portokalidis, V. Schiavoni, S. Signorello, A. Vahldiek-Oberwagner. ACM REP'25.
Reviewers
Share artifacts
Authors
Community
Standardize artifact evaluation
11 / 12
Invisible Probe: Timing Attacks with PCIe Congestion Side-channel.
M. Tan, J. Wan, Z. Zhou, Z. Li. S&P'21.
Reviewers
Share artifacts
Authors
Community
Standardize artifact evaluation
11 / 12
RELOAD+REFRESH: Abusing Cache Replacement Policies to Perform Stealthy Cache Attacks.
S. Briongos, P. MalagĂłn, J. M. Moya, T. Eisenbarth. USENIX'20.
Smallest distinguishable unit and its observation mapping.
Spatial Resolution
Reviewers
Share artifacts
Prioritize resolution metrics
Authors
Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Cache Template Attacks: Automating Attacks on Inclusive Last-Level Caches.
D. Gruss, R. Spreitzer, S. Mangard. USENIX'15.
Smallest distinguishable unit and its observation mapping.
Spatial Resolution
Reviewers
Share artifacts
Prioritize resolution metrics
Authors
Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Temporal Resolution
Time per sample and maximum sampling rate.
TunneLs for Bootlegging: Fully Reverse-Engineering GPU TLBs for Challenging Isolation Guarantees of NVIDIA MIG.
Z. Zhang, T. N. Allen, F. Yao, X. Gao, R. Ge. CCS'23.
Reviewers
Share artifacts
Prioritize resolution metrics
Authors
Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Temporal Resolution
Time per sample and maximum sampling rate.
POWERT Channels: A Novel Class of Covert Communication Exploiting Power Management Vulnerabilities.
S. K. Khatamifard, L. Wang, A. Das, S. Köse, U. R. Karpuzcu. HPCA'19.
Reviewers
Share artifacts
Prioritize resolution metrics
Authors
Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Reviewers
Demand relevant benchmarks
Focus on core properties
Share artifacts
Prioritize resolution metrics
Authors
Community
Standardize artifact evaluation
Promote comprehensive benchmarking practices
11 / 12
Share artifacts
Prioritize resolution metrics
Authors
Reviewers
Community
Demand relevant benchmarks
Focus on core properties
Standardize artifact evaluation
Promote comprehensive benchmarking practices
How should microarchitectural side-channel attacks be meaningfully evaluated and compared?
Practices (resolution metrics...)
Outdated Methodologies
Unfair Comparisons
Reproducibility Issues
Practices
12 / 12
Flaws and their impact on evaluation pro.
Summary of benchmarking flaws across all evaluated papers.
Prevalence of the 19 flaws in the 83 surveyed papers
Fig. 4: Distribution of reported properties in covert-channel evaluations.
Fig. 5: Distribution of side-channel attack types.
59 (Speed)
30 (Speed, Error Rate)
Reported Metrics per Paper
18 (Speed, Error Rate, ...)
218 Attacks
71 Covert Channels
147 Side Channels
Attacked Platforms per Paper
73 CPUs (and 65 Intel)
9 GPUs (and 3 iGPUs)
9 TEEs