PFHub Progress

Daniel Wheeler

https://pages.nist.gov/pfhub

11/05/2019

PFHUb DESign

• Statically generated, no content management system
• Delegate back-end functionality to external services
• Data upload, analysis and display on client
• Upload using GitHub pull requests

PFHub design

github

staticman

Jekyll

DATA app

Figshare

backblaze

s3

Mdf

Upload Schema

• Benchmark ID
• Data

  • Media links

  • Data file links

  • Resource usage

• Metadata

  • Authors

  • Hardware

  • Implementation

  • Summary

  • Timestamp

REView Process

github PR

Surge

TRAVIS CI

REVIEWER

USER

Staticman

What's new?

• 21 merged pull requests
• 10 new uploads
• BM8 (8a.0, 8b.0, 8c.0, 8d.0) pages
• Data processing app on App Engine
• Our own version of Staticman app on Heroku
• JORS paper published
• Completed Vega to Plotly transition
• CoRR link, but abandoned for now
  • sort of works, but can't rely on instance
  • GitHub authentication (not released yet)
  • focus on MDF though, but thanks to Yannick Congo
• Small stuff:
  • commitlint, wiki content + more

In Progress

• Improve upload views on Travis CI / Surge
  • View regardless of test outcome
  • Fast data view and response (< 3 minutes, currently ~12)
  • CI complete in (< 10 minutes)
  • Next step no CI view (seconds)
• Dump data to MDF on upload (directly from form)
  • Already have code for this (Ben Gaelwsky)
• Caching data in app (memcache) and on client (much faster views, no apparent loading)

What NeXT?

• Schema.org schema with help of Zach Trautt
  • Cordra???
• Loads of things: 58 open issues in April 2020 milestone
  • Tags, tag dependent views
  • Fix up BM5, BM6 and BM8
  • GitHub authentication

What Makes a Good Benchmark Suite?

• Robust
  • Widely applicable (intelligible, modest resources)
  • Well determined functional specification
  • Fully abstract specification
• Validating, verifying and confirming
  • Test reliability, correctness and accuracy of
    • Numerics (accuracy, precision, performance)
    • Physical representation (terms, coupling,  geometry, boundaries)
    • Operation (usage, reproducibility, system)
  • Identify good metrics for comparison
  • Validation hierarchy (unit problems to complete system)?
  • Uncertainty?
• Relevant
  • Educational and/or archival value
  • Physical/technical/community interest
  • Unsolved challenge?

The Benchmarks