Topics

• HDF5
• HDF5 tools
• h5py
• PyTables

• Data Model
• Library
• Format

HDF5 is...

A filesystem in a file

Working with groups and group members is similar to working with directories and files in UNIX.

HDF5 Data Model

• Datasets (i.e. files in a filesystem)
• Groups (i.e. directories in a filesystem)
• Attributes (i.e. metadata of file/directory)

Why use HDF5?

• Portable
• Self-describing

• Can contain binary data (in many representations)

• Allows direct access to parts of the file without first parsing the entire contents

• Supports large/complex/heterogeneous data
• File format tool kit (you can design your own file format and use HDF5 under the hood)

Who uses HDF5?

HDF5 Tools

• h5ls
• h5dump
• h5stat
• h5diff
• HDFView
• HDF Compass
• ViTables

HDF5 Tools by HDF Group

Reference

• HDF5 Main Page
• HDF5 Tutorial
• Introduction to HDF5 (Quincey Koziol)
• Introduction to HDF5 (HDF Group)
• Unifying Biological Image Formats with HDF5
• Everything that HDF users have always wanted to know about Hadoop... but were ashamed to ask
• Moving away from HDF5 (Hacker News)

h5py

• Thin, pythonic wrapper around HDF5

• HDF5 errors are converted into Python exceptions

• Written in Cython

• Uses numpy objects

h5py on GitHub

• Higher level abstraction

• It does not want to be a complete wrapper for the entire HDF5 C API

• Can be faster than h5py, thanks to out-of-core querying

• Allow indexing and complex queries

• Built-in compression

• Undo mode

PyTables on GitHub

Scipy 2015

At SciPy 2015, developers from PyTables, h5py, the HDF Group and pandas decided to start a refactor: PyTables will depend on h5py for its bindings to HDF5.

Python & HDF5: a Vision

New Backend Interface for PyTables 4

Code, plz!

Repo with Jupyter notebook and some snippets