November 17-22, 2024

Supercomputing 2024

Atlanta, GA

Kory Draughn

Chief Technologist

iRODS Consortium

Overview

• iRODS 4
  • What's available today
• iRODS 5
  • Goals
  • Current status
  • Remaining work
  • What interesting things does this enable?

iRODS 4 - What's available today

• Open source licensed (BSD-3)
• Stable, reliable, and well-maintained
• Keeps infrastructure flexible
• Community-backed
• Used by research, commercial, and governmental organizations worldwide

iRODS 4.3.3 is available now.

https://irods.org/2024/08/irods-4-3-3-is-released/

iRODS 4 - What's available today (cont.)

• Abstracts storage technologies to keep infrastructure flexible
  • disk, object storage, tape, etc
• Fully scriptable environment
  • Enforce policy using the iRODS Rule Language or Python
  • Generate, extract, and manipulate metadata
  • Orchestrate systems
• Plugin architecture
• Library support for the most popular programming languages
  • C, C++, Python, Java, Go
• Presents as other protocols to meet users where they are
  • HTTP, S3, WebDAV, Fuse, K8s CSI, NFS
• Packaged solutions to common challenges
  • Storage Tiering, Indexing, Ingest, etc

iRODS 5 - Goals

• Make the server cloud-friendly
  • Redesign the startup and shutdown process
• Improve management and flexibility of hostnames
• Make it easier to extend the server's capabilities

https://irods.org/roadmap

iRODS 5 - Current Status

• Server uses a simplified process model
  • SIGTERM for fast shutdown, SIGQUIT for graceful shutdown
  • SIGHUP for reloading configuration
    • Automatic reloading of configuration has been removed
  • Supports systemd
• Startup and shutdown are faster and more deterministic
• Separated catalog/configuration upgrades from server startup
• Removed the Control Plane
  • Server no longer listens on port 1248
• Multiple instances of the iRODS server can be run on a single computer
• Server no longer depends on Python

https://github.com/irods/irods/pull/7988

iRODS 5 - Remaining Work

• Simplify hostname management
  • Hostname information is stored in the catalog
    • Mapping => (key=node1, host=n1.company.org, ...)
      • Key can be mapped to many hosts to serve different purposes
    • Mappings are fetched on server startup
    • Mappings can be refreshed through a reload (i.e. SIGHUP)
  • Server uses unique key in place of actual hostname
    • Allows admin to provision servers without concrete hostnames
    • Key is used to retrieve real hostname
  • Remove need for host_resolution stanza

https://github.com/irods/irods/issues/7428

iRODS 5 - What interesting things does this enable?

• Control of plugin-managed external processes
• Addition of new server child processes
  • Sweepers
  • Stats/Metrics gathering/reporting
• Kubernetes ready
• Local clients can connect via Unix Domain Socket
  • Improve performance by removing TCP/IP layer
    • HTTP API, S3 API, NFSRODS, etc

Thank you!