Adam Retter

adam@evolvedbinary.com
 

Declarative Amsterdam
@ Amsterdam Science Park
2023-11-02

@adamretter

• Director and CTO of Evolved Binary

  • XML / XQuery / XSLT / RDF / SPARQL

  • Scala / Java / C++ / Rust

  • Concurrency and Scalability

• Creator of FusionDB multi-model database

• Contributor to Facebook's Meta's RocksDB (7 yrs.)

• Core contributor to eXist-db XML Database (18 yrs.)

• Founder of EXQuery, and creator of RESTXQ

• Was a W3C XQuery WG Invited expert

• Me: www.adamretter.org.uk

• A loosely undefined set of related concepts!




• A "package" is a container of one or more things

  • Might conform to a standard size, shape, or construction

  • Might ease the storage of things

  • Might ease the transportation of things

• The act of "packaging" is that of containing the things




• We will focus on Packaging of:

  • Software Code

  • Data

Photo by Jiawei Zhao on Unsplash

• Software Reuse

  • Modern software architecture is modular

  • We are dependent on Software Libraries

    • Each may consist of many files

    • In-turn dependent on other libs. Ad infinitum

  • We may wish to publish our App/Libraries

    • May depend on many libraries

    • May consist of many files

• Data Distribution

  • A data set may consist of many files

  • We may need to consume data sets

  • We may wish to publish data sets

Photo by Kelly Sikkema on Unsplash

1. "At implementation time each module and its inputs and outputs are well-defined, there is no confusion in the intended interface with other system modules."




2. "At checkout time the integrity of the module is tested independently"




3. "the system is maintained in modular fashion; system errors and deficiencies can be traced to specific system modules"

Photo by Tom Hermans on Unsplash

Designing Systems Programs, by Gauthier and Ponto (1970)

• The "package" itself is one small part of a larger system

  • Hopefully a standardised file (and metadata?) format and name

• We also need to consider:

  • Consumption

  • Integration

  • Storage

  • Building a new Package (a.k.a. "Packaging")

  • Transportation

  • Publication

Photo by Vlad Tchompalov on Unsplash

• Essential Properties

  • Detailed open specification that standardises its format

    • Internal - What goes where, and how?

    • Interface - What is available from the package, and how?

    • External - Package file fomat(s) and naming convention(s)

  • Standardised metadata describing the package

    • Not implementation specific!

  • Content Agnostic

• Desirable Properties

  • Ease of storage / transportation

    • Single file containing both data and metadata; compressible

  • Easily inspectable

    • Metadata can be easily accessed and understood

  • Verifiable

Photo by Oli Zubenko on Unsplash

• How do we use a package?

  • It's the things inside that we care about!

    • Do they need to be extracted from the package?

• What about tools?

  • Do existing tools understand what a package is?

    • Do they even need to?

    • Could they be updated to support packages?

    • Can new tools be built to bridge between packages and existing tools?

  • Do we need to build new tools?

Photo by Kelly Sikkema on Unsplash

• Output Format, i.e. the "Package"

  • Should be defined elsewhere in a "Package Specification" standard

• Input Format

  • Unknown... Likely tool specific!

    • Needs to be clearly defined and documented for the users

• Existing tools might be usable

  • e.g. Compose: mkdir, cp, tar, and gz

• New tooling could simplify

  • Require certain inputs

  • Validate inputs and outputs

  • Single command

Photo by Josue Isai Ramos Figueroa on Unsplash

• One file, or a data file and accompanying metadata file(s)

  • Amenable to std. operations, e.g: cp, scp, EMail, Upload to Dropbox, etc.

• Publish to where?

  • Anywhere that accepts generic files

  • An environment adapted to the Package format

    • Registry - holds metadata and redirects to the package elsewhere

    • Repository - holds metadata and a copy of the package

    • Typically provide search facilities

    • May provide upload/download capabilities

    • Possibly accessible as a Humane Website

    • Possibly accessible by API - may also provide tools

Photo by David Trinks on Unsplash

Photo by Hush Naidoo Jade Photography on Unsplash

• EXPath Candidate Module - May 2012

  • https://expath.org/spec/pkg

  • An unfinished draft of a potential standard

  • Describes itself as a "packaging system" for components: "XSLT, XQuery, and XProc"

• Some tools (xrepo and Java libs) provided

• Covers:

  • The Package:

    • External file format

    • Layout of files within the package

    • Metadata (including dependencies and exported components)

  • Resolution of Namespace URI to (local) Components

  • On-disk repository layout

Photo by Jen Theodore on Unsplash

Photo by Pawtography Perth on Unsplash

Photo by Priscilla Du Preez 🇨🇦 on Unsplash

Photo by Sylwia Bartyzel on Unsplash

• Marklogic - abandoned prototype.

• BaseX - Supported.

• Saxon - abandoned prototype.

• eXist-db

• Undocumented Metadata Extensions to EXPath Packaging - repo.xml, exist.xml, and repo.xml

  • <license> / <copyright> / <type> / <target> / <prepare> / <finish>

• Consumption / Publication: Public Application Repository (~100 Pkgs.)

• Integration: autodeploy directory, XQuery functions, repository partly in database

• Building: Ant, or Maven.

Photo by Karsten Winegeart on Unsplash

Photo by Mohamed Nohassi on Unsplash

Photo by David Kovalenko on Unsplash

Photo by Nathalie SPEHNER on Unsplash

Photo by Joshua J. Cotten on Unsplash

Photo by Joshua J. Cotten on Unsplash

Photo by Steven Wright on Unsplash

Photo by Karsten Winegeart on Unsplash

Photo by Roberto Nickson on Unsplash

Alexandra

Adam

Tomos