building on the data model

schemas are just another (big) example of something that's "generic over the data model"

which is cool and wholesome

we want a
Type system

hypergenericism is hard to reason about
type systems are good
structs are good
unions are good
let's have good things
having a schema for validation of my data saves me oooooodles of time as a programmer!

and...

documentation, documentation, documentation!
facilitate design discussion
must be language agnostic

we want a
Type system

and...

All of the existing systems are hard to apply here.

Immutable links are consequential.

We care about migration...

And we care about migration that works for decentralized protocols and distributed development practices.
(Which means strict version numbers are *out* -- requires central coordination.)

let's get specific

kinds:
- map (now typed)
- list (now typed)
- string
- int
- (...all the same scalars from DM...)
- struct
- union
- enum
types:
- When you assign a name to one of the above.

typed primitives

## MyString is a named type.
type MyString string

## MyInt is another one.
type MyInt int

## and so on

typed maps

type MyString string

## "String" is the key type;
##  "MyString" is the value type.
type MyMap map {String:MyString}

## or inline in other things:
type MyStruct struct {
    aField {String:MyString}
}

typed lists

Bullet One
Bullet Two
Bullet Three

type MyString string

## Looks familiar already, right?
type MyList list [MyString]

## or inline in other things:
type MyStruct struct {
    aField [MyString]
}

... plus 'nullable'

## Without the 'nullable' keyword,
##  this list can *only* contain
##   strings!
type MyList list [String]

## This list can contain either
##  a string or a 'null' at each
##   entry in the list.
type HoleyList list [nullable String]

'nullable' can be applied to map values,
list values, and struct fields.

structs

## Structs have a known set of fields.
type MyStruct struct {
    x Int
    y String
    z nullable MyStruct
}

... plus 'optional'

'optional' is distinct from 'nullable'!
Means the field can be *missing* entirely.

'optional' only applies to struct fields.

## Structs have a known set of fields.
type MyStruct struct {
    x Int
    y optional nullable String
    z optional MyStruct
}

cardinality

a quick word on...

Schema	Valid Matching Representations	Cardinality
type Foo struct { bar Bool }	{"bar": true} {"bar": false}	2

Schema	Valid Matching Representations	Cardinality
type Foo struct { bar Bool }	{"bar": true} {"bar": false}	2
type Foo struct { bar nullable Bool }	{"bar": true} {"bar": false} {"bar": null}	3 = 2+1
type Foo struct { bar optional Bool }	{"bar": true} {"bar": false} {}	3 = 2+1
type Foo struct { bar optional nullable Bool }	{"bar": true} {"bar": false} {"bar": null} {}	4 (!) = 2+1+1
type Foo struct { bar Bool (default "false") }	{"bar": true} {}	2

cardinality

a quick word on...

Cardinality-counting is an important design foundation.

If the cardinality of two parts of a model aren't the same, then that means one of them is less expressive.

Can use this to reason about compatibility and completeness of models!

... plus defaults

Defaults are a neat feature for reducing serialized verbosity... without changing cardinality.

(This means encountering the 'default' in the serial data is rejected...! Otherwise, the transform would be lossy!)

## 'defaults' can be used to elide
##   common values when serializing;
##    they *don't* change cardinality.
type MyStruct struct {
    y Bool (default false)
    z String (default "word")
}

... with representations

## Structs are represented as maps
##  by default!  So you dont need to
##   say it.
type MyStruct struct {
    x Int
    y Bool
    z String
} representation map

Everything we've seen so far has had an implicit "representation" -- instructions for how it maps onto the Data Model kinds.

... with representations

## This type will serialize as a list!
type MyStruct struct {
    x Int
    y Bool
    z String
} representation tuple

We can customize these.

... with representations

## This will serialize as a STRING!
type MyStruct struct {
    x String
    y String
} representation stringjoin {
    delim ":"
}

... And it can change the kind of representation entirely.

... with representations

## This will serialize as a STRING!
type MyStruct struct { ... }
  representation stringjoin { ... }

## So this map can use it as a key...!
type WildMap map {MyStruct:Whatever}

This is an important feature:
with it, we can use structs as map keys, for example.

unions

Unions (also often known as "sum types") can contain data from any one of their member types...
but only one at a time.

type NeatUnion union {
    | MemberTypeOne "one"
    | MemberTypeTwo "two"
    | MemberTypeThree "three"
} representation keyed

unions always must define a representation

keyed
envelope
inline
kinded

enums

type MyEnum enum {
    | One "one"
    | Two "two"
    | Three "3"
}

let's talk about

representations

Representations allow simple, deterministic, bi-directional transformations.
These transformations take us between the
schema kind and the Data Model's kind.
Most kinds have default representations...
Some (namely, unions) don't, because there's no popular universal agreement about unions in contemporary design patterns.

back to "advanced layouts"

Compared to Schema Representations:
- AdvLayouts allow arbitrary, turing-complete code. Representations don't, and are all fast.
- AdvLayouts can split data into more blocks; Representations can't.
- Representations are a must-have feature for an IPLD library that supports Schemas; AdvLayouts lean on out-of-band components.
- Representations correspondingly are strictly specified core specs; will be stable over time.

back to "advanced layouts"

Schemas are another way to indicate their usage!
- Alternative to in-band signaling with magic keywords.

## This map will be sharded.
type MyMap map {String:MyString}<HAMT>

## Additional config here.
advanced HAMT {
  implementation "experimental/HAMT/v1"
  bitwidth 14
  hashalgo "murmur"
}

back to "advanced layouts"

Schemas are another way to indicate their usage!
- Alternative to in-band signaling with magic keywords.
- Unlocks a cool property....
  
  You can choose whether or not you want to "see through" an advanced layout and address it's individual blocks, or not, by doing traversals with a schema that indicates it, or not.
  
  Useful for "grab me the left-leaning tree"...
  (This maps to "stream the beginning of a file"..!)

mi
gra
tions

migrations

Core concept: "schema 'try stacks'"
Try to fit the data to the first schema...
- if it fits, you're done
Try to fit the data to the second schema...
- if it fits, you're done
Continue like so...

Gives us the ability to do "structural typing" -- it detects matching data, without the use of explicit version numbers!

Foundations for Decentralization:

Data with IPLD