BYOTh

 

Bring Your Own Thesis

PhD-portalen

  • Lever avhandlinger og pressemelding
    • Trykkeri
    • BORA
    • W3 - pressemelding
    • NB - plikavlevering
    • Dokumentsenteret - arkivering
    • ...
  • Gjenbruk av metadata
  • Teste å lage en RDF editor i javascript

Besvarelsesrepo

Redis

Server

(Node/Express)

ISBN

Sebra

Rom

Trykkeri

BORA

W3

NB, etc

Klient

(Angular)

Crossref

Dataporten

Linked Data Platform

Prinsipper

  • CRUD-operasjoner på RDF-data med HTTP-verb
  • Hierarkier bygges automatisk
    • Gjør adgangskontroll lettere
  • Kan holde både RDF-dokumenter og andre filer i samme struktur

 

POST

@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix dcterms: <http://purl.org/dc/terms/> .

<http://example.org/alice/> a ldp:Container, ldp:BasicContainer ;
    dcterms:title "Alice’s data storage on the Web" .
POST /alice/ HTTP/1.1
Host: example.org
Link: <http://www.w3.org/ns/ldp#Resource>; rel="type"
Slug: foaf
Content-Type: text/turtle

@prefix dc: <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .

<> a foaf:PersonalProfileDocument;
    foaf:primaryTopic <#me> ;
    dc:title 'Alice’s FOAF file' .

<#me> a foaf:Person;
    foaf:name 'Alice Smith'  .
HTTP/1.1 201 Created
Location: http://example.org/alice/foaf
Link: <http://www.w3.org/ns/ldp#Resource>; rel="type"
Content-Length: 0
@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
  
<http://example.org/alice/> a ldp:Container, ldp:BasicContainer;
   dcterms:title 'Alice’s data storage on the Web';
   ldp:contains <http://example.org/alice/foaf> . 

Oppdatere dokument

  • PUT:
    • Send ny versjon av dokumentet
    • Får 4xx om beskyttede verdier er endret
  • PATCH:
    • SPARQL patch, slett noen tripler, skriv noen andre
    • Noen tripler får ikke endres av klienten

Oppdatere dokument

Løsning i BYOTh:

  • Hent lagret dokument
  • Filtrer ut de triplene som ikke får endres
  • Slett gamle tripler og skriv nye tripler med PATCH.

Fedora Commons Repository

  • Implementasjon av LDP
  • Brukes i UiBs besvarelsesrepo
  • Filer er blobber i databasen
  • Følger standarden (hva jeg kan se)
    • Har noen egne utvidelser (/fcr:metadata, transactions)
  • UiBs instans har ikke SPARQL-endpoint (!)

JSON-LD

<.../thesis/123-456> <...rdf-schema#type> <...byoth#ThesisContainer> .
<.../thesis/123-456> <...byoth#hasThesis> <.../thesis/123-456> .
<.../thesis/123-456> <...dct/creator> <.../person/user1> .
[
  {
    "@id": "http://your.ldp.com/rest/phd/thesis/123-456",
    "http://www.w3.org/2000/01/rdf-schema#type": [
      {
        "@id": "http://purl.org/byoth#ThesisContainer"
      }
    ],
    "http://purl.org/byoth#hasThesis": [
      {
        "@id": "http://your.ldp.com/rest/phd/thesis/123-456"
      }
    ],
    "http://purl.org/dc/terms/creator": [
      {
        "@id": "http://your.ldp.com/rest/phd/person/user1"
      }
    ]
  }
]

Frames

// frame
{
    "@context": {
        "byoth": "http://purl.org/byoth#",
        "dct": "http://purl.org/dc/terms/",
        "@base": "http://your.ldp.com/rest/phd/",
        "hasThesis": {
            "@id": "byoth:hasThesis"
        }
    },
    "@type": "byoth:ThesisContainer",
    "hasThesis": { "@default": null }
}
// framed document
{
  "@context": {
    "byoth": "http://purl.org/byoth#",
    "dct": "http://purl.org/dc/terms/",
    "@base": "http://your.ldp.com/rest/phd/",
    "hasThesis": {
      "@id": "byoth:hasThesis"
    }
  },
  "@graph": [
    {
      "@id": "thesis/123-456",
      "@type": "byoth:ThesisContainer",
      "hasThesis": {
        "@id": "thesis/123-456"
      },
      "dct:creator": {
        "@id": "person/user1"
      }
    }
  ]
}

Compactification

// framed document
{
  "@context": {
    "byoth": "http://purl.org/byoth#",
    "dct": "http://purl.org/dc/terms/",
    "@base": "http://your.ldp.com/rest/phd/",
    "hasThesis": {
      "@id": "byoth:hasThesis"
    }
  },
  "@graph": [
    {
      "@id": "thesis/123-456",
      "@type": "byoth:ThesisContainer",
      "hasThesis": {
        "@id": "thesis/123-456"
      },
      "dct:creator": {
        "@id": "person/user1"
      }
    }
  ]
}
// compacted and framed document
{
  "@context": {
    "byoth": "http://purl.org/byoth#",
    "dct": "http://purl.org/dc/terms/",
    "@base": "http://your.ldp.com/rest/phd/",
    "hasThesis": {
      "@id": "byoth:hasThesis"
    }
  },
  "@id": "thesis/123-456",
  "@type": "byoth:ThesisContainer",
  "hasThesis": {
    "@id": "thesis/123-456"
  },
  "dct:creator": {
    "@id": "person/user1"
  }
}

@explicit: true

// frame with '@explicit: true'
{
    "@context": {
        "byoth": "http://purl.org/byoth#",
        "dct": "http://purl.org/dc/terms/",
        "@base": "http://your.ldp.com/rest/phd/",
        "hasThesis": {
            "@id": "byoth:hasThesis"
        }
    },
    "@type": "byoth:ThesisContainer",
    "hasThesis": { "@default": null },
    "@explicit": true
}
// compacted and framed document 
{
  "@context": {
    "byoth": "http://purl.org/byoth#",
    "dct": "http://purl.org/dc/terms/",
    "@base": "http://your.ldp.com/rest/phd/",
    "hasThesis": {
      "@id": "byoth:hasThesis"
    }
  },
  "@id": "thesis/123-456",
  "@type": "byoth:ThesisContainer",
  "hasThesis": {
    "@id": "thesis/123-456"
  }
}
// original document
{
  "@context": {
    "byoth": "http://purl.org/byoth#",
    "dct": "http://purl.org/dc/terms/",
    "@base": "http://your.ldp.com/rest/phd/",
    "hasThesis": {
      "@id": "byoth:hasThesis"
    }
  },
  "@id": "thesis/123-456",
  "@type": "byoth:ThesisContainer",
  "hasThesis": {
    "@id": "thesis/123-456"
  },
  "dct:creator": {
    "@id": "person/WRONG_USER"
  }
}

Datamodell

  • Basert på Portland Common Data Model (PCDM)
  • byoth: http://purl.org/byoth#
  • Rot-container er byoth:ThesisContainer et pcdm:Object, inneholder:
    • byoth:Thesis, byoth:Candidate, byoth:Department, etc
    • Sub-containers for filer og artikler
    • Hver av disse vil refereres til som et dokument
  • Faste slugs: /thesis/<id>/candidate, etc.

Dokument-modeller

  • Hver dokument-modell har
    • en type: byoth:Thesis, byoth:Event
    • en JSON-LD frame
    • et Typescript interface
  • Noen dokument-modeller har hjelpefunksjoner:
    • nøstede elementer
    • ordnede lister
    • ...

Eksempel

import { prefixes } from './prefixes';

const TYPE = 'byoth:Person';

const CONTEXT: JsonLdContext = {
    ...prefixes,
    givenName: { '@id': 'schema:givenName' },
    familyName: { '@id': 'schema:familyName' },
    email: { '@id': 'schema:email' },
    birthDate: { '@id': 'dbo:birthDate', '@type': 'xsd:date' },
    orcid: { '@id': 'dbo:orcidId' },
    gender: { '@id': 'schema:gender' }
}

export const CANDIDATE_FRAME: JsonLdFrame = {
    '@context': CONTEXT,
    '@type': TYPE,
    '@explicit': true,
    givenName: defaultEmptyString,
    familyName: defaultEmptyString,
    email: defaultEmptyString,
    birthDate: defaultNull,
    orcid: defaultNull,
    gender: defaultNull,
};
export interface CandidateProperties {
    givenName: string,
    familyName: string,
    email: string,
    brithDate: string,
    orcid: string,
    gender: string
}

export interface Candidate 
    extends CompactedDocumentHeader, CandidateProperties 
{ }

Eksempel 2

export const ORDER_FRAME: JsonLdFrame = {
    '@type': 'byoth:Order',
    '@context': CONTEXT,
    '@explicit': true,
    // ...
    delivery: {
        '@type': 'schema:ParcelDelivery',
        '@explicit': true,
        description: defaultNull,
        address: {
            '@type': 'schema:PostalAddress',
            '@explicit': true,
            postalCode: defaultNull,
            // ...
        }
    },
    // ...
}

export interface AddressProperties {
    postalCode: string,
    // ...
}

export interface Address extends AddressProperties {
    '@type': string,
    '@id': string,
}


export interface Delivery {
    '@type': string,
    '@id': string,
    description?: string,
    address?: Address
}
export function initOrderDelivery(order) {
    order.delivery = {
        '@id': order['@id'] + '#delivery',
        '@type': 'schema:ParcelDelivery',
        address: {
            '@id': order['@id'] + '#delivery-address',
            '@type': 'schema:PostalAddress'
        }
    };
}

LdpClient

ThesisContainerAdapter, ThesisFileAdapter, ...

ClientAuth

REST API

PrintShopAdapter

Passport

 /api/user

 .../submit   .../validate

 /api/theses,

/api/press-releases

Redis

Dataporten

Trykkeri

BORA, W3, Trykkeri, ...

SOAP API

PhD-kandidat

Besvarelses-repo

Intern logikk

JWT

REST API

Dataporten

Trykkeri

BORA, W3, Trykkeri, ...

SOAP API

PhD-kandidat

Besvarelses-repo

LDP Client

export class LdpClient { 
    // ...
 
    async exists(path: string): Promise<boolean> { /* ... */ }

    async get(path: string, model: DocumentModel): Promise<CompactedDocument>
    { /* */ }

    async update(path: string, 
                 model: DocumentModel,
                 doc: CompactedDocument): Promise<CompactedDocument> 
    { /* ... */ }

}

Oppdatere dokument

async update(path: string, 
             model: DocumentModel,
             doc: CompactedDocument): Promise<CompactedDocument> {
    let lock;

    try {
        lock = await this.redlock.lock(this.root + path, 2000);
    } catch (err) {
        throw new LockError(err.message);
    }

    try {
        // perform update
        // ...
    } finally {
        await lock.unlock().catch(console.error);
    }
}

Oppdatere dokument

// perform update

const oldTriples = await this.getTriples(this.root + path)
            .then(triples => Ldp.filterTriples(triples, model.frame))
            .then(triples => this.removeReadonlyTriples(triples));

// ...

Oppdatere dokument

const oldTriples = await this.getTriples(this.root + path)
    .then(triples => Ldp.filterTriples(triples, model.frame))
    .then(triples => this.removeReadonlyTriples(triples));

let newTriples;

try {
    newTriples = await Ldp.filterJsonLdToTriples(doc, model.frame)
        .then(triples => this.removeReadonlyTriples(triples));

    if (!newTriples) {
        throw new FramingError('Document not compatible with model', 422);
    }
} catch (err) {
    if (err instanceof FramingError) {
        err.status = 422;
    }
    throw err;
}

// ...

Oppdatere dokument

const oldTriples = await this.getTriples(this.root + path)
    .then(triples => Ldp.filterTriples(triples, model.frame))
    .then(triples => this.removeReadonlyTriples(triples));

let newTriples;

try {
    newTriples = await Ldp.filterJsonLdToTriples(doc, model.frame)
        .then(triples => this.removeReadonlyTriples(triples));

    if (!newTriples) {
        throw new FramingError('Document not compatible with model', 422);
    }
} catch (err) {
    if (err instanceof FramingError) {
        err.status = 422;
    }
    throw err;
}

const query = `DELETE {\n${oldTriples}}\nINSERT {\n${newTriples}}\nWHERE {}`;

// ...

Oppdatere dokument

// ...

try {
    await axios.patch(this.root + path, query,
        { headers: { ...Ldp.patchHeaders, authorization: this.authorization } });
} catch (err) {
    throw convertAxiosError(err);
}

const updatedTriples = await this.getTriples(this.root + path);
return Ldp.createLdDocument(updatedTriples, model.frame);

Filhåndtering

Filer lastes opp med vanlig POST query.

For å håndtere filmetadata brukes suffix '/fcr:metadata'

async updateFileMetadata(
    path: string,
    model: DocumentModel,
    doc: CompactedDocument): Promise<CompactedDocument> {
    return this.update(path + '/' + this.metadataSuffix, model, doc);
}

ThesisContainerAdapter

type Properties =  'articles' | 'candidate' | 'defense' | 'department'
 | 'files' | 'inputFiles' |  'order' | 'thesis' | 'trialLecture';

export class ThesisContainerAdapter {
    async createContainer(req: Request): Promise<ThesisContainer> { /* ... */ }
    
    async getProperty(containerId: string, property: Properties): 
        Promise<CompactedDocument> { /* ... */ }

    async putProperty(containerId: string, property: Properties, data: CompactedDocument):
        Promise<CompactedDocument> { /* ... */ }

    // ...
}

REST API

Express.js

router = express.Router();

router.get('/api/container/:id', (req, res, next) => {
    data = adapter.getData(req.param.id);

    if (!data) {
        next(Error('Could not get data'));
    }

    res.json(data);
});

REST API

async Express.js

import * as asyncHandler from 'express-async-handler';

router = express.Router();

router.get('/api/container/:id', asyncHandler(async (req, res, next) => {
    data = await adapter.getData(req.param.id);

    if (!data) {
        next(Error('Could not get data'));
    }

    res.json(data);
}));

REST API

How to handle

GET /api/container/:id

if id is

/thesis/12/34/56/123456?

Base64 url encode:

L3RoZXNpcy8xMi8zNC81Ni8xMjM0NTY
 GET /api/container/L3RoZXNpcy8xMi8zNC81Ni8xMjM0NTY

REST API

Express middleware

export function decodeParam(key: string) {
    return function(req: Request, res: Response, next: NextFunction) {
        if (typeof req.params[key] === 'string') {
            req.params[key] = base64url.decode(req.params[key]);
        }
        next();
    };
}
router.route('/container/:id')
.all((req, res, next) => {
    console.log(req.param.id); // L3RoZXNpcy8xMi8zNC81Ni8xMjM0NTY
    next();
})
.all(decodeParam('id'))
.all((req, res, next) = > {
    console.log(req.param.id); // /thesis/12/34/56/123456
    next();
});

REST API

export function containerApi(container: ThesisContainerAdapter,
                             printshop: PrintShopAdapter,
                             isbnHandler: ISBNHandler): Router {
    const router = express.Router();

    router.use('/', ensureAuthenticated());

    // ...

    router.route('/:id/:property')
    .all(decodeParam('id'))
    .all(asyncHandler(ensureContainerOwner(container, 'id')));
    .get(asyncHandler(async (req, res, next) => {
        const data 
            = await container.getProperty(req.params.id, req.params.property, req);
        res.json(data);
    }))
    .put(asyncHandler(async (req, res, next) => {
        const result 
            = await container.putProperty(req.params.id, req.params.property, req.body, req);
        res.json(result);
    }));

    // ...

    return router;
}

REST API

import * as express from 'express';
import * as http from 'http';
import { containerApi } from './routes/container';

app = express();

// get config somehow
const host = config.get( /* ... */ );
const port = config.get( /* ... */ );

app.set('host', host);

// init LdpClient, ThesisContainerAdapter, authentication
// ...

app.use(containerApi(containerAdapter, auth));

const server = http.createServer(app);

server.listen(port,
    () => console.log(`Application listenting on http://${host}:${port}`));

Angular-klient

Siden server-koden er skrevet i javascript kan den deles mellom server og klient

Tre mapper

server/ - all server-spesifikk kode

client/ - all klient-spesifikk kode

model/ - kode som beskriver datamodellen som begge kan bruke

Angular-klient

Klienten har adgang til hjelpefunksjoner for dokumenter (husk addresse-eksempelet):

 

- Vi får hjelp å bygge opp valid JSON LD

Angular-klient

Klienten har adgang til Typescript-definisjoner for alle dokumenttyper:

 

- Veldig mange typer feil kan detekteres statisk ved kompilering.

Angular-klient

REST API

ContainerService

AuthService

...

...

FormHandler

FormComponent

FormStepperComponent

RxJS - Reactive programming/data flow

const subject = new Subject<string>(null);

const subscription = 
    subject.subscribe(value => console.log(value));

subject.next('foo');
subject.next('bar');

setTimeout(() => subscription.unsubscribe(), 1);

// Output:
// foo
// bar

RxJS - Reactive programming/data flow

const subject = new Subject<string>(null);
const upperCaseObservable = subject.pipe(
    map(value => value.toUpperCase())
);

const subscription = 
    upperCaseObservable.subscribe(value => console.log(value));

subject.next('foo');
subject.next('bar');

setTimeout(() => subscription.unsubscribe(), 1);

// Output:
// FOO
// BAR

DocumentService

/** A service that can be used to access, update and persist a document */
export interface DocumentService<T extends CompactedDocument> {
    /** Observable for the document */
    valueChanges: Observable<T>;

    /** Get an instant value of the document */
    value: T;

    /** Set the value of the document */
    setValue(document: T): void;

    /** Get the document once */
    get(): Observable<T>;

    /** Persist the document */
    save(): Observable<T>;

    /** Reset to null document */
    reset();
}

ContainerService

export class ContainerService {
    /** Service for the container's candidate */
    candidate: DocumentService<Candidate>;

    /** Service for the container's defense */
    defense: DocumentService<Defense>;

    // ...

    private properties: DocumentService<CompactedDocument>[];

    // ...

    save(): Observable<any> {
        return forkJoin(this.properties.map(property => property.save()));
    }
}

FormHandler

export class FormHandler {
    subscription: Subscription;

    constructor(protected form: FormGroup,
                protected service: DocumentService<CompactedDocument>) {
        // init with latest value from server
        this.service.get()
        .subscribe(doc => this.patchForm(doc));

        // listen for value changes from other events
        this.subscription = this.service.valueChanges
        .subscribe(doc => this.patchForm(doc));

        // listen for value changes to form
        this.subscription.add(
            this.form.valueChanges
            .subscribe(value => this.patchDocument(value))
        );
    }

    // stop listening when form is destroyed (avoid memory leaks)
    destroy() {
        this.subscription.unsubscribe();
    }

    // ...
}

FormHandler

export class FormHandler {
    // ...

    patchForm(document: CompactedDocument) {
        if (document) {
            this.form.patchValue(document, {emitEvent: false});
        }
    }

    patchDocument(formValue: any) {
        const doc = this.service.value;
        Object.assign(doc, formValue);
        this.service.setValue(doc);
    }
}

FormComponent

@Component({
    selector: 'app-candidate',
    templateUrl: './candidate.component.html',
    styleUrls: ['./candidate.component.css']
})
export class CandidateComponent extends BaseFormComponent {
    form: FormGroup;

    constructor(
        private container: ContainerService,
        private fb: FormBuilder) {
        super();
        this.form = this.fb.group({
            givenName: ['', Validators.required ],
            familyName: ['', Validators.required ],
            email: ['', [ Validators.required, Validators.email ] ]
        });

        this.addFormHandler(
            new FormHandler(this.form, this.container.candidate)
        );
    }
}

FormStepperComponent

@Component({
    selector: 'app-press-release-form',
    templateUrl: './press-release-form.component.html',
})
export class PressReleaseFormComponent implements OnInit {
    steps = [
        { label: { en: 'Candidate', nb: 'Personalia' }, type: CandidateComponent },
        { label: { en: 'Defense', nb: 'Disputas' }, type: EventsComponent }
    ];
    
    // ...
}
<!-- press-release.component.html -->
<app-form-stepper [steps]="steps" [saveFunction]="saveFunction">
</app-form-stepper>

Tester

To hovedtyper

  • Enhetstester
  • Intergrasjonstester

Enhetstester

  • Tester koden funksjon for funksjon
    • Ofte bare test av de publike metoder/eksporterte funksjoner, de private/interne testes bare indirekte
  • Målet er å dekke flest mulig alternativ og edge cases
  • Hver funksjon studeres isolert, kall til andre biblioteker og spesielt databaser vil ofte "mockes" ut.

Integrasjonstester

  • Tester koden fra dens eksterne grensesnitt, med databaser og lignende aktivt
  • Sjekker at de forskjellige komponentene interagerer riktig
  • Detekterer regresjoner

Eksempel enhetstest

describe('LdpClient', () => {
    describe('exists()', () => {
        it('should return true if backend finds uri', async () => {
            mockAxios.onHead().reply(200);

            await expect(ldp.exists('foo')).to.eventually.be.true;
        });
    });
});
// code to test
export class LdpClient {
    async exists(path: string): Promise<boolean> {
        try {
            const response = await axios.head(this.root + path, 
               { headers: { authorization: this.authorization } });
            return response.status === 200;
        } catch (err) {
            // ...
        }
    }
}

Eksempel integrasjonstest


     describe('/api/container/[:id]', () => {
        const path = '/api/container';
        
        it('should deny unauthenticated POST requests', async () => {
            await request(server).post(path).expect(401);
        });

        it('should allow authenticated user to POST container', async () => {
            sandbox.stub(_helpers, 'isAuthenticated').yields(null);
            let id: string;

            await request(server).post(path)
            .expect(201)
            .expect('Location', /^\w+$/)
            .then(res => {
                id = res.header.location;
                expect(id).to.equal(base64url.encode(res.body['@id']), 
                    'Location header id is base64url-encoded version of body id');
                expect(res.body.creator).to.contain(user.id);
            });
        });
    });

Coverage-analyse

  • Måler hvilke linjer og forgreininger som er testet
  • Måler ikke om alle mulige argument og parametere er testet.
  • Måler ikke om alle mulige eksterne feil-kilder er testet
  • Ved 100 % coverage vet man i alle fall at man ikke har noen skrivefeil i noen uvanlige if-condititions
  • Noen deler av koden er lettere å teste med enhetstester og andre med integrasjon, holder med 100 % til sammen
# start test-db
cd server/integration-tests
docker compose up
cd ../..

# run all tests
npm run test

# alternatives
npm run test:server
npm run test:server:integration
npm run test:server:unit
npm run test:client
npm run e2e # nothing there yet

Hva gjenstår?

  • Trykkeri-integrasjon
  • Read-only integrasjoner:
    • W3 (ITA)
    • BORA (ITA)
    • NB (NB)
    • ...
  • Høyere test-dekning for klient
  • Dokumentasjon
  • UX
  • ...

BYOTh

By Simon Mitternacht

BYOTh

Bring Your Own Thesis

  • 719