BYOTh
Bring Your Own Thesis
PhD-portalen
- Lever avhandlinger og pressemelding
- Trykkeri
- BORA
- W3 - pressemelding
- NB - plikavlevering
- Dokumentsenteret - arkivering
- ...
- Gjenbruk av metadata
- Teste å lage en RDF editor i javascript
Besvarelsesrepo
Redis
Server
(Node/Express)
ISBN
Sebra
Rom
Trykkeri
BORA
W3
NB, etc
Klient
(Angular)
Crossref
Dataporten
Linked Data Platform
Prinsipper
- CRUD-operasjoner på RDF-data med HTTP-verb
- Hierarkier bygges automatisk
- Gjør adgangskontroll lettere
- Kan holde både RDF-dokumenter og andre filer i samme struktur
POST
@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
<http://example.org/alice/> a ldp:Container, ldp:BasicContainer ;
dcterms:title "Alice’s data storage on the Web" .
POST /alice/ HTTP/1.1
Host: example.org
Link: <http://www.w3.org/ns/ldp#Resource>; rel="type"
Slug: foaf
Content-Type: text/turtle
@prefix dc: <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
<> a foaf:PersonalProfileDocument;
foaf:primaryTopic <#me> ;
dc:title 'Alice’s FOAF file' .
<#me> a foaf:Person;
foaf:name 'Alice Smith' .
HTTP/1.1 201 Created
Location: http://example.org/alice/foaf
Link: <http://www.w3.org/ns/ldp#Resource>; rel="type"
Content-Length: 0
@prefix ldp: <http://www.w3.org/ns/ldp#> .
@prefix dcterms: <http://purl.org/dc/terms/> .
<http://example.org/alice/> a ldp:Container, ldp:BasicContainer;
dcterms:title 'Alice’s data storage on the Web';
ldp:contains <http://example.org/alice/foaf> .
Oppdatere dokument
- PUT:
- Send ny versjon av dokumentet
- Får 4xx om beskyttede verdier er endret
- PATCH:
- SPARQL patch, slett noen tripler, skriv noen andre
- Noen tripler får ikke endres av klienten
Oppdatere dokument
Løsning i BYOTh:
- Hent lagret dokument
- Filtrer ut de triplene som ikke får endres
- Slett gamle tripler og skriv nye tripler med PATCH.
Fedora Commons Repository
- Implementasjon av LDP
- Brukes i UiBs besvarelsesrepo
- Filer er blobber i databasen
- Følger standarden (hva jeg kan se)
- Har noen egne utvidelser (/fcr:metadata, transactions)
- UiBs instans har ikke SPARQL-endpoint (!)
JSON-LD
<.../thesis/123-456> <...rdf-schema#type> <...byoth#ThesisContainer> .
<.../thesis/123-456> <...byoth#hasThesis> <.../thesis/123-456> .
<.../thesis/123-456> <...dct/creator> <.../person/user1> .
[
{
"@id": "http://your.ldp.com/rest/phd/thesis/123-456",
"http://www.w3.org/2000/01/rdf-schema#type": [
{
"@id": "http://purl.org/byoth#ThesisContainer"
}
],
"http://purl.org/byoth#hasThesis": [
{
"@id": "http://your.ldp.com/rest/phd/thesis/123-456"
}
],
"http://purl.org/dc/terms/creator": [
{
"@id": "http://your.ldp.com/rest/phd/person/user1"
}
]
}
]
Frames
// frame
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@type": "byoth:ThesisContainer",
"hasThesis": { "@default": null }
}
// framed document
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@graph": [
{
"@id": "thesis/123-456",
"@type": "byoth:ThesisContainer",
"hasThesis": {
"@id": "thesis/123-456"
},
"dct:creator": {
"@id": "person/user1"
}
}
]
}
Compactification
// framed document
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@graph": [
{
"@id": "thesis/123-456",
"@type": "byoth:ThesisContainer",
"hasThesis": {
"@id": "thesis/123-456"
},
"dct:creator": {
"@id": "person/user1"
}
}
]
}
// compacted and framed document
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@id": "thesis/123-456",
"@type": "byoth:ThesisContainer",
"hasThesis": {
"@id": "thesis/123-456"
},
"dct:creator": {
"@id": "person/user1"
}
}
@explicit: true
// frame with '@explicit: true'
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@type": "byoth:ThesisContainer",
"hasThesis": { "@default": null },
"@explicit": true
}
// compacted and framed document
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@id": "thesis/123-456",
"@type": "byoth:ThesisContainer",
"hasThesis": {
"@id": "thesis/123-456"
}
}
// original document
{
"@context": {
"byoth": "http://purl.org/byoth#",
"dct": "http://purl.org/dc/terms/",
"@base": "http://your.ldp.com/rest/phd/",
"hasThesis": {
"@id": "byoth:hasThesis"
}
},
"@id": "thesis/123-456",
"@type": "byoth:ThesisContainer",
"hasThesis": {
"@id": "thesis/123-456"
},
"dct:creator": {
"@id": "person/WRONG_USER"
}
}
Datamodell
- Basert på Portland Common Data Model (PCDM)
- byoth: http://purl.org/byoth#
- Rot-container er byoth:ThesisContainer et pcdm:Object, inneholder:
- byoth:Thesis, byoth:Candidate, byoth:Department, etc
- Sub-containers for filer og artikler
- Hver av disse vil refereres til som et dokument
- Faste slugs: /thesis/<id>/candidate, etc.
Dokument-modeller
- Hver dokument-modell har
- en type: byoth:Thesis, byoth:Event
- en JSON-LD frame
- et Typescript interface
- Noen dokument-modeller har hjelpefunksjoner:
- nøstede elementer
- ordnede lister
- ...
Eksempel
import { prefixes } from './prefixes';
const TYPE = 'byoth:Person';
const CONTEXT: JsonLdContext = {
...prefixes,
givenName: { '@id': 'schema:givenName' },
familyName: { '@id': 'schema:familyName' },
email: { '@id': 'schema:email' },
birthDate: { '@id': 'dbo:birthDate', '@type': 'xsd:date' },
orcid: { '@id': 'dbo:orcidId' },
gender: { '@id': 'schema:gender' }
}
export const CANDIDATE_FRAME: JsonLdFrame = {
'@context': CONTEXT,
'@type': TYPE,
'@explicit': true,
givenName: defaultEmptyString,
familyName: defaultEmptyString,
email: defaultEmptyString,
birthDate: defaultNull,
orcid: defaultNull,
gender: defaultNull,
};
export interface CandidateProperties {
givenName: string,
familyName: string,
email: string,
brithDate: string,
orcid: string,
gender: string
}
export interface Candidate
extends CompactedDocumentHeader, CandidateProperties
{ }
Eksempel 2
export const ORDER_FRAME: JsonLdFrame = {
'@type': 'byoth:Order',
'@context': CONTEXT,
'@explicit': true,
// ...
delivery: {
'@type': 'schema:ParcelDelivery',
'@explicit': true,
description: defaultNull,
address: {
'@type': 'schema:PostalAddress',
'@explicit': true,
postalCode: defaultNull,
// ...
}
},
// ...
}
export interface AddressProperties {
postalCode: string,
// ...
}
export interface Address extends AddressProperties {
'@type': string,
'@id': string,
}
export interface Delivery {
'@type': string,
'@id': string,
description?: string,
address?: Address
}
export function initOrderDelivery(order) {
order.delivery = {
'@id': order['@id'] + '#delivery',
'@type': 'schema:ParcelDelivery',
address: {
'@id': order['@id'] + '#delivery-address',
'@type': 'schema:PostalAddress'
}
};
}
LdpClient
ThesisContainerAdapter, ThesisFileAdapter, ...
ClientAuth
REST API
PrintShopAdapter
Passport
/api/user
.../submit .../validate
/api/theses,
/api/press-releases
Redis
Dataporten
Trykkeri
BORA, W3, Trykkeri, ...
SOAP API
PhD-kandidat
Besvarelses-repo
Intern logikk
JWT
REST API
Dataporten
Trykkeri
BORA, W3, Trykkeri, ...
SOAP API
PhD-kandidat
Besvarelses-repo
LDP Client
export class LdpClient {
// ...
async exists(path: string): Promise<boolean> { /* ... */ }
async get(path: string, model: DocumentModel): Promise<CompactedDocument>
{ /* */ }
async update(path: string,
model: DocumentModel,
doc: CompactedDocument): Promise<CompactedDocument>
{ /* ... */ }
}
Oppdatere dokument
async update(path: string,
model: DocumentModel,
doc: CompactedDocument): Promise<CompactedDocument> {
let lock;
try {
lock = await this.redlock.lock(this.root + path, 2000);
} catch (err) {
throw new LockError(err.message);
}
try {
// perform update
// ...
} finally {
await lock.unlock().catch(console.error);
}
}
Oppdatere dokument
// perform update
const oldTriples = await this.getTriples(this.root + path)
.then(triples => Ldp.filterTriples(triples, model.frame))
.then(triples => this.removeReadonlyTriples(triples));
// ...
Oppdatere dokument
const oldTriples = await this.getTriples(this.root + path)
.then(triples => Ldp.filterTriples(triples, model.frame))
.then(triples => this.removeReadonlyTriples(triples));
let newTriples;
try {
newTriples = await Ldp.filterJsonLdToTriples(doc, model.frame)
.then(triples => this.removeReadonlyTriples(triples));
if (!newTriples) {
throw new FramingError('Document not compatible with model', 422);
}
} catch (err) {
if (err instanceof FramingError) {
err.status = 422;
}
throw err;
}
// ...
Oppdatere dokument
const oldTriples = await this.getTriples(this.root + path)
.then(triples => Ldp.filterTriples(triples, model.frame))
.then(triples => this.removeReadonlyTriples(triples));
let newTriples;
try {
newTriples = await Ldp.filterJsonLdToTriples(doc, model.frame)
.then(triples => this.removeReadonlyTriples(triples));
if (!newTriples) {
throw new FramingError('Document not compatible with model', 422);
}
} catch (err) {
if (err instanceof FramingError) {
err.status = 422;
}
throw err;
}
const query = `DELETE {\n${oldTriples}}\nINSERT {\n${newTriples}}\nWHERE {}`;
// ...
Oppdatere dokument
// ...
try {
await axios.patch(this.root + path, query,
{ headers: { ...Ldp.patchHeaders, authorization: this.authorization } });
} catch (err) {
throw convertAxiosError(err);
}
const updatedTriples = await this.getTriples(this.root + path);
return Ldp.createLdDocument(updatedTriples, model.frame);
Filhåndtering
Filer lastes opp med vanlig POST query.
For å håndtere filmetadata brukes suffix '/fcr:metadata'
async updateFileMetadata(
path: string,
model: DocumentModel,
doc: CompactedDocument): Promise<CompactedDocument> {
return this.update(path + '/' + this.metadataSuffix, model, doc);
}
ThesisContainerAdapter
type Properties = 'articles' | 'candidate' | 'defense' | 'department'
| 'files' | 'inputFiles' | 'order' | 'thesis' | 'trialLecture';
export class ThesisContainerAdapter {
async createContainer(req: Request): Promise<ThesisContainer> { /* ... */ }
async getProperty(containerId: string, property: Properties):
Promise<CompactedDocument> { /* ... */ }
async putProperty(containerId: string, property: Properties, data: CompactedDocument):
Promise<CompactedDocument> { /* ... */ }
// ...
}
REST API
Express.js
router = express.Router();
router.get('/api/container/:id', (req, res, next) => {
data = adapter.getData(req.param.id);
if (!data) {
next(Error('Could not get data'));
}
res.json(data);
});
REST API
async Express.js
import * as asyncHandler from 'express-async-handler';
router = express.Router();
router.get('/api/container/:id', asyncHandler(async (req, res, next) => {
data = await adapter.getData(req.param.id);
if (!data) {
next(Error('Could not get data'));
}
res.json(data);
}));
REST API
How to handle
GET /api/container/:id
if id is
/thesis/12/34/56/123456?
Base64 url encode:
L3RoZXNpcy8xMi8zNC81Ni8xMjM0NTY
GET /api/container/L3RoZXNpcy8xMi8zNC81Ni8xMjM0NTY
REST API
Express middleware
export function decodeParam(key: string) {
return function(req: Request, res: Response, next: NextFunction) {
if (typeof req.params[key] === 'string') {
req.params[key] = base64url.decode(req.params[key]);
}
next();
};
}
router.route('/container/:id')
.all((req, res, next) => {
console.log(req.param.id); // L3RoZXNpcy8xMi8zNC81Ni8xMjM0NTY
next();
})
.all(decodeParam('id'))
.all((req, res, next) = > {
console.log(req.param.id); // /thesis/12/34/56/123456
next();
});
REST API
export function containerApi(container: ThesisContainerAdapter,
printshop: PrintShopAdapter,
isbnHandler: ISBNHandler): Router {
const router = express.Router();
router.use('/', ensureAuthenticated());
// ...
router.route('/:id/:property')
.all(decodeParam('id'))
.all(asyncHandler(ensureContainerOwner(container, 'id')));
.get(asyncHandler(async (req, res, next) => {
const data
= await container.getProperty(req.params.id, req.params.property, req);
res.json(data);
}))
.put(asyncHandler(async (req, res, next) => {
const result
= await container.putProperty(req.params.id, req.params.property, req.body, req);
res.json(result);
}));
// ...
return router;
}
REST API
import * as express from 'express';
import * as http from 'http';
import { containerApi } from './routes/container';
app = express();
// get config somehow
const host = config.get( /* ... */ );
const port = config.get( /* ... */ );
app.set('host', host);
// init LdpClient, ThesisContainerAdapter, authentication
// ...
app.use(containerApi(containerAdapter, auth));
const server = http.createServer(app);
server.listen(port,
() => console.log(`Application listenting on http://${host}:${port}`));
Angular-klient
Siden server-koden er skrevet i javascript kan den deles mellom server og klient
Tre mapper
server/ - all server-spesifikk kode
client/ - all klient-spesifikk kode
model/ - kode som beskriver datamodellen som begge kan bruke
Angular-klient
Klienten har adgang til hjelpefunksjoner for dokumenter (husk addresse-eksempelet):
- Vi får hjelp å bygge opp valid JSON LD
Angular-klient
Klienten har adgang til Typescript-definisjoner for alle dokumenttyper:
- Veldig mange typer feil kan detekteres statisk ved kompilering.
Angular-klient
REST API
ContainerService
AuthService
...
...
FormHandler
FormComponent
FormStepperComponent
RxJS - Reactive programming/data flow
const subject = new Subject<string>(null);
const subscription =
subject.subscribe(value => console.log(value));
subject.next('foo');
subject.next('bar');
setTimeout(() => subscription.unsubscribe(), 1);
// Output:
// foo
// bar
RxJS - Reactive programming/data flow
const subject = new Subject<string>(null);
const upperCaseObservable = subject.pipe(
map(value => value.toUpperCase())
);
const subscription =
upperCaseObservable.subscribe(value => console.log(value));
subject.next('foo');
subject.next('bar');
setTimeout(() => subscription.unsubscribe(), 1);
// Output:
// FOO
// BAR
DocumentService
/** A service that can be used to access, update and persist a document */
export interface DocumentService<T extends CompactedDocument> {
/** Observable for the document */
valueChanges: Observable<T>;
/** Get an instant value of the document */
value: T;
/** Set the value of the document */
setValue(document: T): void;
/** Get the document once */
get(): Observable<T>;
/** Persist the document */
save(): Observable<T>;
/** Reset to null document */
reset();
}
ContainerService
export class ContainerService {
/** Service for the container's candidate */
candidate: DocumentService<Candidate>;
/** Service for the container's defense */
defense: DocumentService<Defense>;
// ...
private properties: DocumentService<CompactedDocument>[];
// ...
save(): Observable<any> {
return forkJoin(this.properties.map(property => property.save()));
}
}
FormHandler
export class FormHandler {
subscription: Subscription;
constructor(protected form: FormGroup,
protected service: DocumentService<CompactedDocument>) {
// init with latest value from server
this.service.get()
.subscribe(doc => this.patchForm(doc));
// listen for value changes from other events
this.subscription = this.service.valueChanges
.subscribe(doc => this.patchForm(doc));
// listen for value changes to form
this.subscription.add(
this.form.valueChanges
.subscribe(value => this.patchDocument(value))
);
}
// stop listening when form is destroyed (avoid memory leaks)
destroy() {
this.subscription.unsubscribe();
}
// ...
}
FormHandler
export class FormHandler {
// ...
patchForm(document: CompactedDocument) {
if (document) {
this.form.patchValue(document, {emitEvent: false});
}
}
patchDocument(formValue: any) {
const doc = this.service.value;
Object.assign(doc, formValue);
this.service.setValue(doc);
}
}
FormComponent
@Component({
selector: 'app-candidate',
templateUrl: './candidate.component.html',
styleUrls: ['./candidate.component.css']
})
export class CandidateComponent extends BaseFormComponent {
form: FormGroup;
constructor(
private container: ContainerService,
private fb: FormBuilder) {
super();
this.form = this.fb.group({
givenName: ['', Validators.required ],
familyName: ['', Validators.required ],
email: ['', [ Validators.required, Validators.email ] ]
});
this.addFormHandler(
new FormHandler(this.form, this.container.candidate)
);
}
}
FormStepperComponent
@Component({
selector: 'app-press-release-form',
templateUrl: './press-release-form.component.html',
})
export class PressReleaseFormComponent implements OnInit {
steps = [
{ label: { en: 'Candidate', nb: 'Personalia' }, type: CandidateComponent },
{ label: { en: 'Defense', nb: 'Disputas' }, type: EventsComponent }
];
// ...
}
<!-- press-release.component.html -->
<app-form-stepper [steps]="steps" [saveFunction]="saveFunction">
</app-form-stepper>
Tester
To hovedtyper
- Enhetstester
- Intergrasjonstester
Enhetstester
- Tester koden funksjon for funksjon
- Ofte bare test av de publike metoder/eksporterte funksjoner, de private/interne testes bare indirekte
- Målet er å dekke flest mulig alternativ og edge cases
- Hver funksjon studeres isolert, kall til andre biblioteker og spesielt databaser vil ofte "mockes" ut.
Integrasjonstester
- Tester koden fra dens eksterne grensesnitt, med databaser og lignende aktivt
- Sjekker at de forskjellige komponentene interagerer riktig
- Detekterer regresjoner
Eksempel enhetstest
describe('LdpClient', () => {
describe('exists()', () => {
it('should return true if backend finds uri', async () => {
mockAxios.onHead().reply(200);
await expect(ldp.exists('foo')).to.eventually.be.true;
});
});
});
// code to test
export class LdpClient {
async exists(path: string): Promise<boolean> {
try {
const response = await axios.head(this.root + path,
{ headers: { authorization: this.authorization } });
return response.status === 200;
} catch (err) {
// ...
}
}
}
Eksempel integrasjonstest
describe('/api/container/[:id]', () => {
const path = '/api/container';
it('should deny unauthenticated POST requests', async () => {
await request(server).post(path).expect(401);
});
it('should allow authenticated user to POST container', async () => {
sandbox.stub(_helpers, 'isAuthenticated').yields(null);
let id: string;
await request(server).post(path)
.expect(201)
.expect('Location', /^\w+$/)
.then(res => {
id = res.header.location;
expect(id).to.equal(base64url.encode(res.body['@id']),
'Location header id is base64url-encoded version of body id');
expect(res.body.creator).to.contain(user.id);
});
});
});
Coverage-analyse
- Måler hvilke linjer og forgreininger som er testet
- Måler ikke om alle mulige argument og parametere er testet.
- Måler ikke om alle mulige eksterne feil-kilder er testet
- Ved 100 % coverage vet man i alle fall at man ikke har noen skrivefeil i noen uvanlige if-condititions
- Noen deler av koden er lettere å teste med enhetstester og andre med integrasjon, holder med 100 % til sammen

# start test-db
cd server/integration-tests
docker compose up
cd ../..
# run all tests
npm run test
# alternatives
npm run test:server
npm run test:server:integration
npm run test:server:unit
npm run test:client
npm run e2e # nothing there yet
Hva gjenstår?
- Trykkeri-integrasjon
- Read-only integrasjoner:
- W3 (ITA)
- BORA (ITA)
- NB (NB)
- ...
- Høyere test-dekning for klient
- Dokumentasjon
- UX
- ...
Copy of BYOTh
By Ingrid Cutler
Copy of BYOTh
Bring Your Own Thesis
- 536