How we create an Offline First app

with Persistence Storage

Timofey Lavrenuyk

KeepSolid

About me

From Odessa

8+ years in development

Have an experience of creating native apps

Love progressive web apps

At the beginning...

Offline Mode

Kernel

RPC Server

Native app architecture

Kernel

Cryprograpthy

Work with Local DB

Work with PDF

Connection with RPC Server

What

can do?

Then 2017 become...

We need a web-version of app

We urgently need a WEB APP!!!11

WEB-version

1.0

No Author mode

Depends on internet connection

Based on PHP back-end

Old WEB-version

Kernel

SPA

REST API

RPC Server

Old web-app architecture

Offline First web-application

Mission:

Write a web application as if there is no internet connection.

Has author mode

Works Offline

Not worse than native clients

New WEB-app

Kernel

SPA

Proxy

RPC Server

New web-app architecture

Static data

User data

File data

What to store in Offline?

Static data

Service Worker

{
  "globDirectory": "dist",
  "globPatterns": [
    "index.html",
    "*.js",
    "assets/**/*.{png,svg}",
    "assets/*.{png,svg}"
  ],
  "swSrc": "src/service-workers/main.worker.js",
  "swDest": "dist/service-worker.js"
}

workbox.config.js

{
  "assets": [
      "src/assets",
      "src/service-workers",
      "src/manifest.json",
      {
        "glob": "workbox-sw.js",
        "input": "node_modules/workbox-sw/build",
        "output": "./workbox-3.5.0"
      },
      {
        "glob": "workbox-core.dev.js",
        "input": "node_modules/workbox-core/build/",
        "output": "./workbox-3.5.0"
      },
      {
        "glob": "workbox-precaching.dev.js",
        "input": "node_modules/workbox-precaching/build/",
        "output": "./workbox-3.5.0"
      }
    ],
}

angular.json

importScripts('workbox-3.5.0/workbox-sw.js');

workbox.setConfig({
  debug: true,
  modulePathPrefix: 'workbox-3.5.0/'
});
workbox.skipWaiting();
workbox.clientsClaim();
workbox.precaching.precacheAndRoute([]);

main.worker.js

workbox injectManifest
workbox.precaching.precacheAndRoute([
  {
    "url": "index.html",
    "revision": "4f8109353581284e76b88e568d642376"
  },
  {
    "url": "0.js",
    "revision": "2af14762103b4c1f18e620bd30a53d2e"
  },
  {
    "url": "main.js",
    "revision": "ef2fb7f5913e6614c9ee3621ad299d1b"
  }
])

service-worker.js

Static data

User data

Service Worker

Persistent Storage

 CacheStorage

Not flexible

File API

FileWriter deprecated

Cookies

WATTT??

WebSQL

Deprecated

IndexedDB

Good limit

Nice browser support

Not deprecated

LocalStorage

Low limit

IndexedDB

object-oriented database

store objects indexed with key

asynchronous

has low level api

can store JS-objects and Blobs

Browser support

Debug

cd ~Library/Application\ Support/Firefox/Profiles/xxxxxxxx.default/

 storage/default/https+++app.keepsolid.com/idb

~Library/Safari/Databases/___IndexedDB/http_localhost_4200/KEEPSOLID_SIGN_DB

IndexedDB in practice

1. Open or create database

var open = indexedDB.open("MyDatabase", 1);

2. Create Schema

open.onupgradeneeded = () => {
    const db = open.result;
    const store = db.createObjectStore("MyObjectStore", {keyPath: "id"});
    const index = store.createIndex("NameIndex", ["name.last", "name.first"]);
};

4. Request data

const getJohn = store.get(12345); // by id
const getBob = index.get(["Smith", "Bob"]); // by index

3. Create transaction

open.onsuccess = () => {
    const db = open.result
    const tx = db.transaction("MyObjectStore", "readwrite");
    const store = tx.objectStore("MyObjectStore");
    const index = store.index("NameIndex");
}

5. Get data

getJohn.onsuccess = () => {
    console.log(getJohn.result.name.first);  // => "John"
};
getBob.onsuccess = () => {
    console.log(getBob.result.name.first);   // => "Bob"
};

5. Close transaction

tx.oncomplete = function() {
    db.close();
};

Boilerblate!!!1

Disadvantages of IndexedDB

  • Hard to maintain
  • No JOIN support
  • Cannot partially update document
  • No sort
  • Transaction Auto-commit

"No one uses IndexedDB in its pure form."

- All JS-developers

Library or

Wrapper

  • Localforage
  • Idb
  • ZangoDb
  • MiniMongo
  • jsStore
  • PouchDB
  • Dexie

DB Engine

  • YDN-DB
  • AlaSQL
  • Lovefield

SQL-like API

  • select, insert, update, delete
  • group by, order by, limit, skip
  • join

Crossbrowser 

  • Chrome
  • Firefox
  • IE 11+, Edge
  • Safari 10+

Great perfomance

  • Analyze and optimize queries

Create Schema

const schemaBuilder = lf.schema.create('KEEPSOLID_SIGN_DB', 1.0);

schemaBuilder.createTable('Documents')
    .addColumn('id', lf.Type.STRING)
    .addColumn('parentId', lf.Type.OBJECT)
    .addColumn('type', lf.Type.STRING)
    .addColumn('signOrder', lf.Type.NUMBER)
    .addColumn('encryptionKey', lf.Type.STRING)
    .addIndex('idxSignOrder', ['signOrder'], false, lf.Order.DESC);
    .addPrimaryKey(['id']);

schemaBuilder.connect().then((db) => {
  // Now we can work with database
});

Field types

  • String
  • Number
  • Integer (32bit)
  • Boolean
  • Object
  • Date
  • Array Buffer

SQL-like API

SQL
Lovefield
SELECT *
  FROM Documents
  WHERE type = "TEMPLATE"
Database
  .select()
  .from(document)
  .where(document.type.eq('TEMPLATE'))
  .exec()
    
SELECT encryptionKey
  FROM Documents
  WHERE signOrder >= 1
  ORDER BY signOrder DESC
  LIMIT 10
Database
  .select(document.encryptionKey)
  .from(document)
  .where(document.signOrder.gte(1))
  .orderBy(document.signOrder, lf.Order.DESC)
  .limit(10)
  .exec()
    
SQL
Lovefield
SELECT *
  FROM Documents d, Files f
  WHERE d.fileId = f.id
  AND d.id = '123'
Database
  .select()
  .from(document, file)
  .where(lf.op.and(
     document.fileId.eq(file.id),
     document.id.eq('123'),
  ))
  .exec()
    
SELECT * FROM document
  INNER JOIN file
  ON document.fileId = file.id
  WHERE document.id = '123'
Database
   .select()
   .from(document)
   .innerJoin(
     file,
     document.fileId.eq(file.id)
   )
   .where(document.id.eq('123'))
   .exec()
function idb_and (index1, keyRange1, index2, keyRange2, onfound, onfinish) {
    var openCursorRequest1 = index1.openCursor(keyRange1);
    var openCursorRequest2 = index2.openCursor(keyRange2);
 
    assert(index1.objectStore === index2.objectStore);
    var primKey = index1.objectStore.keyPath;
    
    var set = {};
    var resolved = 0;
 
    function complete() {
        if (++resolved === 2) onfinish();
    }
 
    function union(item) {
        var key = JSON.stringify(item[primKey]);
        if (!set.hasOwnProperty(key)) {
            set[key] = true;
            onfound(item);
        }
    }
 
    openCursorRequest1.onsuccess = function (event) {
        var cursor = event.target.result;
        if (cursor) {
            union(cursor.value);
        } else {
            complete();
        }
    }
 
    openCursorRequest2.onsuccess = function (event) {
        var cursor = event.target.result;
        if (cursor) {
            union(cursor.value);
        } else {
            complete();
        }
    }
}

Pure API

Interesting features

  • Create Schema in YAML (SPAC)
  • Storage Types: IndexedDB, Memory, Firebase
  • Support of Import/Export to/from Javascript-object
  • Data Observation support

Data Observation

var query = db.select()
              .from(documents)
              .where(documents.id.eq('1'));

var handler = function(changes) {
  // Will be called whenever data change occurs.
};
db.observe(query, handler);

db.update(documents)
  .set(documents.title, 'New Title')
  .where(documents.id.eq('1'))
  .exec();

db.unobserve(query, handler);

Nuances

Schema editing

Error log

Constraint error: (202) Attempted to insert NULL value to non-nullable field Documents.signOrder.

User data

How we connected

Lovefield to the project

Data Layer

UI

Redux

Web Worker

Indexed DB

Why Web Worker?

Less load on Main Thread

Performance

const document = {
    id: 1,
    name: 'Test'
}
myWorker.postMessage(document);

1) Clone (Structured Clone Algorithm)

2) Serialize

const newDocument = clone({
    id: 1,
    name: 'Test'
})
myWorker.postMessage(JSON.stringify(newDocument));

Transferable Objects

transferring data from one context to another

ArrayBuffer

Encoding API

const encoder = TextEncoder('utf-8');
const decoder = TextDecoder('utf-8');

const uInt8Array = encoder.encode('Some string');
const decodedString = decoder.decode(uInt8Array);
var uInt8Array = stringToUintArray(JSON.stringify(result));

myWorker.postMessage(uInt8Array.buffer, [uInt8Array.buffer]);

Performance of converting String to ArrayBuffer

Less RAM consumed

ArrayBuffer transfer is very fast

  

But it is not all that bad...

File Storage

ArrayBuffer

FileReader API

var fileReader = new FileReader();
fileReader.onload = (event) => {
    const arrayBuffer = event.target.result;
};
fileReader.readAsArrayBuffer(blob);
const blob = new Blob([arrayBuffer]);

const objectUrl = URL.createObjectURL(blob);

blob:https://app.keepsolid.com/d45b2410-ad60-45be-9359-3f43524698db

File data

  • Data transfer to the Web Worker is not always effective.
  • If you need to manipulate a lot of data, but it makes sense to use Web Worker 

Web Worker communication

Effect

Action

Redux

Web Worker

Success/Failure Action

Comutter

Redux Devtools

Limited Storage Size

Browser Limit
Chrome < 6%
Firefox < 10%
Safari < 50mb
Edge Depends of HDD size
IE11 < 250mb

  Free space is shared between all types of storage.

  • IndexedDB
  • LocalStorage
  • Cache
  • ...

How to be ready to exceed the limit?
 

Storage Quota Estimate API

navigator.storage.estimate().then(estimate => {
  // estimate.quota - how much bytes is available
  // estimate.usage - how many bytes of storage is used
});

Browser support

Private Mode

indexedDB == null

Storage Usage == 0

Works fine

Works fine

Cleaning Storage by Browser

How to prevent cleaning?
 

Browser Method
Chrome LRU
Firefox LRU
Safari none
Edge none

Persistent Storage API

navigator.storage.persist().then(function(persistent) {
    if (persistent)
     // The browser will not clean Storage
    else
     // Browser will clear Storage in case of low memory
}

Chrome will approve if:

  • Site bookmarked
  • Site added to Homescreen
  • The user actively interacts with the application.
  • Push notifications enabled 

Firefox

Browser support

Offline detection

Online and Offline events

window.addEventListener('online',  () => {
    // There is an internet connection
});

window.addEventListener('offline', () => {
    // There is no internet connection
});

Browser Support

Network Information API

navigator.onLine //true, false

navigator.effectiveType // 2g, 3g, 4g

Browser Support

But

Online WiFi connection

Ping Pong

Future of

Persistence Storage

Writable files API

File and Directory Entries API

Draft

In development

IndexedDB 2.0

Planned Indexed DB Improvements

  • Promises
  • Observers
  • Simple API
  • Better requests (More primitives, Query Language, Free text search)

Conclusion

Offline first was on native platforms a long time ago

and gradually becomes relevant on the web

Chrome Mobile

Chrome Desktop

Offline First app is

Persistent Storage

Browser APIs

Architecture

Service Worker

  Thanks for attention

 Timofey Lavrenuyk

@geek_timofey

Made with Slides.com