How we create an Offline First app
with Persistence Storage
Timofey Lavrenuyk
KeepSolid
About me
From Odessa
8+ years in development
Have an experience of creating native apps
Love progressive web apps
At the beginning...
Offline Mode
Kernel
RPC Server
Native app architecture
Kernel
Cryprograpthy
Work with Local DB
Work with PDF
Connection with RPC Server
What
can do?
Then 2017 become...
We need a web-version of app
We urgently need a WEB APP!!!11
WEB-version
1.0
No Author mode
Depends on internet connection
Based on PHP back-end
Old WEB-version
Kernel
SPA
REST API
RPC Server
Old web-app architecture
Offline First web-application
Mission:
Write a web application as if there is no internet connection.
Has author mode
Works Offline
Not worse than native clients
New WEB-app
Kernel
SPA
Proxy
RPC Server
New web-app architecture
Static data
User data
File data
What to store in Offline?
Static data
Service Worker
{
"globDirectory": "dist",
"globPatterns": [
"index.html",
"*.js",
"assets/**/*.{png,svg}",
"assets/*.{png,svg}"
],
"swSrc": "src/service-workers/main.worker.js",
"swDest": "dist/service-worker.js"
}
workbox.config.js
{
"assets": [
"src/assets",
"src/service-workers",
"src/manifest.json",
{
"glob": "workbox-sw.js",
"input": "node_modules/workbox-sw/build",
"output": "./workbox-3.5.0"
},
{
"glob": "workbox-core.dev.js",
"input": "node_modules/workbox-core/build/",
"output": "./workbox-3.5.0"
},
{
"glob": "workbox-precaching.dev.js",
"input": "node_modules/workbox-precaching/build/",
"output": "./workbox-3.5.0"
}
],
}
angular.json
importScripts('workbox-3.5.0/workbox-sw.js');
workbox.setConfig({
debug: true,
modulePathPrefix: 'workbox-3.5.0/'
});
workbox.skipWaiting();
workbox.clientsClaim();
workbox.precaching.precacheAndRoute([]);
main.worker.js
workbox injectManifest
workbox.precaching.precacheAndRoute([
{
"url": "index.html",
"revision": "4f8109353581284e76b88e568d642376"
},
{
"url": "0.js",
"revision": "2af14762103b4c1f18e620bd30a53d2e"
},
{
"url": "main.js",
"revision": "ef2fb7f5913e6614c9ee3621ad299d1b"
}
])
service-worker.js
Static data
User data
Service Worker
Persistent Storage
CacheStorage
Not flexible
File API
FileWriter deprecated
Cookies
WATTT??
WebSQL
Deprecated
IndexedDB
Good limit
Nice browser support
Not deprecated
LocalStorage
Low limit
IndexedDB
object-oriented database
store objects indexed with key
asynchronous
has low level api
can store JS-objects and Blobs
Browser support
Debug
cd ~Library/Application\ Support/Firefox/Profiles/xxxxxxxx.default/
storage/default/https+++app.keepsolid.com/idb
~Library/Safari/Databases/___IndexedDB/http_localhost_4200/KEEPSOLID_SIGN_DB
IndexedDB in practice
1. Open or create database
var open = indexedDB.open("MyDatabase", 1);
2. Create Schema
open.onupgradeneeded = () => {
const db = open.result;
const store = db.createObjectStore("MyObjectStore", {keyPath: "id"});
const index = store.createIndex("NameIndex", ["name.last", "name.first"]);
};
4. Request data
const getJohn = store.get(12345); // by id
const getBob = index.get(["Smith", "Bob"]); // by index
3. Create transaction
open.onsuccess = () => {
const db = open.result
const tx = db.transaction("MyObjectStore", "readwrite");
const store = tx.objectStore("MyObjectStore");
const index = store.index("NameIndex");
}
5. Get data
getJohn.onsuccess = () => {
console.log(getJohn.result.name.first); // => "John"
};
getBob.onsuccess = () => {
console.log(getBob.result.name.first); // => "Bob"
};
5. Close transaction
tx.oncomplete = function() {
db.close();
};
Boilerblate!!!1
Disadvantages of IndexedDB
- Hard to maintain
- No JOIN support
- Cannot partially update document
- No sort
- Transaction Auto-commit
"No one uses IndexedDB in its pure form."
- All JS-developers
Library or
Wrapper
Localforage Idb - ZangoDb
- MiniMongo
- jsStore
- PouchDB
- Dexie
DB Engine
- YDN-DB
- AlaSQL
- Lovefield
SQL-like API
- select, insert, update, delete
- group by, order by, limit, skip
- join
Crossbrowser
- Chrome
- Firefox
- IE 11+, Edge
- Safari 10+
Great perfomance
- Analyze and optimize queries
Create Schema
const schemaBuilder = lf.schema.create('KEEPSOLID_SIGN_DB', 1.0);
schemaBuilder.createTable('Documents')
.addColumn('id', lf.Type.STRING)
.addColumn('parentId', lf.Type.OBJECT)
.addColumn('type', lf.Type.STRING)
.addColumn('signOrder', lf.Type.NUMBER)
.addColumn('encryptionKey', lf.Type.STRING)
.addIndex('idxSignOrder', ['signOrder'], false, lf.Order.DESC);
.addPrimaryKey(['id']);
schemaBuilder.connect().then((db) => {
// Now we can work with database
});
Field types
- String
- Number
- Integer (32bit)
- Boolean
- Object
- Date
- Array Buffer
SQL-like API
SQL
Lovefield
SELECT *
FROM Documents
WHERE type = "TEMPLATE"
Database
.select()
.from(document)
.where(document.type.eq('TEMPLATE'))
.exec()
SELECT encryptionKey
FROM Documents
WHERE signOrder >= 1
ORDER BY signOrder DESC
LIMIT 10
Database
.select(document.encryptionKey)
.from(document)
.where(document.signOrder.gte(1))
.orderBy(document.signOrder, lf.Order.DESC)
.limit(10)
.exec()
SQL
Lovefield
SELECT *
FROM Documents d, Files f
WHERE d.fileId = f.id
AND d.id = '123'
Database
.select()
.from(document, file)
.where(lf.op.and(
document.fileId.eq(file.id),
document.id.eq('123'),
))
.exec()
SELECT * FROM document
INNER JOIN file
ON document.fileId = file.id
WHERE document.id = '123'
Database
.select()
.from(document)
.innerJoin(
file,
document.fileId.eq(file.id)
)
.where(document.id.eq('123'))
.exec()
function idb_and (index1, keyRange1, index2, keyRange2, onfound, onfinish) {
var openCursorRequest1 = index1.openCursor(keyRange1);
var openCursorRequest2 = index2.openCursor(keyRange2);
assert(index1.objectStore === index2.objectStore);
var primKey = index1.objectStore.keyPath;
var set = {};
var resolved = 0;
function complete() {
if (++resolved === 2) onfinish();
}
function union(item) {
var key = JSON.stringify(item[primKey]);
if (!set.hasOwnProperty(key)) {
set[key] = true;
onfound(item);
}
}
openCursorRequest1.onsuccess = function (event) {
var cursor = event.target.result;
if (cursor) {
union(cursor.value);
} else {
complete();
}
}
openCursorRequest2.onsuccess = function (event) {
var cursor = event.target.result;
if (cursor) {
union(cursor.value);
} else {
complete();
}
}
}
Pure API
Interesting features
- Create Schema in YAML (SPAC)
- Storage Types: IndexedDB, Memory, Firebase
- Support of Import/Export to/from Javascript-object
- Data Observation support
Data Observation
var query = db.select()
.from(documents)
.where(documents.id.eq('1'));
var handler = function(changes) {
// Will be called whenever data change occurs.
};
db.observe(query, handler);
db.update(documents)
.set(documents.title, 'New Title')
.where(documents.id.eq('1'))
.exec();
db.unobserve(query, handler);
Nuances
Schema editing
Error log
Constraint error: (202) Attempted to insert NULL value to non-nullable field Documents.signOrder.
User data
How we connected
Lovefield to the project
Data Layer
UI
Redux
Web Worker
Indexed DB
Why Web Worker?
Less load on Main Thread
Performance
const document = {
id: 1,
name: 'Test'
}
myWorker.postMessage(document);
1) Clone (Structured Clone Algorithm)
2) Serialize
const newDocument = clone({
id: 1,
name: 'Test'
})
myWorker.postMessage(JSON.stringify(newDocument));
Transferable Objects
transferring data from one context to another
ArrayBuffer
Encoding API
const encoder = TextEncoder('utf-8');
const decoder = TextDecoder('utf-8');
const uInt8Array = encoder.encode('Some string');
const decodedString = decoder.decode(uInt8Array);
var uInt8Array = stringToUintArray(JSON.stringify(result));
myWorker.postMessage(uInt8Array.buffer, [uInt8Array.buffer]);
Performance of converting String to ArrayBuffer
Less RAM consumed
ArrayBuffer transfer is very fast
But it is not all that bad...
File Storage
ArrayBuffer
FileReader API
var fileReader = new FileReader();
fileReader.onload = (event) => {
const arrayBuffer = event.target.result;
};
fileReader.readAsArrayBuffer(blob);
const blob = new Blob([arrayBuffer]);
const objectUrl = URL.createObjectURL(blob);
blob:https://app.keepsolid.com/d45b2410-ad60-45be-9359-3f43524698db
File data
- Data transfer to the Web Worker is not always effective.
- If you need to manipulate a lot of data, but it makes sense to use Web Worker
Web Worker communication
Effect
Action
Redux
Web Worker
Success/Failure Action
Comutter
Redux Devtools
Limited Storage Size
Browser | Limit |
---|---|
Chrome | < 6% |
Firefox | < 10% |
Safari | < 50mb |
Edge | Depends of HDD size |
IE11 | < 250mb |
Free space is shared between all types of storage.
- IndexedDB
- LocalStorage
- Cache
- ...
How to be ready to exceed the limit?
Storage Quota Estimate API
navigator.storage.estimate().then(estimate => {
// estimate.quota - how much bytes is available
// estimate.usage - how many bytes of storage is used
});
Browser support
Private Mode
indexedDB == null
Storage Usage == 0
Works fine
Works fine
Cleaning Storage by Browser
How to prevent cleaning?
Browser | Method |
---|---|
Chrome | LRU |
Firefox | LRU |
Safari | none |
Edge | none |
Persistent Storage API
navigator.storage.persist().then(function(persistent) {
if (persistent)
// The browser will not clean Storage
else
// Browser will clear Storage in case of low memory
}
Chrome will approve if:
- Site bookmarked
- Site added to Homescreen
- The user actively interacts with the application.
- Push notifications enabled
Firefox
Browser support
Offline detection
Online and Offline events
window.addEventListener('online', () => {
// There is an internet connection
});
window.addEventListener('offline', () => {
// There is no internet connection
});
Browser Support
Network Information API
navigator.onLine //true, false
navigator.effectiveType // 2g, 3g, 4g
Browser Support
But
Online ≈ WiFi connection
Ping Pong
Future of
Persistence Storage
Writable files API
File and Directory Entries API
Draft
In development
IndexedDB 2.0
Planned Indexed DB Improvements
- Promises
- Observers
- Simple API
- Better requests (More primitives, Query Language, Free text search)
Conclusion
Offline first was on native platforms a long time ago
and gradually becomes relevant on the web
Chrome Mobile
Chrome Desktop
Offline First app is
Persistent Storage
Browser APIs
Architecture
Service Worker
Thanks for attention
Timofey Lavrenuyk
@geek_timofey
How we create an Offline First app with Persistence Storage
By Timofey Lavrenyuk
How we create an Offline First app with Persistence Storage
- 723