Presented by: Core Dev Team
Technical Walkthrough
BLOCKHAIN LAYER
STREAMING LAYER
STORAGE LAYER
CONSENSUS PROTOCOL
ARCHITECTURE OVERVIEW
NEM Catapult
PeerStream Protocol - Media Routing, Presence and Messaging
IPFS P2P Storage
Proof of Importance, Proof of Storage and Proof of Bandwidth
API Gateway
External DApp Development
IPFS (Interplanetary File System)
IPFS is a protocol and network designed to create a content-addressable, peer-to-peer method of storing and sharing hypermedia in a distributed file system
How IPFS Works
Bob adds a file to IPFS and is assigned a unique fingerprint called a cryptographic hash. Automatic deduplication of data prevents duplicate files.
(example: QmR7GSQM93Cx5e...)
Alice downloads the file using the unique hash.
When looking up files, you're asking the network to find nodes storing the content behind a unique hash
Ken downloads the same file using the hash.
Downloads simultaneously from different nodes as files are segmented into chunks, each with 256kb size
NEM Blockchain Platform
Block |
~120 TXNs Transaction 1..120 |
Block |
~120 TXNs Transaction 1..120 |
Block |
~120 TXNs Transaction 1..120 |
Block |
~120 TXNs Transaction 1..120 |
A message of at most 1024 bytes can be attached to each transfer transaction. This makes storing of large files in the blockchain impossible!
Combining NEM and IPFS
The ProximaX Solution
Block |
Transfer Transaction Message |
{
"digest":"04d185c832f6.....",
"hash":"QmWDQegEhLdCUWF6a...",
"keywords":"plain,file,text",
"metaData":"{'name':'some-data'}",
"name":"filename.txt",
"timestamp":1521154256963,
"type":"text/plain"
}
Field | Description |
---|---|
digest | This is a content based signature of the file for validation |
hash | The base58 encoded string IPFS reference |
keywords | A custom field that can be used to tag a specific file. Searchable |
metaData | JSON structured text for additional meta data. Searchable |
name | The name of the file (with or without extension) |
timestamp | The actual time that the file was uploaded to IPFS |
type | Content type of the file |
ProximaX P2P Storage Benefit
High Availability
Files can be replicated across peers to ensure that they are always available to be consumed by any peers on the network.
Fault tolerant
Peers on the network allows operational continuity in the event of single or multiple peer failure
Deduplication
The solution will have a data compression technology that detects duplicate files and avoid copying the same file to the
network.
ProximaX P2P Storage Benefit
Content-addressable storage
A mechanism for storing information that can be retrieved based on its content, not its storage location
Scalability
The storage solution will scale based on the number of peers that participates on the network
Web Service HTTP API
The P2P storage nodes will have a built in Exposed Web Services endpoints for developers to easily integrate the storage with their decentralised applications.
ProximaX Ecosystem
NEM Blockchain Platform Consensus Protocol
Full client ProximaX node - it runs an IPFS locally
Light client ProximaX node - only interfaces through the HTTP REST Endpoint
SDK DEMO
ProximaX Testnet Gateways
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
Testnet Router
Latency Based DNS Routing
Sample Application using ProximaX SDK
Lightweight Client
SDK DEMO 1
Upload Directly to Testnet2 Singapore
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
Sample Application using ProximaX SDK
Lightweight Client
SDK DEMO 2
Download from Testnet3 Sydney
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
Sample Application using ProximaX SDK
Lightweight Client
SDK DEMO 3
Upload using password
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
Sample Application using ProximaX SDK
Lightweight Client
File is symmetrically encrypted using PBKDF2 with HMAC, then SHA256 signed
SDK DEMO 4
Download using password
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
Sample Application using ProximaX SDK
Lightweight Client
File is symmetrically decrypted using PBKDF2 with HMAC, then SHA256 signed
Hands-on SDK DEMO
Upload using Lightweight Client
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
Lightweight Client
Application running locally using the ProximaX SDK
Hands-on SDK DEMO 1
Upload using Lightweight Client
private RemotePeerConnection remotePeerConnection
= new RemotePeerConnection("https://testnet.gateway.proximax.io");
/************************************************
* RemotePeerConnection
* DEMO 1: Uploads a file, announces a transaction with a message containing IPFS Hash and metadata
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/uploadfile", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> uploadFile() {
final Upload upload = new Upload(remotePeerConnection);
try {
final UploadFileParameter parameter = UploadFileParameter.create()
.senderOrReceiverPrivateKey(senderPrivateKey)
.receiverOrSenderPublicKey(receiverPublicKey)
.file(new File("files_upload/uploadfile.txt"))
.keywords("plain,file")
.metadata(Collections.singletonMap("MY_KEY", "MY_VALUE"))
.build();
// Upload the file
final UploadResult uploadResult = upload.uploadFile(parameter);
JSONObject jsonObject = new JSONObject()
.appendField("nem_hash", uploadResult.getNemHash());
return ResponseEntity.ok().body(jsonObject.toJSONString());
} catch (IOException | UploadException e) {
return ResponseEntity.badRequest().body(new JSONObject().appendField(
"error", e.getMessage()).toJSONString());
}
}
Hands-on SDK DEMO 2
Download using Lightweight Client
private RemotePeerConnection remotePeerConnection
= new RemotePeerConnection("https://testnet.gateway.proximax.io");
/************************************************
* RemotePeerConnection
* DEMO 2: Downloads the file uploaded from DEMO 1 using the returned nem transaction hash
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/downloadfile/{nemHash}", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> downloadFile(@PathVariable(value = "nemHash") String nemHash) {
final Download download = new Download(remotePeerConnection);
try {
// Download the file
final DownloadFileResult result = download.downloadFile(DownloadParameter.create()
.nemHash(nemHash).build());
Path path = Paths.get("files_download/" + nemHash + ".txt");
Files.write(path, result.getData());
return ResponseEntity.ok().build();
} catch (IOException | DownloadException e) {
return ResponseEntity.badRequest().body(new JSONObject().appendField(
"error", e.getMessage()).toJSONString());
}
}
Hands-on SDK DEMO
NEM Asymmetrical Encryption
Testnet1 Ireland
Testnet2 Singapore
Testnet3 Sydney
NEM
Public IPFS
File is asymmetrically encrypted/decrypted using NEM's Ed25519 private keys, then SHA256 signed.
Lightweight Client
Application running locally using the ProximaX SDK
Hands-on SDK DEMO 3
Upload using NEM Asymmetrical Encryption
private RemotePeerConnection remotePeerConnection
= new RemotePeerConnection("https://testnet.gateway.proximax.io");
/************************************************
* RemotePeerConnection - Privacy Strategy
* DEMO 3: Upload an encrypted file - NEM Key asymmetrical encryption
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/uploadsecurefile", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> uploadSecureFile() {
final Upload upload = new Upload(remotePeerConnection);
try {
final UploadFileParameter parameter = UploadFileParameter.create()
.senderOrReceiverPrivateKey(senderPrivateKey)
.receiverOrSenderPublicKey(receiverPublicKey)
.file(new File("files_upload/uploadsecurefile.txt"))
.keywords("secure,file")
.metadata(Collections.singletonMap("MY_KEY", "MY_VALUE"))
.securedWithNemKeysPrivacyStrategy() // Sets the privacy strategy using NEM Key asymmetrical encryption
.build();
// Upload the secure file
final UploadResult uploadResult = upload.uploadFile(parameter);
JSONObject jsonObject = new JSONObject()
.appendField("nem_hash", uploadResult.getNemHash());
return ResponseEntity.ok().body(jsonObject.toJSONString());
} catch (IOException | UploadException e) {
return ResponseEntity.badRequest().body(new JSONObject().appendField(
"error", e.getMessage()).toJSONString());
}
}
Hands-on SDK DEMO 4
Download using NEM Asymmetrical Encryption
private RemotePeerConnection remotePeerConnection
= new RemotePeerConnection("https://testnet.gateway.proximax.io");
/************************************************
* RemotePeerConnection
* DEMO 4: Downloads the encrypted file uploaded from DEMO 3 using the returned nem transaction hash
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/downloadsecurefile/{nemHash}", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> downloadSecureFile(@PathVariable(value = "nemHash") String nemHash) {
final Download download = new Download(remotePeerConnection);
try {
// Download the file
final DownloadFileResult result = download.downloadFile(
DownloadParameter.create()
.nemHash(nemHash)
.securedWithNemKeysPrivacyStrategy(senderPrivateKey, receiverPublicKey)
.build());
Path path = Paths.get("files_download/" + nemHash + ".txt");
Files.write(path, result.getData());
return ResponseEntity.ok().build();
} catch (IOException | DownloadException e) {
return ResponseEntity.badRequest().body(new JSONObject().appendField(
"error", e.getMessage()).toJSONString());
}
}
Hands-on SDK DEMO 5
Search by Keyword
private RemotePeerConnection remotePeerConnection
= new RemotePeerConnection("https://testnet.gateway.proximax.io");
/************************************************
* RemotePeerConnection
* DEMO 5: Search by Keyword
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/searchbykeyword/{keyword}",
produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> searchByKeyword(@PathVariable(value = "keyword") String keyword) {
Search search = new Search(remotePeerConnection);
try {
// Search the transactions associated to the NEM account by keyword
List<ResourceHashMessageJsonEntity> result = search.searchByKeyword(
senderPrivateKey, senderPublicKey, keyword);
JSONObject jsonObject = new JSONObject();
int index = 0;
for (ResourceHashMessageJsonEntity r : result) {
JSONObject jsonTemp = new JSONObject()
.appendField("ipfs_hash", r.getHash())
.appendField("keywords", r.getKeywords())
.appendField("metadata", r.getMetaData())
.appendField("digest", r.getDigest())
.appendField("name", r.getName());
jsonObject.appendField(Integer.valueOf(index++).toString(), jsonTemp);
}
return ResponseEntity.ok().body(jsonObject.toJSONString());
} catch (ApiException | InterruptedException | ExecutionException | PeerConnectionNotFoundException e) {
return ResponseEntity.badRequest().body(
new JSONObject().appendField("error", e.getMessage()).toJSONString());
}
}
Hands-on SDK DEMO
Full ProximaX Node (with IPFS)
Local Peer Connection
NEM
Public IPFS
Full ProximaX Node (with IPFS)
Application running locally using the ProximaX SDK with ProximaX Platform IPFS locally installed
Hands-on SDK DEMO 6
Upload using Local Peer Connection
private LocalHttpPeerConnection localHttpPeerConnection = new LocalHttpPeerConnection(
ConnectionFactory.createNemNodeConnection("http", "104.128.226.60", 7890), // NEM NODE
ConnectionFactory.createIPFSNodeConnection("/ip4/127.0.0.1/tcp/5001") // IPFS NODE
);
/************************************************
* LocalHttpPeerConnection
* DEMO 6: Uploads a file, announces a transaction with a message containing IPFS Hash and metadata
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/uploadfilelocal", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> uploadFileLocalConnection() {
// Set LocalHttpPeerConnection
final Upload upload = new Upload(localHttpPeerConnection);
try {
final UploadFileParameter parameter = UploadFileParameter.create()
.senderOrReceiverPrivateKey(senderPrivateKey)
.receiverOrSenderPublicKey(receiverPublicKey)
.file(new File("files_upload/uploadfilelocalconnection.txt"))
.keywords("plain,file,local")
.metadata(Collections.singletonMap("MY_KEY", "MY_VALUE"))
.build();
// Upload the file
final UploadResult uploadResult = upload.uploadFile(parameter);
JSONObject jsonObject = new JSONObject()
.appendField("nem_hash", uploadResult.getNemHash());
return ResponseEntity.ok().body(jsonObject.toJSONString());
} catch (IOException | UploadException e) {
return ResponseEntity.badRequest().body(new JSONObject().appendField(
"error", e.getMessage()).toJSONString());
}
}
Hands-on SDK DEMO 7
Download using Local Peer Connection
private LocalHttpPeerConnection localHttpPeerConnection = new LocalHttpPeerConnection(
ConnectionFactory.createNemNodeConnection("http", "104.128.226.60", 7890), // NEM NODE
ConnectionFactory.createIPFSNodeConnection("/ip4/127.0.0.1/tcp/5001") // IPFS NODE
);
/************************************************
* LocalHttpPeerConnection
* DEMO 7: Downloads the file uploaded from DEMO 6 using the returned nem transaction hash
*************************************************/
@RequestMapping(method = RequestMethod.GET, path = "/downloadfilelocal/{nemHash}", produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<String> downloadFileLocalConnection(@PathVariable(value = "nemHash") String nemHash) {
// Set LocalHttpPeerConnection
final Download download = new Download(localHttpPeerConnection);
try {
// Download the file
final DownloadFileResult result = download.downloadFile(DownloadParameter.create()
.nemHash(nemHash).build());
Path path = Paths.get("files_download/" + nemHash + "-local.txt");
Files.write(path, result.getData());
return ResponseEntity.ok().build();
} catch (IOException | DownloadException e) {
return ResponseEntity.badRequest().body(new JSONObject().appendField(
"error", e.getMessage()).toJSONString());
}
}
Encryption and Privacy Strategies
Plain Content Strategy
Content is stored without any encryption and will be readable by any user downloading it
Secure with NEM Keys
Uses the user's NEM private and public keys to encrypt the file and can only be decrypted by the same set of keys. The file will be downlodable but needs to be decrypted with the same set of keys to get the decrypted file. The most secure way of encrypting the file
Secure with Password
Uses a 50 character passphrase to protect the file. It it set to 50 characters to add difficulty for any brute force attacks
Custom
Developers can use a custom encryption/decryption process
ProximaX P2P and NEM Network Setup
-
Public P2P Storage with Private Mijin blockchain Network
-
Private P2P Storage with Private Mijin blockchain Network
-
Private P2P Storage with Public NEM Network
- Public P2P Storage with Public NEM Network
Work in Progress
-
Explore the SDK for other features
- ProximaX Applications
-
Support for other programming languages
(C#, Typescript, Golang)
-
PeerStream Media streaming and messaging
-
Consensus Protocol in Catapult
- Developer Site - https://developer.proximax.io
ProximaX SDK Walkthrough - (with script)
By Carlo Cayos
ProximaX SDK Walkthrough - (with script)
ProximaX SDK demo deck with script
- 501