Creating database entries from

combined sources

File1.txt

Supplementary meta data

  • from MS Excel files
  • from the filename
  • arbitrarily added
col1, col2

1,          2.14,

2,          3.56,

3,          4.56

4,          5.43

5,          4.45

File2.txt
col1, col2

1,          2.14,

2,          3.56,

3,          4.56

4,          5.43

5,          4.45

Column format text files

File2.xls

id:             123b

name:       Jon

material:  Steel

date:        1/4/18

time:        12:00

method:   tensile

 

Flexible and extendable format

File3.txt
col1, col2

1,          2.14,

2,          3.56,

3,          4.56

4,          5.43

5,          4.45

  • JSON objects available in:
  • binary for speedy access
  • ASCII for human readability
{
  name : Jon
  col1_data : [1, 2, 3, 4]
  col2_data : [2.34, 3.45, 4.34, 5.31]
  filename : 2
}
{
  name : Dave
  col1_data : [1, 2, 3, 4]
  col2_data : [2.34, 3.45, 4.34, 5.31]
  filename : 2
}

Uploading to a local, remote or

distributed database

{
  name : Fred
  col1_data : [1, 2, 3, 4]
  col2_data : [2.34, 3.45, 4.34, 5.31]
  filename : 2
}
{
  name : Jon
  col1_data : [1, 2, 3, 4]
  col2_data : [2.34, 3.45, 4.34, 5.31]
  filename : 2
}

Cloud servers around the world

Local computer

  1. initial creation
  2. stress testing
  3. performance analysis
  4. pre-deployment validation
  • Improved accessibility
  • Multiple locations to reduce latency
  • Access to scalable compute resources

RESTfull APIs

Dormant code waiting for input, scalable for many concurrent users

--> URL based input -->

<-- JSON base output <--

www.localhost.com/?material=Eurofer&uploader=Jon

Indicates beginning

of arguments

 

argument name

argument value

 

indicates start

of new argument

 

indicates separation between argument name and value

=

&

?

can work with your own domain


Meta data filters are made dynamically according to the fields present in the database

Interactive graph with hover text, box zoom, pan and much more

Table displaying results of query updated with changes to the filter dropboxs

Dynamic GUI created on the fly using React.js and stateless functional components

Select the fields for the X and Y axis on the plot

All dropboxes show available contents for that field and can accept text input that filters contents

Graph requires more input before it is available

Required input is highlighted in red to guide user

Plotted items are added to a second table to allow removal after the items are no longer visible in the query results table

Technology used

 A web framework for creating the stateless GUI.

Allows app like performance by efficient update of the render

React.js

MongoDB

NoSQL database offering excellent scaling through horizontal

partition of data (sharding) and web friendly JSON-like format

Serverless functions offer URL endpoints and scalable infrastructure for RESTfull APIs

AWS Lambda

Automatic and dynamic workflow

React.js

MongoDB

RESTfull APSs

File1.txt
col1, col2

1,          2.14,

2,          3.56,

3,          4.56

4,          5.43

5,          4.45

File2.xls

id:             123b

name:       Jon

material:  Steel

date:        1/4/18

time:        12:00

method:   tensile

 

custom python scripts for each URL endpoint

Interactive GUI

Scalable database

Raw data files + meta data

URL request

JSON response

PyMongo Python library

Python scripts

Version control

Github