Summary of the backend and tech stack used

slides.com/delestro/qim-platform-backend

Felipe Delestro

fima@dtu.dk

Accessible at https://platform.qim.dk/ with domain being managed by DTU Compute

Server is hosted in a VM at
comp-vmfima.compute.dtu.dk (130.225.69.184) 

My user fima has sudo rights in the VM, and is running the services.

https access from the internet, no VPN required

Nginx reverse proxy to http services running in the VM

Flask server runs the platform

Interfaces running with Gradio

sftp web client using Filestash

Access routes

Users can access the platfom directly from the internet, and interact with services that are running in the virtual machine

Authentication

  • Log in form is managed by Flask, and asks for username and password
     
  • Credentials are validated by an SSH connection to login.gbar.dtu.dk
     
  • An encrypted session (werkzeug & PBKDF2) for the user is kept on server side, used for future SSH interactions with the HPC cluster

Data processing

  • The platform's main service is the Jupyter launcher. No data processing occurs in the VM itself.
     
  • The user's session opens a SSH connection and submits a job to the HPC cluster, starting the Jupyter server.
     
  • Once the server is running, the platform creates an SSH tunnel between the HPC node and the VM, exposing the Jupyter server to the internet via the platform.
     
  • Access to the server is protected by a token, generated by the jupyter launcher interface.

Data ACCESS

  • Users can upload/download small datasets using the web sftp client (Filestash)
     
  • Large data upload is possible via globus, but users need to start their endpoint manually, establish the connection, and start the transfer themselves.
     
  • We plan to build a system that allows large data uploads directly from the platform interface, but nothing is operational yet.
     
  • As the Jupyter server is running on the HPC cluster, users can access their data stored in DTU's filesystem.

Other details

  • Documentation for the qim3d library is also hosted in the same VM; it uses an MKDocs server
     
  • The platform uses a simple SQLite database to store information about users and projects
     
  • We have an Admin dashboard,  and it's access is restricted to selected users (flag in the database)
     
  • The platform shows status about the HPC cluster (user's jobs, queues, nodes etc)
     
  • Source code is private and hosted in Compute's GitLab.

https://platform.qim.dk/