File Checker

A Service to check the equity between directories in Staging and Production/Asia

Specs

  • User right clicks on a folder on staging
  • User is asked a folder on Production/Asia to compare against
  • The folder tree is compared with each other
  • Comparing files is done binary level (hash)
  • Any missing/extra/different files in production are logged in in job output
  • As task is potentially long-lasting, it must be scheduled as job, that can be checked later (similar to video encoding jobs)
Functionality Deployment
Backend APIs done done
Frontend WIP pending

Development Status

 

  • POST /check

    • Lost of optimisations on space complexity.

    • Modeling, queries and connection to DB.

    • Test (unit test and to production).

  • GET /check/list
    • List all taks/jobs and it's status.
    • Codes: 200, 404, 500, etc.
  • GET /list/:id
    • Detailed json with each file path being compared and the result of that comparation.

Development: BAckend

Creating the request: POST /check

Processing: /check/list    taskId: 89.

Logs

Job Finished: Files checked 1171

{
  "Id": 89,
  "Created": "2021-12-13T07:39:09.367Z",
  "Completed": "2021-12-13T07:40:42.327Z",
  "SourceDirectory": "/787043/30/storage/0/3ae2d6d2-57e7-4e57-a8ba-e3973d6c9204",
  "TargetDirectory": "/542609/roulette/live/30/0/v0",
  "Status": 200,
  "Result": "{\"countFiles\":1171,\"ok\":1171,\"different\":0,\"notFound\":0,\"error\":0}"
}

Status

100: Processing.

200: All files in staging are equal to the ones in production.

404: There is at least one file that was not found in production.

500: There was an error performing this tasks.

300: There were multiple errors: Some not found, some errored.

409: There were at least one file different.

Database

ComparisonTasks

ComparisonTasksFiles

Database

Pending

  • Pagination on list tasks endpoints.
  • Another optimisation on big lists of files.
  • Documenting:
    • APIs.
    • Deployment process.

Known Issues

  • When a user creates a directory with the format: 11111111-1111-1111-1111-11111111 the path constructor to production fails.
  • With some very big directories (>2000 files) it may fails with memory errors. Need more test and optimizations.

 

  • 3 Views.
    • 2 for tasks/job status: Master-detail.
    • 1 for Comparison.
  • Comparison view.
    • Selection of two directories.
    • opt: right click to select staging and redirection to comparison view.

Development: Frontend

Ex. Tasks status (Master)

Ex. Tasks status (Detail)

Concept.

Comparison view

File Checker

A Service to check the equity between directories in Staging and Production

Made with Slides.com