File Checker
A Service to check the equity between directories in Staging and Production/Asia
Specs
- User right clicks on a folder on staging
- User is asked a folder on Production/Asia to compare against
- The folder tree is compared with each other
- Comparing files is done binary level (hash)
- Any missing/extra/different files in production are logged in in job output
- As task is potentially long-lasting, it must be scheduled as job, that can be checked later (similar to video encoding jobs)
| Functionality | Deployment | |
|---|---|---|
| Backend APIs | done | done |
| Frontend | WIP | pending |
Development Status
-
POST /check
-
Lost of optimisations on space complexity.
-
Modeling, queries and connection to DB.
-
Test (unit test and to production).
-
- GET /check/list
- List all taks/jobs and it's status.
- Codes: 200, 404, 500, etc.
- GET /list/:id
- Detailed json with each file path being compared and the result of that comparation.
Development: BAckend
Creating the request: POST /check
Processing: /check/list taskId: 89.
Logs
Job Finished: Files checked 1171
{
"Id": 89,
"Created": "2021-12-13T07:39:09.367Z",
"Completed": "2021-12-13T07:40:42.327Z",
"SourceDirectory": "/787043/30/storage/0/3ae2d6d2-57e7-4e57-a8ba-e3973d6c9204",
"TargetDirectory": "/542609/roulette/live/30/0/v0",
"Status": 200,
"Result": "{\"countFiles\":1171,\"ok\":1171,\"different\":0,\"notFound\":0,\"error\":0}"
}Status
100: Processing.
200: All files in staging are equal to the ones in production.
404: There is at least one file that was not found in production.
500: There was an error performing this tasks.
300: There were multiple errors: Some not found, some errored.
409: There were at least one file different.
Database
ComparisonTasks
ComparisonTasksFiles
Database
Pending
- Pagination on list tasks endpoints.
- Another optimisation on big lists of files.
- Documenting:
- APIs.
- Deployment process.
Known Issues
- When a user creates a directory with the format: 11111111-1111-1111-1111-11111111 the path constructor to production fails.
- With some very big directories (>2000 files) it may fails with memory errors. Need more test and optimizations.
- 3 Views.
- 2 for tasks/job status: Master-detail.
- 1 for Comparison.
- Comparison view.
- Selection of two directories.
- opt: right click to select staging and redirection to comparison view.
Development: Frontend
Ex. Tasks status (Master)
Ex. Tasks status (Detail)
Concept.
Comparison view
File Checker
A Service to check the equity between directories in Staging and Production
Minimal
By Sam W
