NCEI/SESB 2016
Ingests, qc's and formats ASOS/AWOS instrument data incorporating other data sets.
ASOS: Automated Surface Observing System
Additional data on higher resolution (1-min/5-min also collected)
AWOS: Automated Weather Observing System
NOTE: This modem dial-in data source is referred to as ‘ASOS Instrument Network’ on subsequent slides.
Interactive QC on a subset of ~480 ASOS stations
LCD Publications, CD Publications
Flags set in Level 1 are reviewed by Met Tech
Level 1
Automated checks; sets flags for all but CRN and SURFRAD networks
Level 2
ISD Format checking; run in DS and again by DAB All stations receive this level of QC at a minimum
Level 3
Initial Ingest and automated QC
Computer-Aided Manual QC
Final Formatting, publishing and Archival
Stage 1 - Ingest & QC
Stage 2 - Manual QC
Stage 3 - Publishing
Workflow
Stages over time
Codebase Lacks consistency across system.
Missing effective or comprehensive error handling.
Leverages approaches and language features that are no long preferred for system development.
Un-integrated polyglot system.
(FORTRAN, Java, Bash, Cron, etc.)
No comprehensive logging.
No error resilience and high reliability requirements.
Boundaries between workflow and computational tasks inconsistent.
(i.e. little internal uniformity)
Primarily manual introducing a high risk of un-recoverable human errors.
Inflexible deployment approach.
Issues require human intervention from a skilled operator familiar with the system architecture, etc.
Single point of failure (Humbolt)
Missing means to monitor status in real-time without understanding the tasks being taken and where to look for intermediate results.
No resilience to intermittent ASOS device availability.
Nicely de-composable at the task and unit level.
Many tasks are not precision math with high computational cost.
Significant amounts of dead code in current codebase.
Output formats well-defined and slow-changing.
ASOS Instrument Network
Changes very slowly.
Data format extremely stable.
Dial-in tech unchanged for almost a decade.
Upcoming improvements include direct networking access.
MADIS already provides almost all data we obtain via dial-up.
Phase 1 (Ingest)
Each functional unit (i.e. CF6 ingest) is broken down separately followed by the total for the ingest phase:
Sources
METAR: 2037
DSM/MSM: 2146
CF6: 878
ASOS Network: 2488
MAPSO: 3130
Total LOC: 10,679
The codebase totals approximately 75k lines of executable code. Most of this is generated. Approximately 50% is business logic.
This section is monolithic being a number of complex BASH scripts that are hard-coded into one piece of application logic.
Contents
Scripts: 12,605
Ish_drvr.f: 370
Blddfty.f: 267
Mk3505name.f: 53
Fixpacisd.f: 146
Total LOC: 13,441
Phase 3 (Publishing)
Workflow Stages
Units
Modules
A module comprises a single computational task communicating results via a plugin-api.
Error Handling
Logging
Events
Module API
A unit is a conceptually grouped set of tasks comprised of modules and workflow.
Modules
Modules
Unit API
Ex: Unit Completed Action
Ex: Recoverable Module error
Encapsulates and executes business logic of set of functional units.
Unit
Unit
Unit
Enforces Business Logic
Handles Errors
Ingest System
Archive
Downstream Processing
User Activity
Reporting
Remove dead code.
Move hard-coded paths, deployment/runtime configuration options into a system-global set of properties that can be set during deployment or at runtime.
Build a comprehensive and well-structured code repository.
Add automated build and deployment tools.
See Assessment for detailed breakdown of tasks in each language.
Language-level concerns differ in scope and complexity for each language.
Reformat for consistency and readability.
Delete commented-out code.
Add or enhance top-level documentation for each file.
Eliminate hard-coded paths, emails, etc., which are pervasive throughout most of the files.
Replace ‘include’ directives with ‘use’ statements pointing to well-organized modules.
Merge files containing only very small subroutines or variable declarations into larger files to eliminate the need for 80 separate Fortran files.
Reorganize files according to related functionality, rather than keeping all Fortran files together in a single directory.
Very minimal GOTO removal only where obvious and easy.
Using MADIS to upgrade ingesting
'High-Res' ASOS Instrument data.
Unit API
(ASOS Instrument Network)
Unit
Unit
Unit
Unit
Functional Module API
Generalized Functional Module API and container implemented
(ASOS Instrument Network)
Workflow Engine designed to integrate into Ingest System
(ASOS Instrument Network)
Station Unreachable
Dial-Out Functional Unit
Put Station back on processing list
Modems Down
Unrecoverable Error
Process Station List
Log Event
Escalate Failure/Retry Unit
Halt Processing
API's allow targeted re-design that can be pushed to production.
Units
Unit API
Unit API
Data Retrieval
Parsing and formatting
MADIS
Data Retrieval
Parsing and formatting
MADIS
Functional Unit API
(NOTE: All estimates are in 'Full-Time Employee' time. See Document for details)
There are a number of 'orthogonal' improvements that could be done in arbitrary order (if done at all).
FORTRAN: 8-16 FTE months
JAVA: 9–18 FTE months
BASH: ~7 FTE months
Architecture: 5‐10 FTE weeks
Total: ~2.1 - 3.6 FTE years
Pros
Cons
Migrate off IBM Server
Refactor FORTRAN code
Refactor Java code
Re-write BASH
Pros
Cons
"Mission critical re-engineering, no architecture."
Migrate off IBM Server
Pros
Cons
(Recommended)
The approach we believe that is most likely to succeed in addressing both short-term concerns and longer-term organizational goals is an incremental approach that advances a mature architecture and is compatible with the common ingest system under development for future deployment on the 3-tier production environment.