NCEI/SESB 2016
Ingests, qc's and formats ASOS/AWOS instrument data incorporating other data sets.
ASOS: Automated Surface Observing System
Additional data on higher resolution (1-min/5-min also collected)
AWOS: Automated Weather Observing System
NOTE: This modem dial-in data source is referred to as ‘ASOS Instrument Network’ on subsequent slides.
Interactive QC on a subset of ~480 ASOS stations
LCD Publications, CD Publications
Flags set in Level 1 are reviewed by Met Tech
Level 1
Automated checks; sets flags for all but CRN and SURFRAD networks
Level 2
ISD Format checking; run in DS and again by DAB All stations receive this level of QC at a minimum
Level 3
Initial Ingest and automated QC
Computer-Aided Manual QC
Final Formatting, publishing and Archival
Stage 1 - Ingest & QC
Stage 2 - Manual QC
Stage 3 - Publishing
Workflow
Stages over time
Codebase Lacks consistency across system.
Missing effective or comprehensive error handling.
Leverages approaches and language features that are no long preferred for system development.
Un-integrated polyglot system.
(FORTRAN, Java, Bash, Cron, etc.)
No comprehensive logging.
No error resilience and high reliability requirements.
Boundaries between workflow and computational tasks inconsistent.
(i.e. little internal uniformity)
Primarily manual introducing a high risk of un-recoverable human errors.
Inflexible deployment approach.
Issues require human intervention from a skilled operator familiar with the system architecture, etc.
Single point of failure (IBM Server)
Missing means to monitor status in real-time without understanding the tasks being taken and where to look for intermediate results.
No resilience to intermittent ASOS device availability.
Nicely de-composable at the task and unit level.
Many tasks are not precision math with high computational cost.
Significant amounts of dead code in current codebase.
Output formats well-defined and slow-changing.
ASOS Instrument Network
Changes very slowly.
Data format extremely stable.
Dial-in tech unchanged for almost a decade.
Upcoming improvements include direct networking access.
MADIS already provides almost all data we obtain via dial-up.
Phase 1 (Ingest)
Each functional unit (i.e. CF6 ingest) is broken down separately followed by the total for the ingest phase:
Sources
METAR: 2037
DSM/MSM: 2146
CF6: 878
ASOS Network: 2488
MAPSO: 3130
Total LOC: 10,679
The codebase totals approximately 75k lines of executable code. Most of this is generated. Approximately 50% is business logic.
This section is monolithic being a number of complex BASH scripts that are hard-coded into one piece of application logic.
Contents
Scripts: 12,605
Ish_drvr.f: 370
Blddfty.f: 267
Mk3505name.f: 53
Fixpacisd.f: 146
Total LOC: 13,441
Phase 3 (Publishing)
Pros
Cons
Migrate off IBM Server
Refactor FORTRAN code
Refactor Java code
Re-write BASH in modern language
Pros
Cons
"Mission critical re-engineering, no re-architecture."
Upgrade 'High-Res' ASOS ingest
Reformat for consistency and readability.
Delete commented-out code.
Add or enhance top-level documentation for each file.
Eliminate hard-coded paths, emails, etc., which are pervasive throughout most of the files.
Replace ‘include’ directives with ‘use’ statements pointing to well-organized modules.
Merge files containing only very small subroutines or variable declarations into larger files to eliminate the need for 80 separate Fortran files.
Reorganize files according to related functionality, rather than keeping all Fortran files together in a single directory.
Very minimal GOTO removal only where obvious and easy.
Pros
Cons
(Recommended)
Migrate off IBM Server
Address global concerns/minimal refactoring
Implement Unit API
Integrate API with FORTRAN code
Integrate API with Java code
Upgrade 'High-Res' ASOS ingest
Re-write Bash in modern language
(Optional) Integrate with Common Ingest under development.
Workflow Stages
Units
Modules
A module comprises a single computational task communicating results via a plugin-api.
Error Handling
Logging
Events
Module API
A unit is a conceptually grouped set of tasks comprised of modules and workflow.
Modules
Modules
Unit API
Ex: Unit Completed Action
Ex: Recoverable Module error
Encapsulates and executes business logic of set of functional units.
Unit
Unit
Unit
Enforces Business Logic
Handles Errors
Ingest System
Archive
Downstream Processing
User Activity
Reporting
The approach we believe that is most likely to succeed in addressing both short-term concerns and longer-term organizational goals is an incremental approach that advances a mature architecture and is compatible with the common ingest system under development for future deployment on the 3-tier production environment.
API's allow targeted re-design that can be pushed to production.
Units
Unit API
Unit API
Data Retrieval
Parsing and formatting
MADIS
Workflow Engine designed to integrate into Ingest System
Station Unreachable
Dial-Out Functional Unit
Put Station back on processing list
Modems Down
Unrecoverable Error
Process Station List
Log Event
Escalate Failure/Retry Unit
Halt Processing
Data Retrieval
Parsing and formatting
MADIS
Functional Unit API
Collects and distributes "High-Res" data. (1,5 Minute)