GUNNER

A task runner optimized for performance and flexibility

(c) README

Facts

  • Task runner for assets with caching system
  • First commit by Hendrik Klindworth in 2014
  • Written in JavaScript, runs on node.js
  • Used by Forge and Elvenar

Concepts

  • config.json - defines basic configuration
  • gunnerfile.js - runnable script, defines tasks
  • tasks - contain steps, can depend on other tasks
  • steps - file processing job, depends on operation
  • operation (AKA op) - piece of code to run a step
  • cache - system for avoiding running steps
  • cleanup - special mode to remove leftovers

Concepts: config.json

  • 3 mandatory fields
    • srcDir - source
    • destDir - destination
    • cacheDir - cache data
  • can contain any other fields
  • accessible as gunner.config
  • override with --setConfig
  • srcDir will be the working directory

Concepts: gunnerfile.js

  • normal javascript code
  • executed internally by gunner 
  • has prepared gunner environment
  • defines tasks with gunner API

Concepts: tasks

  • contain steps to process files
  • defined with gunner.addTask
  • executed sequentially
  • order defined by addTask calls
  • can depend on other tasks
    • dependencies specified in addTask
    • dependencies will be called first

Concepts: step

  • actual file processing job
  • defined as a part of task
  • runs in parallel with other steps of a task
  • object with 3 fields
    • src - glob string or function returning array of input file paths (relative to config.srcDir)
    • dest - destination path (relative to config.destDir)
    • op - operation to invoke on src files

Concepts: op(eration)

  • named and versioned reusable piece of code
  • invoked by gunner when processing steps
  • takes source files and destination
  • returns array of output files
  • object with 3 fields
    • name - string with a simple op identifier
    • hash - hash string produced from version and given configuration, used for caching
    • exec - actual function with operation code, preferably as asynchronous as possible

Concepts: cache

  • used to avoid processing unchanged assets
  • one cache JSON file per task
  • JSON object contains entries for steps
    • key is produced from op hash, input files and destination
    • value contains modification time and content hash for input and output files
      • modification time is a fast-path to avoid expensive content hash checks
  • a step is re-run when any of input or output files are changed/added/missing
  • cache file is updated after step processing

Concepts: cleanup

  • invoke with the same configuration and --cleanup
  • to be executed right after a normal run
  • reads cache files for defined steps and gets "known" output files
  • deletes any files in destDir that are not "known"
  • also deletes cache files in cacheDir for unknown tasks

Best practices?

  • one step per file for 1:1 processing
    • simple copy, image resize, etc.
  • no writing same file in different steps!
    • messes up cache resulting in step re-runs and potential asset regeneration
  • be mindful about op.hash
    • bump version when changing behaviour
    • normally hash step constructor arguments
      • avoid hashing callbacks! (e.g. gunner-flatten)
      • add constants by code to hash (e.g. magic strings/numbers)

How we use it

(in Forge Browser)

  • two-stage processing with two gunner configurations:
  • browser-asset​
    • collects assets from SVN checkout folders
    • places assets into folders as required by the client
    • simply copies ready-for-browser files, does additional processing for others
      • resize, convert and compress images
      • extract building animation "subframes"
      • convert sounds
  • browser-texture
    • collects prepared assets from browser-asset​
    • combines assets into atlases (texture or sounds)
    • generates manifest files

How we use it

(in Forge Browser)

  • after running browser-asset and browser-texture:
    • merge the output from both of them
    • run more manifest generation scripts
      • asset/reflib mapping
      • checksum symlinks
      • typed reflib code
    • commit and push the output to the game_foe_browser_resources repo
  • probably the messiest part and a huge TODO:
    • could/should be done with gunner in a browser-texture run
    • no need for symlinks anymore
    • mappings are messy
    • lots (?) of assets unused by client
    • etc.

Questions?

gunner

By Dan Korostelev

gunner

  • 439