A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 1

Live coding a

1. Initialise a Crystal app

2. Make HTTP calls

3. Read from config files

4. Concurrently check URLs with Channels and Fibers

Design

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 2

Live coding a

Starts at 7pm UTC

1. Encapsulating data into classes

2. Type aliases

3. Extracting tasks into modules

4. Running tasks periodically

1. Encapsulating data into classes

2. Type aliases

3. Extracting tasks into modules

4. Running tasks periodically

1. Encapsulating data into classes

2. Type aliases

3. Extracting tasks into modules

4. Running tasks periodically

✅

✅

Previous design

New design

1. Encapsulating data into classes

2. Type aliases

3. Extracting tasks into modules

4. Running tasks periodically

✅

✅

✅

1. Encapsulating data into classes

2. Type aliases

3. Extracting tasks into modules

4. Running tasks periodically

✅

✅

✅

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 3

Live coding a

Starts at 7pm UTC

1. Better config handling

2. Sensible monkey patching

3. Logging across fibers

4. Graceful termination

Current design

Handling configuration

  • conventions
  • YAML mapping
  • custom deserialisers

1. Better config handling

2. Sensible monkey patching

3. Logging across fibers

4. Graceful termination

✅

Monkey patching

Using explicit types in method signatures

module Enumerable(T)
  def >>(channel : Channel(T))
    #...
  end
end

1. Better config handling

2. Sensible monkey patching

3. Logging across fibers

4. Graceful termination

✅

✅

Logging across fibers

  • logging fiber names
  • injecting loggers into modules with 
macro extended

1. Better config handling

2. Sensible monkey patching

3. Logging across fibers

4. Graceful termination

✅

✅

✅

Graceful termination

1. User sends
interruption

2. Generator stops

X

3. Fibers timeout

Graceful termination

  • Dealing with Signals
  • interrupting a fiber
  • timers, heartbeats and select statements

1. Better config handling

2. Sensible monkey patching

3. Logging across fibers

4. Graceful termination

✅

✅

✅

✅

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 4

Live coding a

Starts at 7pm UTC

Graceful termination

  1. User requests termination
  2. Pipeline Fibers are terminated in order
  3. Main Fiber terminates

✅

1. User sends
interruption

2. Generator stops

X

3. Fibers timeout

Graceful termination, heartbeat-based

Graceful termination, golang style

1. User sends
interruption

2. Close url stream

X

3. Propagate channel closure

4. Notify Main fiber

Owning a channel

We say that a Fiber creating a channel owns it

Closing a channel

Channel#close broadcasts the closure to all the receiving Fibers

Terminating workers

Idea: rely on a supervisor Fiber listening to a countdown Channel

Graceful termination

  1. User requests termination
  2. Pipeline Fibers are terminated in order
  3. Main Fiber terminates

✅

✅

Getting things done

Fibers at the end of the pipeline notify Main via a done channel

Graceful termination

  1. User requests termination
  2. Pipeline Fibers are terminated in order
  3. Main Fiber terminates

✅

✅

✅

Graceful termination, golang style

1. User sends
interruption

2. Close url stream

X

3. Propagate channel closure

4. Notify Main fiber

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 5

Live coding a

Starts at 7pm UTC

Current design

New design

Server

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 6

Live coding a

Starts at 7pm UTC

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 7

Live coding a

Starts at 7pm UTC

Current design

Server

Testing fibers

  1. Decoupling concurrency and business logic
  2. Testing fibers coordination
  3. Time-based testing
  4. Testing servers*

A concurrent,
terminal-based tool
written in Crystal

URL Checker

Part 8

Live coding a

Starts at 8pm UTC

On Saturday, 4 Jan

Today's plan

  1. Cleanup
  2. Alerting
  3. Polish the UI

1. Cleanup

  • Improve termination: when is the writer done?
  • Update diagnostic_logger dependency
  • Inline Enumerable#>>Channel

Current termination strategy

[done]

X

X

Main awaits...

X

... then terminates

Better termination strategy

[done]

X

X

Main awaits...

X

... then terminates

[done]

X

1. Cleanup

  • Improve termination: when is the writer done?
  • Update diagnostic_logger dependency
  • Inline Enumerable#>>Channel

✅

✅

✅

2. Alerting

  • Introduce Alerting stage
  • Move from tuples to types
  • Implement alerting logic
  • Display alert

Stages, updated

A simple alerting algorithm

  1. Track the last n failures
  2. If the elapsed time between the 1st and last most recent failures is less than t, then raise an alert

A simple alerting algorithm

Example: raise an alert if
3 failures occur in 10 minutes

A simple alerting algorithm

Most recent failures

A simple alerting algorithm

New failure

New failure

A simple alerting algorithm

Elapsed time = 6m

< 10m

ALERT!

A simple alerting algorithm

X

New failure

Let's code!

  1. Track the last n failures
  2. If the elapsed time between the 1st and last most recent failures is less than t, then raise an alert

3. Polish the UI

  • CRT: a shard for terminal UIs
  • The

cycle

Wow!

URL Checker

Part 8

Live coding a

Thanks for watching!

Made with Slides.com