Flink Operations

Flink Slots

Task managers have "slots" a "slot" can be thought of as a worker
Because of context shifts, flink recommends one slot per core: https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/config.html#configuring-taskmanager-processing-slots
We can break this rule, but this means performance tuning, at first we had many slots per core, not much work was done because of hand offs between threads.

Job and Operator Parallelism

We can set parallelism per job or per operator
If we set it per job, all operators appear to get the same parallelism
Tuning each operator is part of the flink production ready checklist and should always be done: https://ci.apache.org/projects/flink/flink-docs-release-1.2/ops/production_ready.html

Relationship between slots and Parallelism

In order to properly tune an operator, you need to know how many slots will be consumed
Understanding how to count this relies on understanding operator "chaining" vs "unchaining" (by default operators are "chained" but this is configurable via the streaming api)
If you're "unchaining" an operator, the slots appear to equal parallelism, that means each operator instance gets it's own thread.
For "chaining" flink notices that many times the same thread could continue executing the next operation without thread headoff/network traffic, so the operation is "chained".

Chaining vs Unchaining

https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/#task-chaining-and-resource-groups

Chaining vs Unchaining

https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/#task-chaining-and-resource-groups

Summary

Start with 1 slot per core per flink docs
Slots are difficult to calculate due to chaining, and will intentionally apply a heavy load to each box to avoid copying/shuffling data

Snapshots

At least Once vs Exactly Once

If execution pauses from when the first barrier reaches an operator until the last barrier reaches an operator, we get exactly once processing, if we continue processing we get at least once.

(t1, B, t2) -> Op1

(t3, t4, B) ----^

https://ci.apache.org/projects/flink/flink-docs-master/internals/stream_checkpointing.html

Where to view Stats?

Summary

Snapshot time is a funny way to say "worst case stream latency".

Until we can make all streams flow more evenly, we need to use at least once processing.

The actual time we take snap shotting is just a second or two.

Async Operators

https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/asyncio.html

Summary

Async as much as we can

The flink docs around this topic are a mix of wrong and undocumented.