How we kill people:

What if we brought Safety into Software

Thomas Depierre

haruspex.dev

@Di4naO

We build Dynamic Systems

Law of Stretched Systems

every system is stretched to operate at its capacity; as soon as there is some improvement, for example in the form of new technology, it will be exploited to achieve a new intensity and tempo of activity.

Larry Hirschhorn

The Knight Capital Group Accident

In laymen’s terms, Knight Capital Group realised a $460 million loss in 45-minutes. Remember, Knight only has $365 million in cash and equivalents. In 45-minutes Knight went from being the largest trader in US equities and a major market maker in the NYSE and NASDAQ to bankrupt.

We build Complex Systems

Complex systems are systems whose behaviour is intrinsically difficult to model due to the dependencies, relationships, or interactions between their parts or between a given system and its environment.

fault tolerance isn’t composable

Peter Alvaro

Your "Hello world !" is incredibly complex too.

This is a systemic problem.

 

All Software is concerned.

Yes your small tool too

 

Yes your "non critical software" too

Fun >>>>>>>> Business Critical >>>>>> Safety Critical

Reliability

  • Doing what is specified
  • Repeatedly
  • TDD, Type System, Proof, etc

Safety

  • Avoiding Loss Events
  • Financial, Assets, Human life
  • Systemic property
  • Not a technical problem only !

Safe >>>>>>>> Reliable

Software Systems are SocioTechnical

  • Live in an environment
  • Operators
  • Users
  • Writers/Engineers
  • Humans are part of the system
  • A Software System is not a technical only entity

Solutions ?

  • I do not have a perfect solution to offer.

  • But i can offer the same path that other engineering field use for Complex SocioTechnical systems

Cross Field Knowledge

Safety II

Safety Differently

Resilience Engineering

Cognitive Systems Engineering

Humans create safety continuously through normal work

How can we support them

Attitude

Learning

Tooling

Accept that failure will happen

 

Stop lying to ourselves about things like "run on my machine" or "that is not my problem"

Attitude

Stop doing Root Cause Analysis

 

Stop doing 5 Whys

 

Learning Reviews

Learning

Observability as first class citizen

 

Ergonomics. Please

 

Operable software

Tooling

"Ten challenges for making automation a 'team player' in joint human-agent activity"

https://ieeexplore.ieee.org/document/1363742

Where to go for more ?