We

Machine

At every interface between 2 surfaces, friction consumes energy, produces heat and wears down moving parts.

Data friction is what happens between 'data surfaces': where data moves between people, substrates, organisations or machines. From one lab to another, from one discipline to another, from a sensor to a computer, or from one data format to another.

Every movement of data across an interface comes at a cost of time, energy and human attention.

source

source

Know what goes into your pipeline

Go out in the field

Metadata

Documentation

Why build a pipeline in the first place?

1. Speed and time saving

source

1. Speed and time saving
2. Humans out of the loop: less errors, independent of individuals
3. Better understanding: all the data processing jobs explicitly written down
4. Debugging

Single tool pipeline?

Multitool pipeline

CC Jonathan Cohen

What should be the next piece of tube in your pipeline?

Some tools will help you

Know the jargon

Time = €

Get's you out of the flow

xkcd.com/303

Coding friction

compiling

building

reloading

calling servers

CC Peter Morgan

Analysis

Slicing and dicing

Adjust parameters

Views and visualizations

Modelling and predictive analytics

Go the last mile

Building the pipe is not the goal,

providing insights and making informed decisions is

Communicate what you found

(and how you found it)

Consider the user!

Use the right tools & techniques

=LEFT((RIGHT((RIGHT(A2,LEN(A2)-FIND(“;”,A2))),LEN((RIGHT(A2,LEN(A2)-FIND(“;”,A2))))-FIND(“;”,(RIGHT(A2,LEN(A2)-FIND(“;”,A2)))))),FIND(“;”,(RIGHT((RIGHT(A2,LEN(A2)-FIND(“;”,A2))),LEN((RIGHT(A2,LEN(A2)-FIND(“;”,A2))))-FIND(“;”,(RIGHT(A2,LEN(A2)-FIND(“;”,A2)))))),1)-1)

Learning a new tool = friction