Tidy data

1. clean
2. tidy
3. non-tidy data

data cleaning

blank values

• unknown

• not applicable

• non existent

• 0, "", []

open time

missing: contains uncertainty!

non existent: ok

2019-11-01: impossible

#legs

0: ok for snake

for tree: not applicable

-3: impossible

• 0.5 vs 50%

• 132 vs "132"

• 105% open rate

• "gmail.com" vs "gmail.com   "

data tidying

clean & flexible

Formal rules

• column           variable

• observation           row

• table            variable type

• tables are linked

? variable, ? observation

compare observations

combine variables

ggplot(dt, aes(x = date, y = value, col = variable)) + 
    geom_point() + 
    geom_line() + 
    labs(x  = NULL, y = NULL) + 
    theme(legend.title = element_blank())

ggplot(dt, aes(x = date)) + 
    geom_point(aes(y = num_send), col = ems_colors[['green1']]) + 
    geom_line(aes(y = num_send), col = ems_colors[['green1']]) + 
    geom_point(aes(y = num_open), col = ems_colors[['blue1']]) + 
    geom_line(aes(y = num_open), col = ems_colors[['blue1']]) + 
    labs(x = NULL, y = NULL) + 
    geom_point(data = data.table('v' = c('num_send', 'num_open'), 
                                 'date' = as.Date('2017-03-01'),
                                 'y' = 2500),
               mapping = aes(col = v, y = y, shape = NA)) + 
    geom_line(data = data.table('v' = c('num_send', 'num_open'), 
                                 'date' = as.Date('2017-03-01'),
                                 'y' = 2500),
               mapping = aes(col = v, y = y, linetype = NA)) + 
    theme(legend.title = element_blank())

ggplot(dt, aes(x = date, y = num_open / num_send)) + 
    geom_point() + 
    geom_line() + 
    scale_y_continuous(labels = scales::percent_format()) + 
    labs(x = NULL, y = 'open rate')

gather
separate
spread

unite

gather

spread

separate

unite

separate

unite

Non-tidy data

• efficiency

• history

graph
corpus
matrix

        data  

tools & usage