Deep Learning Frameworks

 

      Benjamin Akera

 

What is a Deep Learning framework?

Deep learning frameworks offer building blocks for designing, training and validating deep neural networks, through a high level programming interface

A tale of Three Deep Learning Frameworks

Agenda for Today's talk

  • What is Deep Learning and why?
  • Short Survey of 3 DL frameworks
    • Tensorflow
    • Keras
    • Pytorch
  • Training Options
    • Single node
    • Distributed
  • Q&A

What is Deep Learning?

Composing representations of data in a hierarchical manner 

What's Tensorflow?

  • Open source from Google 2015
    • Current Version 2.13 API
    • 2.0 in early alpha
    • Declarative Toolkit
  • Fast: Backend C/C++
  • Data flow graphs
    • Nodes are functions/operators
    • Edges are input or data (tensors)
    • Lazy Execution
    • Eager execution (1.7)

Tensorflow Key API Concepts

x = tf.constants(42,name='x')
w = tf.Variable(1.34, name = 'w')
input = tf.Placeholder("float")

with tf.Session() as sess:
1,[1,2],[[2,3]],[4,5]] ...

g = tf.Graph("my_graph")
with g.as_default():
    c = tf.add(x,w)
    m = tf.matmul(a,b)
c = tf.add(x,w);
m = tf.matmul(a,b)
  • Constants
  • Variables
  • Placeholders
  • Operations
  • Sessions
  • Tensors
  • Graphs

Tensorflow Code

import tensorflow as tf

a = tf.placeholder(tf.float32,shape=(2,1))
b = tf.placeholder(tf.float32,shape=(1,2))


c = tf.matmul(a,b)

sess = tf.Session()
print(sess.run(c,{a:[[1],[2]],b:[[3,4]]}))


[[3.4.][6.8.]]

3-4: create tf placeholder types, a&b  Define their input shapes as tensors
1: Import tensorflow
9: create a Tf Session, with input parameters for placeholders 'a' & 'b'
'c' as an operation won't run until sess.run() Lazily evaluated
13: TF Session output

Tensorflow Code: MNIST


import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot = True)

## Create the model
x = tf.placeholder(tf.float32,[None,784])
w = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x,W) + b
y_ = tf.placeholder(tf.int64, [None])
...
# Define Loss and optimizer
cross_entropy = tf.losses.sparse_softmax_cross_entropy(labels=y_,logits=y)
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

# Create Session
sess = tf.InteractiveSession()
tf.global_variables_initializer().run

## Train
for _ in range(1000):
    batch_xs,batch_ys = mnist.train.next_batch(100)
    sess.run(train_steps, feed_dict{x:batch_xs, batch_ys})


Tf placeholders and variables
Define our model TF variable for predicted value y
Train the model

...

Estimators

Layers

GPU

TPU

Android

iOS

...

Datasets

Metrics

Keras Models

Python

C++

Java

Go

Tensorflow Distributed Execution Engine

CPU

High-Level TF
APIs
Mid-Level TF
APIs
Low-Level TF
APIs
TF Kernel
Use canned estimators
Build models

TF Stack

   Why Tensorflow?

  • 124K+ stars!
  • 70K+ forks
  • Popular opensource code
  • Tensorflow Hub & Blog
    • Code Example & tutorials
    • Learn + share from others

1. Community:

Why Tensorflow?

2. Tools

Visualizing Tensors: Tensorboard

Deploy + serving models: TFX

Tensorflow:  We get it....So what?

  • Steep learning curve, But Powerful
  • Low level APIs, but offers control!!
  • Expert machine learning, just learn!!
  • Tensorflow 2.0, ease of use , eager execution!
  • Better, keras integration helps, indeed!
  • Non intuitive debugging
  • Open source Python library APIs for Deep Learning
    • Current version 2.3.3 APIs  by Francois Chollet(Google)
  • APIs: With Tensorflow, CNTK and Theano backends
  • Easy to use declarative APIs
    • Build layers
      • Great for neural network applications
        • CNN
        • RNN and LSTMs
  • Fast Experimentation, modular, extensible

Why Keras?

  • Focuses on Developer Experience
  • Popular & Broader community
  • Supports multiple back ends
  • Modularity
    • Sequential
    • Functional

Keras Code: MNIST

2.Compile Network

1. Define

Network

3.Fit Network

4. Evaluate

Network

5. Make Predictions

from keras import models
from keras import layers
mnist = tf.keras.datasets.mnist

(train_images, train_labels),(test_images, test_labels)=
    prepare_data(mnist.load_data())
network= models.Sequential()

network.add(layers.Dense(512, activation='relu',input_shape(28 * 28,)))

network.add(layers.Dense(10, activation='softmax'))

network.compile(optimizer=’rmsprop',loss='categorical_crossentropy',
                    metrics=['accuracy'])
results = network.evaluate(test_images, test_labels)

predictions = network.predict(new_images)
network.fit(train_images, train_labels,epochs=5, 
            batch_size=128)

Throwing Light on PyTorch 

The Pythonic Deep Learning Framework

 

What is PyTorch ?

(obviously its something with python)

The What and Why of Pytorch

  • Python based Deep Learning Framework
  • Developed by Facebook Research Group
  • Dynamic Computational Graphs
  • Easy Debugging
  • More Pythony 

What ?

Why ?

Why might you want to try PyTorch?

Research

Why might you want to try PyTorch?

NumPy

Why might you want to try PyTorch?

Celebrity endorsements

What makes it different

Dynamic computation graphs

What makes it different

Dynamic computation graphs

Sure but what does that really mean?

Can use regular control flow statements eg. for loops

Much easier to debug, can set breakpoints anywhere in the code

PyTorch is Imperative not Symbolic

a = 1
b = 3
c = a + b
d = c * 3
e = c - 5

print(d)

"e" is calculated

a = 1
b = 3
c = a + b
d = c * 3
e = c - 5

f = run(d)

print(f)

"e" is not calculated

Title Text

import numpy as np
import torch
from torch.autograd import Variable
model = torch.nn.Linear(1, 1)
loss_fn = torch.nn.MSELoss(size_average=False)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
for t in range(10000):
    x = Variable(torch.from_numpy(np.random.random((1,1)).astype(np.float32)))
    y = x * 3
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print loss.data[0]

Simple example PyTorch

Source: Tensorflow sucks

Title Text

import tensorflow as tf
import numpy as np
X = tf.placeholder("float")
Y = tf.placeholder("float")
W = tf.Variable(np.random.random(), name="weight")
pred = tf.multiply(X, W)
cost = tf.reduce_sum(tf.pow(pred-Y, 2))
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(cost)
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    for t in range(10000):
        x = np.array(np.random.random()).reshape((1, 1, 1, 1))
        y = x * 3
        (_, c) = sess.run([optimizer, cost], feed_dict={X: x, Y: y})
        print c

Comparative code in Tensor Flow 

Source: Tensorflow sucks

Best things about Tensor Flow that PyTorch isn't quite as good at

Deployment and multi GPU training

TensorBoard visualisation

In summary

Think about using Tensorflow if you

Are doing research and want to jump into production quickly

Want to take advantage of the huge TF community

Want to use tensorboard, Google Cloud Support + Many other tools

Want to get scooped by Google :) 

As an Expert etc

In summary

Think about using PyTorch if you

Are doing research, especially NLP

Like debugging things easily

Are a machine learning hipster

But also try Sonnet by DeepMind

In summary

Think about using Keras if you

Want to get started with Deep Learning 

Are a beginner

Honorable Mentions ...

Think about using CNTK if you

Want to use Deep Learning for Production on Microsoft Azure

Think about using DL4J if you

Are a Java Developer 

Think about using ONNX

For Interoperability with other frameworks 

To Conclude ...

Questions ?

 

BenjaminAkera

BenjaminAkera

akeraben@gmail.com 

Deep Learning Frameworks

By Benjamin Akera

Deep Learning Frameworks

Presentation made at the IndabaX ug workshop 5th April 2019

  • 60

More from Benjamin Akera