Neural Networks and Deep Learning

by deeplearning.ai

딥레스콜라 스터디그룹

모두의연구소 DLC 강화학습Lab 박석

Week 1

Welcome

AI = New Electricity ?

Leading Big Transformation

This course

Neural Networks and Deep Learning - Cat Recognition
Improving DNN by Hyper-parameter Tuning, Regularization and Optimization
Structuring your ML Project - Train/Dev/Test - Very Tough
Convolutional Neural Network - Image
Natural Language Processing - building sequence model by RNN,LSTM

Intro

Introduction

to deep learning

Be able to explain the major trends driving the rise of deep learning, and understand where and how it is applied today

What is neural network?

It is a powerful learning algorithm inspired by how the brain works.

Example 1 - Single NN

Example 2 - Multiple NN

Supervised learning for Neural Network

In supervised learning, we are given a data set and already know what our correct output should look like, having the idea that there is a relationship between the input and the output.

Supervised learning problems are categorized into "regression" and "classification" problems.

Examples - SL

Structured vs. Unstructured

Why DL is taking off?

Deep learning is taking off due to a large amount of data available through the digitization of the society, faster computation and innovation in the development of neural network algorithm.

Data + GPU H/W + Algorithm

1. Being able to train a big enough neural network

2. Huge amount of labeled data

Iterative NN Training Process

Faster computation helps to iterate and improve new algorithm

Interview

Geoffrey Hitton

1966 : 고등학교시절, 수학을 잘했던 친구로 부터 인간두뇌 홀로그램을 보게 되면서 처음 관심을 갖게 됨(Lashley's experiments : 기억은 두뇌의 여러부분에 흩어져서 저장되어 있다는 사실을 실험)
대학시절 : physiology and physics 로 시작...philosophy...결국...psychology 선택..."지능이란 무엇인가?" 라는 문제 풀기..
psychology는 해답을 주지 못했다...carpenter
PhD in AI @Edinburgh in Britain...couldn't get a job
1982 : backprop algorithm 연구 시작 @UCSD in California w/ David Rumelhart and Ron Williams ( 실제 최초의 아이디어는 Rumerlhart 것이지만...논문으로 최초로 나온 것은 Paul Werbos 이었다. 별로 관심을 받지 못했음)
1986 : backprop algorithm 논문발표 into Nature
1990 : Word Embedding 를 이야기 하기 시작 w/ Benzio

Geoffrey Hitton's Story

가장 멋진일 : Boltzmann machines 을 w/ Terry Sejnowski 발견한 일
2007 : resurgence of neural net 을 이끌어 내어 현재의 Deep Learning 이 나온 것은 restricted Boltzmann machines 의 역할이 크다.
1993 : 1st variational Bayesian learning 관련 논문 발표 w/ Van Camp
2014 : @Google about using ReLUs and initializing with the identity matrix
1987 : recirculation algorithm w/ Jay McClelland ( euroscientists의 Spike-timing-dependent plasticity 이론과 같음...where the new thing is good and the old thing is bad in the learning rule.)
2015 : Multiple Time Skill w/ Jimmy Ba => LSTM
Now : have a little Google Brain team in Toronto => Unsupervised Learning, Wegstein algorithm, GAN