SANS

GERARD

Developer Evangelist

International Speaker

Spoken at 170 events in 37 countries

Serverless Training

appsyncmasterclass.com

What if an AI could do your job?

How long do you think it may take?

〞

AI is likely to overtake humans in the next five years

– Elon Musk

OpenAI

Greg Brockman

Elon Musk

Sam Altman

Co-Founders

Our mission is to build safe

artificial general intelligence (AGI)

that benefits all humanity.

2015

Introducing OpenIA

Initially a non-profit company, later on becomes "capped-profit", raises $1B.

2018

GPT is released

First Generative Pre-trained Transformer model with 117M parameters.

2020

GPT-3 and OpenAI API are released

Largest model to date with 175B parameters.

2021

GitHub Copilot, DALL-E and Codex are released

All with 12B parameters.

2022

Insert & Edit released for GPT-3 and Codex. DALL-E2 is released

Iterate over existing text and refactor code via prompt. DALL-E2 has now 3.5B parameters.

NLP Models comparison

GPT-1

BERT

Parameters (Bilions)

2018

2019

2020

2021

GPT-2

RoBERTa

Codex

GPT-3

Switch

175B

1.5B

T5

11B

12B

Year

1.6T

1

10

100

1000

Gopher

280B

GPT-3

GPT-3 in action

The day started as any other day with a couple of hours of intense work before lunch.

The day started as any other day

Prompt

Hyperparameters

Response

.

🐠

Conversational AI

Question: Who discovered America?

Answer: Christopher Columbus.

Question: What year was that?

Answer: 3054.

Question: Who discovered America?

Answer: Christopher Columbus.

Question: What year was that?

Question: Who discovered America?

Answer: Christopher Columbus.

Question:

Question: Who discovered America?

That doesn't look right...

AI generated text may be...

Biased

Non-factual

Inaccurate

1+1= 3

GPT-3 Model Stages

Design

Predictions

Training

GPT-3

GPT-3 Model

Released: June 2020
Common Crawl, WebText2 (Reddit), Books1, Books2 and Wikipedia
175B parameters, 96 layers, 12,288 dimensions
Total 550GB

GPT-3

GPT-3 Training DataSet

499 Billion tokens

+100 Languages

Russian

Romanian

Polish

Danish

Japanese

Swedish

Finish

French

German

Italian

Portuguese

Dutch

English

Spanish

Guten Morgen!

Training phase

Adjust model predictions using output

Wikipedia

Christopher is

Christopher Columbus was

Input

Output

GPT-3

Christopher Columbus discovered America

Christopher Columbus discovered America in

Christopher Columbus discovered America in 1492

Christopher Columbus discovered America in 1492 .

Christopher Columbus discovered America in 1492.

Christopher Columbus discovered America in 1492.

Christopher Columbus discovered America in 1492.

Christopher Columbus discovered America in 1492 .

Hyper-dimensional Graph

who

is

America

Columbus

discover

d_1

d_{2}

d_2

d_{12288}

d_{1}

d_{3}

Coding Tools

// Reverse a string in one line
function reverseString(str) {
 return str.split('').reverse().join('');
}

// Reverse a string in one line

How did this happen!!?

GPT-3 can be used for...

Writing

Tools

Automated Summaries

Translations

Coding

Tools

Conversational

AI

Instructions

Codex

Codex Model

Released: August 2021
GitHub snapshot (May-2020)
54M public repositories
12B parameters, 96 layers, 12,288 dimensions
Total 179GB, 159GB sanitised

Codex Language Support

Perl

Go

Ruby

Python

JavaScript

TypeScript

SQL

PHP

Shell

Rust

Swift

C#

GitHub Copilot

GitHub Copilot Architecture

OpenAI

GitHub

Visual Code

Copilot Service

Codex

Technical Preview

JavaScript

TypeScript

Go

Ruby

Python

Use comments

Or cherry-pick alternatives

Insert & Edit

Refactoring

DALL-E2

Generating images

An astronaut

lounging in a tropical resort in space

in a vaporwave style

An astronaut

lounging in a tropical resort in space

as pixel art

An astronaut

lounging in a tropical resort in space

in a photorealistic style

Prompt

DALL-E2

Editing images

Prompt

DALL-E2

Add flamingo

Original

Codex Experiment

This is only the beginning

The future of AI is today

Try it!

OpenAI Playground

Copy.ai