一個小故事...

我

悲慘的我

欲哭有淚的我

?

或許也有人跟我一樣？

• 不想背著厚厚的解答

• 詳解翻好久，一題一題對都找不到

• 解答真的是對的嗎，它到底在說甚麼？

• ......

學霸之路

到底該怎麼做呢？

LSTM

RNN

• 只有單向記憶
• 記憶力不足
• 正確性太低

到底該怎麼做呢？

到底該怎麼做呢？

註：雖然也有雙向LSTM，但是Vanishing還是問題

Attention is All You Need!

Vaswani et. al, 2017

ELMo Model

GPT Model

ELMo & GPT Models

ELMo Model

GPT Model

• 用非常大型的模型、非常多的資料來實現多重用途的NLP模型
   
• Context-Based Word Encodings

ELMo & GPT Models

GPT

ELMo

Context-Based 

Bi-Directional

Deeply Context-Based

Say Hello to BERT!

Devlin et. al, 2018

BERT

Bidirectional （雙向）

BERT

Bidirectional （雙向）

Encoder （編碼）

BERT

Bidirectional （雙向）

Encoder （編碼）

Representations（表示）

BERT

Bidirectional （雙向）

Encoder （編碼）

Representations（表示）

Transformers（表示）

完美解決問題！

為什麼要用BERT ?

因為去年BERT學長得冠軍

...

克漏字的最佳模型

雙向性

準確性

易用性

例子

例子

例子

What we had done ?

I

取得題目

110學測 英文試題

以近幾年的學測指考英文以及模擬題本＆參考書作為資料來源

II

辨識題目

利用 Google Tesseract 對圖片進行文字辨識

1. Tom is really a naughty boy. He likes to and play jokes on his younger sister when their parents are not around.

(A) alert (B) spare (C) tease (D) oppose

preprocessing

文字處理，提取需要的題目內容

III

利用Bert模型完成克漏字

對句子、選項進行預處理

pretrained 過的 Bert 模型

得到最佳的克漏字結果

碰到的小問題

Tom is really a naughty boy. He likes _____ to and play jokes on his younger sister when their parents are not around.

Tom is really a _____ naughty boy. He likes to and play jokes on his younger sister when their parents are not around.

Tom is really a naughty boy. He likes to and play _____ jokes on his younger sister when their parents are not around.

Tom is really a naughty boy. He likes to and play _____ jokes on his younger sister when their parents are not around.

Solution

暴力嘗試每種可能 找最佳的匹配結果

Tom is really a naughty boy. He likes to tease and play jokes on his younger sister when their parents are not around.

IV

使用者介面

在本地server架了一個 discord 聊天機器人

作為 Clozer 與使用者互動的窗口

The quick brown fox _____ over the lazy dog.

The quick brown fox jumps over the lazy dog.

Future Prospects

Roberta - 一種基於 Bert 的 pretrained model

嘗試解決篇章結構、閱讀測驗等題目

開發一個完整的輔助學習app