Natural Language Processing

Text

Tokens

Input Ids

tokenizer.tokenize

tokenizer.convert_tokens_to_ids

tokenizer.encode

Var1

Var2

Var3

Var4

I

Walked

My

Dog