Denoising the hype
Allen J. Romano
Denoising the Noise: AGI, GAI, LLMs, and ChatGPTs
allenjromano@gmail.com
(March 6, 2023)
Some Noise
Oct. 2022
March 2023
Why this?
Why now?
— ChatGPT has captured the hype cycle for three reasons, none of which is about new technology ...
Text
Web3 / Crypto hype cycle according to Gartner.com
ChatGPT == best product demo ever
〞
Sandhini Agarwal (AI Policy at OpenAI): I think it was definitely a surprise for all of us how much people began using it. We work on these models so much, we forget how surprising they can be for the outside world sometimes.
https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/
?
Underlying Technology
Tech
Tools and Such
Apps
Application Building
Build
Non-revolutionary Technology
— It just isn't all that novel
〞
In terms of underlying techniques, ChatGPT is not particularly innovative. It’s nothing revolutionary, although that’s the way it’s perceived in the public. It’s just that, you know, it’s well put together, it’s nicely done.”
– Yann LeCun, Meta AI
https://www.zdnet.com/article/chatgpt-is-not-particularly-innovative-and-nothing-revolutionary-says-metas-chief-ai-scientist/
from "Opportunities and Risks of Foundation Models" https://arxiv.org/pdf/2108.07258.pdf
from "Opportunities and Risks of Foundation Models" https://arxiv.org/pdf/2108.07258.pdf
Brute Force
- More data, more parameters
- Better data
- More languages
- Multimodal (Image <-> Text <-> Audio <-> Video <-> ??Thought??)
- Models check themselves
Finesse
- New architectures
- Sparse Models (all current production models are "dense" models)
Immediate Horizons
from Deepmind ai, "Multimodal Few-Shot Learning with Frozen Language Models": https://arxiv.org/abs/2106.13884
Minimal Multimodal model (for illustration)