The Illustrated Transformer
jalammar.github.ioDiscussions:
Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments)
Translations: Arabic, Chinese (Simplified) 1, Chinese (Simplified) 2, French 1, French 2, Italian, Japanese, Korean, Persian, Russian, Spanish 1, Spanish 2, Vietnamese
Watch: MIT’s Deep Learning State of the Art lecture referencing this post
Featured in courses at Stanford, Harvard, MIT, Princeton, CMU and others
Update: This post has now become a book! Check out LLM-book.com which contains (Chapter 3) an updated and expanded version of this post speaking about the latest Transformer models and how they've evolved in the seven years since the original Transformer (like Multi-Query Attention and RoPE Positional embeddings).
In the previous post, we looked at Attention – a ubiquitous method in modern deep learning models. Attention is a concept that helped improve the performance of neural machine translation applications. In this post, we will look at The Transformer – a model that uses attention to boost the speed with which these models can be trained. The Transformer outperforms the Google Neural Machine Translation model in specific tasks. The biggest benefit, however, comes from how The Transformer lends itself to parallelization. It is in fact Google Cloud’s recommendation to use The Transformer as a reference model to use their Cloud TPU offering. So let’s try to break the model apart and look at how it functions.
The Transformer was proposed in the paper Attention is All You Need. A TensorFlow implementation of it is available as a part of the Tensor2Tensor package. Harvard’s NLP group created a guide annotating the paper with PyTorch implementation. In this post, we will attempt to oversimplify things a bit and introduce the concepts one by one to hopefully make it easier to understand to people without in-depth knowledge of the subject matter.
2025 Update: We’ve built a free short course that brings the contents of this post up-to-date with animations:
A High-Level Look
Let’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.
Home - Keep The Future Human
keepthefuturehuman.aiHumanity is on the brink of developing artificial general intelligence that exceeds our own. It's time to close the gates on AGI and superintelligence... before we lose control of our future.
AIs have a favorite number, and it's not 42
theregister.com: Ask a model to guess a number from 1 to 50 and it's likely to answer 27
What Is ChatGPT Doing … and Why Does It Work?
writings.stephenwolfram.comStephen Wolfram explores the broader picture of what's going on inside ChatGPT and why it produces meaningful text. Discusses models, training neural nets, embeddings, tokens, transformers, language syntax.
Man-Computer Symbiosis
groups.csail.mit.edu
Neural networks and deep learning
neuralnetworksanddeeplearning.com
A Recipe for Training Neural Networks
karpathy.github.ioMusings of a Computer Scientist.
The Bitter Lesson
incompleteideas.net
Ten Principles of Good Design | Tony Gaeta
tonygaeta.comDieter Rams' Ten Principles of Good Design.
Why Local-First Apps Haven’t Become Popular?
marcobambini.substack.comOffline-first apps promise instant loading and privacy, but in practice, very few apps get offline support because getting sync right is surprisingly hard.
DORA | Get Better at Getting Better
dora.devDORA is a long running research program that seeks to understand the capabilities that drive software delivery and operations performance. DORA helps teams apply those capabilities, leading to better organizational performance.
Shape Up: Stop Running in Circles and Ship Work that Matters
basecamp.comShape Up will help you break free of “best practices” that aren’t really working, think deeper about the right problems, and start shipping meaningful projects your team can celebrate.
Stirling-PDF: #1 Locally hosted web application that allows you to perform various operations on PDF files
github.com#1 Locally hosted web application that allows you to perform various operations on PDF files - Stirling-Tools/Stirling-PDF
Software Engineering - The Soft Parts
addyosmani.comToday I will share some of the software engineering soft skills I have learned from my first 10 years on Google Chrome, where I am a Senior Staff Engineering...
Gradual Disempowerment
gradual-disempowerment.ai
ChatGPT Is a Blurry JPEG of the Web
newyorker.comOpenAI’s chatbot offers paraphrases, whereas Google offers quotes. Which do we prefer?
The Scaling Hypothesis
gwern.netOn GPT-3: meta-learning, scaling, implications, and deep theory. The scaling hypothesis: neural nets absorb data & compute, generalizing and becoming more Bayesian as problems get harder, manifesting new abilities even at trivial-by-global-standards-scale. The deep learning revolution has begun as foretold.
As We May Think
theatlantic.com“Consider a future device ... in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.”The Illustrated Transformer
jalammar.github.ioDiscussions:
Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments)
Translations: Arabic, Chinese (Simplified) 1, Chinese (Simplified) 2, French 1, French 2, Italian, Japanese, Korean, Persian, Russian, Spanish 1, Spanish 2, Vietnamese
Watch: MIT’s Deep Learning State of the Art lecture referencing this post
Featured in courses at Stanford, Harvard, MIT, Princeton, CMU and others
Update: This post has now become a book! Check out LLM-book.com which contains (Chapter 3) an updated and expanded version of this post speaking about the latest Transformer models and how they've evolved in the seven years since the original Transformer (like Multi-Query Attention and RoPE Positional embeddings).
In the previous post, we looked at Attention – a ubiquitous method in modern deep learning models. Attention is a concept that helped improve the performance of neural machine translation applications. In this post, we will look at The Transformer – a model that uses attention to boost the speed with which these models can be trained. The Transformer outperforms the Google Neural Machine Translation model in specific tasks. The biggest benefit, however, comes from how The Transformer lends itself to parallelization. It is in fact Google Cloud’s recommendation to use The Transformer as a reference model to use their Cloud TPU offering. So let’s try to break the model apart and look at how it functions.
The Transformer was proposed in the paper Attention is All You Need. A TensorFlow implementation of it is available as a part of the Tensor2Tensor package. Harvard’s NLP group created a guide annotating the paper with PyTorch implementation. In this post, we will attempt to oversimplify things a bit and introduce the concepts one by one to hopefully make it easier to understand to people without in-depth knowledge of the subject matter.
2025 Update: We’ve built a free short course that brings the contents of this post up-to-date with animations:
A High-Level Look
Let’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.
The Unreasonable Effectiveness of Recurrent Neural Networks
karpathy.github.ioMusings of a Computer Scientist.
AI Doesn’t Reduce Work—It Intensifies It
simonwillison.netAruna Ranganathan and Xingqi Maggie Ye from Berkeley Haas School of Business report initial findings in the HBR from their April to December 2025 study of 200 employees at a …
power_of_proximity_nov2023.pdf
pallais.scholars.harvard.edu
This is a Motherfucking App
adnjoo.github.io
Does Your Startup Really Need Complex Cloud Infrastructure? - Hadi Javeed's blog
hadijaveed.meNotes about startups, AI, health-care and overall engineering
no hello
nohello.netplease don't say just hello in chat
How to build a startup pitch deck
openvc.appThis playbook is my home-made, battle-tested process to prepare the structure, content, and design of a typical early-stage startup pitch deck. These 2 hours of prep work will save you 10x more time down the road, I promise.
Developer Productivity for Humans, Part 7: Software Quality | IEEE Journals & Magazine | IEEE Xplore
ieeexplore.ieee.orgThat's all for now. Come back later for more.
.jpg)
