Stanford CS224N NLP with Deep Learning | 2023 | Lecture 9 - Pretraining

Pre-training revolutionizes modern natural language processing by shifting from task-specific training to large-scale, unsupervised reconstruction of input sequences. By masking tokens or spans within text, models learn complex linguistic structures, semantics, and world knowledge, which serve as robust parameter initializations for downstream tasks. Subword modeling effectively addresses vocabulary limitations, while architectures like BERT, GPT, and T5 demonstrate that pre-training on massive, diverse datasets yields superior performance compared to training from scratch. Recent advancements highlight emergent capabilities in large decoders, such as in-context learning and chain-of-thought reasoning, which allow models to solve complex problems without explicit fine-tuning. This paradigm significantly reduces the need for labeled data, though it necessitates careful consideration of model scale, computational efficiency, and the inherent biases present in large-scale training corpora.