The Fundamental Architecture of LLMs: A Perspective Through Information Theory and Lossy Compression
LLMs are sophisticated lossy compression algorithms. Understanding them through information theory explains hallucinations as compression artifacts and reveals how to optimize them through data engineering techniques like RAG and fine-tuning.