Week 14 - Fine-Tuning & LLM Customization on AI Platform Engineering Handbook

Fine-Tuning Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

Fine-Tuning Fundamentals

The Big Picture First

Throughout almost this entire series, we’ve treated the underlying language model itself as a fixed, unchanging thing — something you connect to through an API, prompt cleverly, and build tools and RAG systems around, but never actually modify directly. Fine-tuning is about something genuinely different: actually changing the model itself, adjusting its own internal learned parameters, so that it behaves differently going forward, for every single future conversation, without needing to be specifically re-instructed each time through a prompt. Here’s the useful way to think about the relationship between prompting and fine-tuning: prompting is like giving someone very detailed, careful instructions right before they start a specific task — genuinely helpful, but it has to be repeated every single time. Fine-tuning is more like actually training and coaching that same person over time, so the new behavior becomes a genuinely natural, built-in part of how they approach things going forward, without needing to be told all over again each time. Let’s build up an understanding of how this actually works.

Human Feedback & Alignment

Mon, 01 Jan 0001 00:00:00 +0000

Human Feedback & Alignment

The Big Picture First

Everything we covered in the previous two Week 14 explanations focused on teaching a model new knowledge, new formats, or new domain-specific skills. This final section addresses a genuinely different, and arguably even more fundamental question: how do you actually teach a model to behave the way humans genuinely want it to behave, in the first place — to be genuinely helpful, honest, and safe, rather than simply being good at predicting statistically plausible-sounding text? This is precisely the gap that instruction tuning, which we touched on earlier, only partially closes. A model can be instruction-tuned to follow the general format of instructions, while still genuinely producing responses that are subtly unhelpful, misleading, or occasionally genuinely harmful. Alignment techniques — RLHF, DPO, and everything else covered in this section — are specifically about closing that remaining gap, using genuine human judgment about what “good” actually looks like, to further shape a model’s behavior in that particular direction. This is, in a very real sense, part of the same broader process that actually shaped how I, Claude, myself actually behave.

Parameter-Efficient Fine-Tuning (PEFT)

Mon, 01 Jan 0001 00:00:00 +0000

Parameter-Efficient Fine-Tuning (PEFT)

The Big Picture First

At the very end of the previous explanation, we touched briefly on LoRA and parameter-efficient fine-tuning as a way to make fine-tuning more affordable. This whole section is dedicated entirely to unpacking that idea properly, because it turns out to be one of the single most practically important developments in making fine-tuning genuinely accessible to regular developers and smaller teams, rather than remaining something only the very largest, best-resourced companies could ever realistically afford to do. Here’s the core problem PEFT solves: a modern large language model can have many billions of individual internal parameters, and traditional “full fine-tuning” means updating every single one of them — which requires an enormous amount of GPU memory, not just to hold the model itself, but to hold all the additional bookkeeping information needed to actually calculate how each of those billions of parameters should change. PEFT is a whole family of clever techniques that dramatically reduce this requirement, making fine-tuning something that can genuinely be done on much more modest, affordable hardware. Let’s work through exactly how.