GPU Computing Fundamentals

The Big Picture First#

Everything we’ve discussed throughout this entire series so far — RAG, agents, LLMOps, security — has assumed that somewhere, underneath all of it, there’s actual physical computing hardware doing the real, heavy-lifting work of actually running these models. Week 13 finally opens up that hood and looks at what’s actually happening down at the hardware level. Here’s the core fact that explains basically everything in this whole explanation: training and running large language models involves doing an absolutely enormous number of relatively simple mathematical calculations (mostly multiplying and adding numbers together, over and over, at truly massive scale), and it turns out that regular computer processors, despite being genuinely powerful and flexible, are actually not the best-suited tool for this particular specific kind of work. A different kind of processor — the GPU — turns out to be dramatically better suited to it. Understanding why is really the foundation for understanding this entire week’s material.

1. AI Infrastructure Overview#

AI infrastructure refers to the complete stack of physical hardware and supporting software that actually makes it possible to train and run AI models at real, meaningful scale — everything from the actual physical computer chips doing the raw calculations, all the way up through the software layers that let a developer actually write and run their AI code without needing to personally understand every single detail of the underlying hardware.

It helps to think of this as a layered stack, where each layer builds on top of the one below it. At the very bottom sits the actual physical hardware — the specialized processors (which we’ll get into shortly), the physical memory, and the networking that connects many machines together. Above that sits a layer of low-level software specifically designed to let higher-level programs actually communicate with and properly make use of that specialized hardware (we’ll cover several of these pieces later in this explanation — the driver stack, the CUDA toolkit, and so on). And above that sits the higher-level software frameworks (like the popular ones such as PyTorch and TensorFlow, which you may have already come across if you’ve done any hands-on AI development) that most developers actually interact with directly, day to day, without needing to think too hard about most of what’s happening underneath.

Why does this whole layered infrastructure matter so much? Because the specific hardware and software choices made at each one of these layers genuinely have an enormous, direct impact on how fast a model can actually be trained, how much it costs to actually run, and how many users it can genuinely serve at once. A team that deeply understands this whole stack can make genuinely much better decisions about what hardware to actually buy or rent, and how to actually configure it well, compared to a team that treats it all as a complete, mysterious black box they never really think about at all.

2. GPU Architecture#

GPU stands for Graphics Processing Unit, and to understand why it’s become so central to AI, it genuinely helps to understand where it actually originally came from. GPUs were originally built for an entirely different purpose — rendering graphics for video games and other visual applications, where a computer needs to calculate the color and position of literally millions of individual pixels on a screen, many times every single second, to actually produce smooth, moving, visual images.

Here’s the key architectural insight that turned out to matter enormously for AI, well beyond its original graphics purpose: calculating the value of one single pixel is a relatively simple, small, well-defined mathematical task, and, critically, the calculation for one pixel doesn’t generally depend on the calculation for a different, other pixel — meaning millions of these simple calculations can genuinely all be done at the exact same time, completely independently and in parallel, rather than needing to be done strictly one after another. So GPU designers built chips specifically optimized around this exact insight — rather than having just a few, individually very powerful processing units (which is genuinely how a regular computer processor, a CPU, is actually built), a GPU instead contains many thousands of smaller, individually simpler processing units, all working together at the exact same time, in parallel.

It turns out that the core mathematical operations involved in running an AI model — things like multiplying huge grids of numbers together (called matrix multiplication, which is genuinely the fundamental core mathematical operation underlying almost everything a neural network actually does) — have this exact same basic character as rendering graphics pixels: an enormous number of relatively simple calculations that can genuinely all be done at the same time, in parallel, rather than one at a time, in sequence. This is precisely why GPUs, originally built for an entirely different purpose, turned out to be so remarkably, unexpectedly well-suited to AI workloads once people actually started trying to use them that way.

3. CPU vs GPU#

Now that we understand the basic architectural difference, it’s worth directly comparing CPUs and GPUs side by side, because understanding exactly when each one is genuinely the better tool for a given job is a genuinely foundational, important piece of understanding AI infrastructure well.

A CPU (Central Processing Unit) is the general-purpose “brain” of a regular computer, and it’s specifically designed to be highly flexible and very good at handling a wide, genuinely diverse range of different, varied tasks, often one complex task quickly after another, in a fairly sequential order. A CPU typically has a relatively small number of individually very powerful, very sophisticated processing cores (commonly somewhere between around 4 and maybe 64 cores in a typical modern machine), each one capable of handling genuinely complex, varied logic and making rapid, sophisticated decisions.

A GPU, by contrast, as we just discussed, has thousands of individually much simpler processing cores, all specifically optimized for doing the exact same type of relatively simple calculation, over and over, on many different pieces of data all at the same time. A helpful, often-used analogy here: think of a CPU as being like a small team of a few genuinely brilliant, highly skilled individual specialists, each one capable of tackling a wide range of genuinely different, complex problems one after another. Think of a GPU as being like an enormous crowd of thousands of people, each one only capable of doing one simple task (like, say, basic addition), but who can all do that same simple task simultaneously, all at once, together.

For tasks that genuinely require complex, varied, sequential decision-making (like running a regular website’s core application logic, or managing a computer’s own overall operating system), a CPU is genuinely the better, more suitable tool. For tasks that genuinely involve doing the exact same relatively simple type of calculation an enormous number of times, in parallel (like training or running an AI model), a GPU is dramatically, genuinely faster — often by a factor of many multiples, or even many, many times faster, for exactly this specific kind of work.

4. CUDA Fundamentals#

Understanding that GPUs are well-suited to AI workloads is one thing — but a GPU’s specialized hardware, all on its own, doesn’t automatically know how to actually run AI code. CUDA (which stands for Compute Unified Device Architecture) is a platform and programming approach, created by NVIDIA (currently the dominant maker of GPUs specifically used for AI), that specifically lets developers actually write code that can properly run on and make full use of a GPU’s own particular, specialized parallel architecture.

Here’s the genuine problem CUDA actually solves. Writing code that correctly, efficiently splits a given task up across thousands of individual, simple parallel processing cores, and then correctly coordinates and combines all their many individual results back together again afterward, is genuinely a meaningfully different, more complex kind of programming challenge than writing regular, ordinary, sequential code intended for a genuinely conventional CPU. CUDA gives developers a genuinely structured, well-documented way of actually writing this kind of specifically parallel code, without needing to understand every single last low-level detail of the GPU’s own actual physical hardware.

CUDA has become genuinely enormously significant and important throughout the whole AI industry, well beyond just being one single technical tool among several roughly interchangeable options, precisely because of how broadly and how deeply it’s actually become adopted and embedded — most of the major AI frameworks (like PyTorch, which we mentioned earlier) are genuinely built with deep, extensive CUDA support, and a truly massive amount of the world’s existing AI code and existing AI infrastructure genuinely, specifically assumes CUDA is actually available and genuinely being properly used. This is a real, significant part of why NVIDIA’s own specific GPUs have become so genuinely dominant throughout the AI industry — it’s not merely that their actual physical hardware happens to be particularly good (though it genuinely is), it’s that CUDA, as a whole broader software platform and surrounding ecosystem, has become so genuinely deeply, widely embedded and integrated throughout the entire industry’s software.

5. CUDA Cores#

CUDA cores are the actual, individual, specific processing units that physically sit inside an NVIDIA GPU — they’re genuinely the actual physical hardware components that CUDA, as the broader software platform we just discussed, is specifically designed to actually let a developer’s code make proper, efficient use of.

Going back to our earlier CPU-versus-GPU comparison, CUDA cores are genuinely the specific, concrete embodiment of that “thousands of simple processing units” architectural idea we discussed earlier. A single modern, high-end NVIDIA GPU can genuinely contain many thousands of individual CUDA cores — sometimes even reaching into the range of several thousand to well over ten thousand individual cores, depending on the exact specific model and generation of the actual chip in question. Each single individual CUDA core, entirely on its own, is genuinely relatively simple and not all that powerful, but having thousands of them all genuinely, actually working together simultaneously, in parallel, is precisely what actually gives a GPU its whole overall enormous total computational power for exactly the kinds of parallel workloads we’ve been discussing throughout this whole explanation.

It’s genuinely worth understanding that the raw, total number of CUDA cores a given GPU has isn’t actually the complete, entire full picture of its overall real performance, all on its own — the actual speed and efficiency of each individual core, how quickly the whole chip can actually access its own memory (which we’ll cover as its own separate, distinct topic shortly below), and other, additional specialized hardware features (like the tensor cores we’ll also specifically cover shortly) all genuinely, meaningfully matter too. But the total raw number of CUDA cores remains a genuinely useful, common, general, rough indicator of a given GPU’s own overall raw parallel processing capability, and it’s a genuinely commonly cited, referenced specification whenever people are actually comparing different specific GPU models against one another.

6. GPU Memory#

GPU memory (often referred to specifically as VRAM, for “video RAM,” reflecting graphics cards’ own original historical purpose) refers to the actual, dedicated physical memory that sits directly on a given GPU itself, used specifically for storing the actual data the GPU is actually, currently working with — as genuinely distinct and separate from a regular computer’s own main, separate system memory (RAM), which a CPU actually, instead, primarily uses for its own separate work.

Understanding GPU memory genuinely matters an enormous amount in practice, for a fairly simple, practical, concrete reason: an AI model, along with all of the actual data it’s genuinely currently working with (like a batch of text currently being processed), needs to actually genuinely fit within a given GPU’s own available memory capacity in order for that GPU to actually be able to properly work with and process it at all. Modern large language models can genuinely be enormous — some of the largest ones out there require many tens, or even hundreds, of gigabytes of memory space just to actually hold the model’s own learned parameters alone, well before you even get to accounting for any of the actual additional working data needed for actually running it. If a given model genuinely doesn’t actually fit within a single GPU’s own available memory, you either genuinely need a GPU that actually has considerably more memory capacity available, or you genuinely need to actually split the given model up cleverly across multiple different separate GPUs working together (a genuinely real, meaningful technical challenge in its own right, connecting quite directly to the NCCL topic we’ll get to shortly below).

GPU memory capacity has actually become one of the single most significant, important, and genuinely often limiting practical factors in modern AI infrastructure, precisely because of how large today’s models have genuinely grown to actually become. A GPU with genuinely excellent raw processing speed, but genuinely insufficient available memory capacity, simply, genuinely can’t actually run a given particular large model at all — no matter how fast that GPU’s own individual cores might otherwise genuinely, individually be. This is exactly why, when people actually compare and choose between different specific GPU models for genuine, serious AI work, memory capacity is very often one of the single most heavily weighted, important considered factors, right alongside raw computational processing speed itself.

7. Tensor Cores#

Tensor cores are a further, more specialized type of processing unit found specifically on newer NVIDIA GPUs, genuinely distinct and separate from the more general-purpose CUDA cores we already discussed above — and they’re specifically, deliberately built to accelerate one particular, very specific type of mathematical operation that happens to be especially, genuinely central and important to how AI models actually work.

That specific particular operation is matrix multiplication — which we already actually briefly mentioned earlier, back in the GPU Architecture section, as being genuinely the fundamental, core mathematical operation underlying almost everything a neural network actually, fundamentally does. While regular CUDA cores are already genuinely quite good at handling this particular kind of calculation, tensor cores are even further, additionally specialized specifically and exclusively for it — they’re genuinely, specifically built to actually perform certain particular specific kinds of matrix math considerably faster than even regular CUDA cores alone genuinely, actually could.

This specific additional specialization genuinely, actually matters an enormous amount for real-world, practical AI performance. NVIDIA has reported that tensor cores can genuinely provide dramatically faster performance, for exactly the specific kinds of calculations that AI models actually specifically, heavily rely on, compared to using regular CUDA cores alone, entirely on their own, for that exact same specific kind of underlying work. This is a genuinely excellent, clear, concrete illustration of a fairly broader, more general important principle in modern computer hardware design: rather than only ever building genuinely more and more general-purpose processing power, chip designers increasingly, deliberately build specific, dedicated hardware specifically optimized for particular, especially common and especially important specific types of calculations — and given precisely how absolutely central matrix multiplication genuinely, specifically is to essentially all of modern AI, tensor cores genuinely represent a particularly clear, well-motivated, sensible example of exactly this same broader specialization principle actually, concretely playing out in real, practical, real-world hardware.

8. NVIDIA Driver Stack#

A driver, as a completely general software concept, is the specific piece of software that actually lets a computer’s own operating system properly communicate with and correctly control a given specific piece of physical hardware. The NVIDIA driver stack refers specifically to the actual collection of software layers that genuinely, actually let a computer properly recognize and correctly, properly make use of an installed NVIDIA GPU in the first place.

Without a genuinely properly installed and correctly, properly functioning driver, an operating system genuinely, actually has no reliable, correct way of actually knowing how to properly communicate with a given specific physical GPU at all — the actual physical hardware chip might genuinely be correctly, physically installed and present inside a given machine, but without the correct, matching driver software genuinely also properly installed alongside it, the rest of the overall computer system genuinely, actually can’t correctly, actually make any real, meaningful use of it whatsoever. The driver is what actually genuinely handles the lowest-level, most fundamental communication with the actual physical hardware itself — things like actually correctly sending genuine, real computational work to the GPU to be properly processed, and then actually correctly, properly retrieving the genuine, real resulting results back once that work has actually, genuinely finished being properly completed.

For anyone genuinely, actually setting up AI infrastructure themselves in practice, getting the correct, matching driver version genuinely, properly installed correctly is often one of the single most common, genuinely frustrating early practical setup and configuration challenges people actually, genuinely run into. Different specific driver versions genuinely need to actually, properly match up correctly with the other various different specific software layers sitting above them in this whole stack (like the CUDA toolkit, which we’ll actually cover next, right below) — a genuine, real mismatch between these different specific software layers’ own respective, particular versions is a genuinely extremely common, well-known, and often quite frustrating source of real technical problems specifically when actually, practically trying to get a given GPU-based AI system properly, correctly, and reliably up and running in the first place.

9. CUDA Toolkit#

The CUDA toolkit is the genuinely complete collection of actual software tools, code libraries, and documentation that NVIDIA actually, genuinely provides specifically to let developers actually write, properly build, and correctly run genuine CUDA-based code — it sits as a genuinely distinct software layer, one level up above the actual raw driver we just discussed, specifically providing the higher-level tools a developer actually, genuinely needs to properly do real, actual, hands-on AI development work.

Where the driver, as we just discussed, specifically handles the most fundamental, lowest-level communication between the operating system and the actual physical hardware chip itself, the CUDA toolkit specifically provides the genuinely higher-level tools a developer actually, genuinely uses day to day — things like a genuine compiler (which is specifically the tool that actually translates a developer’s own human-written code into the actual specific low-level instructions a GPU can genuinely, actually understand and correctly execute), and a whole range of genuine, various pre-built code libraries that already, genuinely handle a lot of the common, genuinely repeated, everyday underlying computational work, so that individual developers genuinely, actually don’t each need to separately write all of that same underlying complex code entirely themselves, completely from total scratch, every single time.

For most everyday developers who are actually just using a genuinely higher-level framework like PyTorch (which we already briefly mentioned earlier, back at the very start of this whole explanation), a lot of the actual CUDA toolkit’s own underlying complexity genuinely, actually stays fairly comfortably hidden and abstracted away underneath — a given framework like PyTorch itself already, genuinely handles most of the actual direct interaction with CUDA on a developer’s own behalf, so a typical developer genuinely doesn’t usually, actually need to write raw, direct CUDA code themselves, entirely by hand. But having the genuinely correct, properly matching version of the CUDA toolkit actually, properly installed underneath everything else remains genuinely essential regardless, since these genuinely higher-level frameworks are themselves actually, ultimately built directly right on top of it.

10. cuDNN#

cuDNN (which specifically stands for “CUDA Deep Neural Network library”) is yet a further, additional specialized software library, built specifically, deliberately on top of the more general CUDA toolkit we just discussed, and it’s specifically, deliberately optimized for the particular, specific kinds of calculations that genuinely, specifically show up repeatedly throughout deep learning and neural networks in particular — rather than the more broadly general, wider range of parallel computing tasks that the more general CUDA toolkit itself is actually, more broadly designed to genuinely support.

Think of the relationship here as being somewhat similar in overall spirit to the relationship between CUDA cores and tensor cores that we already discussed earlier — CUDA, as the more general underlying toolkit, genuinely already lets you actually do general-purpose parallel computing work of many different, varied kinds, while cuDNN specifically, further, additionally optimizes for the particular, specific operations that genuinely, specifically come up over and over again in deep learning work specifically — things like the particular specific mathematical operations genuinely involved in a neural network’s own internal individual layers.

The genuine practical value cuDNN actually, genuinely provides is real, meaningful additional speed, specifically for exactly these particular deep-learning-specific operations, well beyond what you’d actually get from simply using the more general CUDA toolkit alone, entirely on its own, without cuDNN’s own further, additional specific optimizations layered properly on top of it. Just like with the CUDA toolkit itself, most everyday individual developers genuinely, actually don’t interact with cuDNN directly themselves, by hand — it instead, genuinely operates quietly underneath the higher-level frameworks like PyTorch that a typical developer actually, genuinely, directly works with day to day — but its own specific presence, and having the genuinely correct, properly matching version actually, properly installed underneath everything else, meaningfully, genuinely affects how fast a given model actually, genuinely trains and actually genuinely runs in real, practical, everyday practice.

11. NCCL#

NCCL (pronounced “nickel,” and specifically standing for “NVIDIA Collective Communications Library”) specifically addresses a genuinely real, important, practical problem that naturally, genuinely comes up once you actually need to use multiple separate GPUs together, working properly, closely, and reliably in real coordination with one another — which, given how enormous today’s largest AI models have genuinely, actually become (connecting directly back to the GPU memory discussion earlier above), is genuinely an extremely common, everyday, and genuinely important real-world practical scenario.

When you’re actually, genuinely training or running a given AI model across several separate GPUs at once, working properly together, those different individual GPUs genuinely, actually need to be able to efficiently, reliably communicate with each other — for example, actually, genuinely sharing and properly, correctly combining their own individual, separate partial results together, so that the whole overall combined system can actually, genuinely arrive at one single, unified, correct, final overall answer. NCCL is specifically, deliberately designed to actually, genuinely handle exactly this particular kind of communication as efficiently and as reliably as reasonably, genuinely possible — providing developers with a set of well-designed, pre-built, ready-to-use tools specifically for actually, reliably coordinating and properly sharing data across multiple different GPUs, without each individual developer separately, individually needing to write all of that same underlying complex multi-GPU coordination logic entirely by hand themselves, completely from total scratch, every single time.

This particular capability genuinely, actually becomes increasingly critical and important the larger a given specific AI workload actually, genuinely gets. Training one of today’s genuinely largest modern language models often actually genuinely requires many hundreds, or sometimes literally even many thousands, of individual GPUs, all genuinely working together properly, closely, and reliably in real, actual coordination — and the genuine overall efficiency of exactly how well those many separate individual GPUs can actually, genuinely communicate with each other becomes a genuinely real, significant, and often quite substantial factor in determining the whole overall system’s own genuine total real-world practical performance, right alongside the genuine raw individual computational power of each individual GPU considered entirely separately, all on its own.

12. GPU Scheduling#

GPU scheduling refers to the genuinely real, important, practical process of actually deciding which specific task or which specific job genuinely, actually gets to actually use a given available GPU, and genuinely, actually exactly when — this genuinely becomes an especially real, important, and often quite significant practical concern specifically in any kind of shared environment, where multiple genuinely different users, or multiple genuinely different teams, or multiple genuinely different separate applications, all genuinely need to actually, meaningfully share access to some kind of overall limited, shared pool of available GPU resources together.

Here’s the genuine, real underlying practical problem GPU scheduling actually, genuinely needs to properly solve. GPUs, particularly genuinely powerful, high-end ones specifically well-suited to serious, real AI work, remain genuinely quite expensive pieces of specialized physical hardware, and organizations genuinely, typically, sensibly don’t actually want to simply, wastefully buy an entirely completely separate, dedicated GPU for literally every single individual team or every single individual specific project they might genuinely happen to have running internally. Instead, a genuinely shared pool of available GPUs is typically actually, sensibly maintained together, and a genuinely dedicated scheduling system specifically, actively decides which particular specific job actually, genuinely gets to actually use which particular specific available GPU, and for genuinely exactly how long, in a genuinely fair, efficient, and sensible overall way.

Good GPU scheduling genuinely, actually needs to properly, sensibly balance several genuinely real, distinct, competing considerations together at once. It genuinely, actually needs to try to keep the whole overall available pool of GPUs genuinely, actually as busy and as fully, efficiently utilized as reasonably possible (since an idle, unused, genuinely expensive GPU sitting there completely unused represents real, genuine, unnecessarily wasted money, sitting there entirely idle). It genuinely, actually needs to try to be reasonably, appropriately fair across the whole organization’s own various different competing users and teams, rather than genuinely letting just any one single particular team or user consistently, unfairly monopolize the entire whole shared available pool of resources all to themselves alone. And it genuinely, actually needs to properly, sensibly handle jobs of genuinely different, varying relative priority appropriately — for example, appropriately letting a genuinely urgent, time-sensitive production request reasonably, sensibly take real priority over a genuinely much less time-sensitive, more experimental research job that could, quite reasonably, genuinely just as easily wait its own proper turn a while longer instead. This connects quite directly and closely back to the rate limiting and cost allocation concepts we already, previously covered back in the earlier Week 11 enterprise architecture material — GPU scheduling is really, genuinely the same broader, fundamental underlying idea of fairly, sensibly, and efficiently sharing a genuinely limited, valuable, shared resource across many different competing internal consumers, just now specifically, particularly applied here directly to actual, real physical computing hardware itself, rather than to the more purely software-based API rate limits we had already originally, specifically discussed back in that earlier explanation.