LearnViz CompSci
Home / Learn / Concurrency & Parallelism / GPU Memory Hierarchy

Advanced Topic 38 of 51 Concurrency & Parallelism

GPU Memory Hierarchy

Global, shared and register memory on GPUs.

How it works

GPUs have a multi-level memory hierarchy with different sizes, latencies and scopes. Registers are fastest and private to each thread; shared memory is a small, fast, programmer-managed scratchpad shared by threads of a block; L1 and L2 caches reduce traffic; global memory, the large off-chip DRAM or HBM, has high bandwidth but hundreds of cycles of latency; constant and texture memory offer cached read-only access; and local memory spills registers to global memory. Accesses by a warp coalesce into few transactions when addresses are contiguous.

Where it is used

Optimizing CUDA and OpenCL kernels, such as tiled matrix multiplication that stages data in shared memory, depends on exploiting this hierarchy.

Common pitfall

Uncoalesced or strided access wastes bandwidth, shared-memory bank conflicts serialize accesses and register pressure reduces occupancy or causes spills. Data transfers between host and device are much slower than on-device memory. Forgetting synchronization when threads share data in shared memory causes races. Profile memory throughput and restructure data layouts, such as structure of arrays.

Where it fits in the course

GPU Memory Hierarchy is topic 38 of 51 in Concurrency & Parallelism, rated Advanced. It follows CUDA Programming. The next topic is Parallel Sorting.

Practice and explore

How to study this topic

  1. Read the explanation above and restate the definition in your own words.
  2. Open the interactive lesson and answer its mini quiz.
  3. Read the common pitfall, then write a short example that avoids it.
  4. Check your understanding in the Concurrency & Parallelism quiz, and revisit the topic if you miss a question.

Frequently asked questions

What is GPU Memory Hierarchy?
Global, shared and register memory on GPUs. It belongs to Concurrency & Parallelism and is an advanced-level topic.
Where do people use GPU Memory Hierarchy?
Optimizing CUDA and OpenCL kernels, such as tiled matrix multiplication that stages data in shared memory, depends on exploiting this hierarchy.
What is the most common mistake with GPU Memory Hierarchy?
Uncoalesced or strided access wastes bandwidth, shared-memory bank conflicts serialize accesses and register pressure reduces occupancy or causes spills.
What should I learn before this topic?
Start with CUDA Programming, the topic just before it in Concurrency & Parallelism.
How can I practice it?
Answer the mini quiz in the interactive lesson, then take the Concurrency & Parallelism quiz.
Is GPU Memory Hierarchy hard to learn?
It is an Advanced topic. Expect to read the explanation more than once and to connect it with earlier topics. Working through a small example by hand helps most.
How long does it take to learn GPU Memory Hierarchy?
Most learners can read the explanation and finish the interactive lesson in under an hour. Fluency comes from applying the idea, so plan a few short practice sessions with the quiz.
Which topics are related to GPU Memory Hierarchy?
CUDA Programming, Parallel Sorting, MPI are closely connected. The full list is in the related topics section of this page.
What should I study after GPU Memory Hierarchy?
Continue with Parallel Sorting, the next topic in Concurrency & Parallelism.

Related topics