📘 Parallelism and Modern Hardware

Single-core speed has grown slowly since the mid-2000s. Almost all modern performance gains come from doing more things at once: more cores, wider vector units, GPUs, and specialized accelerators. This page explains those forms of parallelism and what they mean for the code you write.

Table of Contents

1. Concurrency vs Parallelism
2. Flynn's Taxonomy
3. Levels of Parallelism
4. Multi-Core and SMT
5. SIMD and Vectorization
6. GPUs
7. AI Accelerators
8. NUMA and Multi-Socket Systems
9. Heterogeneous Chips, Chiplets, and Power
10. Buses and I/O
11. A Performance Mindset
12. Questions

1. Concurrency vs Parallelism

Rob Pike's phrasing: concurrency is about structure, parallelism is about execution. A concurrent program may or may not run in parallel.

2. Flynn's Taxonomy

GPUs are often described as SIMT (single instruction, multiple threads): many threads run the same instruction in lockstep groups.

3. Levels of Parallelism

4. Multi-Core and SMT

A multi-core processor has several complete cores on one chip, each with private L1 and L2 caches and a shared L3. Server CPUs now reach 128 to 192 cores per socket (AMD EPYC, Intel Xeon 6, ARM designs like AWS Graviton4 with 96 cores).

Simultaneous multithreading (SMT), marketed as Hyper-Threading, runs two hardware threads per core. They share execution units and caches, filling slots one thread leaves idle (for example during a cache miss).

Scaling limits on multi-core:

• Amdahl's law: the serial part caps speedup.
• Contention: locks, atomic counters, and shared cache lines serialize work (see False Sharing).
• Memory bandwidth: all cores share the memory channels.
• Overheads: thread creation, synchronization, and load imbalance.

5. SIMD and Vectorization

SIMD instructions apply one operation to a vector of values in a wide register.

// Scalar: one add per iteration
for (int i = 0; i < n; i++) c[i] = a[i] + b[i];

// With AVX2 the compiler (or you, with intrinsics) processes 8 floats at a time
for (int i = 0; i < n; i += 8) {
    __m256 va = _mm256_loadu_ps(&a[i]);
    __m256 vb = _mm256_loadu_ps(&b[i]);
    _mm256_storeu_ps(&c[i], _mm256_add_ps(va, vb));
}

Ways to get SIMD:

1. Auto-vectorization: compilers vectorize simple loops at -O2 / -O3; check with -fopt-info-vec (GCC) or -Rpass=loop-vectorize (Clang).
2. Libraries: NumPy, BLAS, simdjson, Arrow compute kernels, and database engines (DuckDB, ClickHouse) already use SIMD.
3. Portable APIs: Java Vector API (incubating), C# Vector<T>, Rust std::simd (nightly), Highway (C++).
4. Intrinsics or assembly: maximum control, least portable.

What blocks vectorization: branches inside the loop, pointer aliasing (fix with restrict), non-contiguous access, and loop-carried dependencies. Data layout matters: structure-of-arrays vectorizes far better than array-of-structures.

6. GPUs

A GPU trades single-thread speed for massive throughput: thousands of simple cores, very high memory bandwidth, and hardware thread scheduling that hides memory latency by switching between many threads.

CUDA execution model (NVIDIA; AMD ROCm/HIP is similar)

__global__ void add(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}
// launch: add<<<(n + 255) / 256, 256>>>(a, b, c, n);

GPU performance pitfalls:

GPU programming options, from high to low level: PyTorch, JAX, RAPIDS, then Triton, then CUDA C++ / HIP, plus cross-vendor Vulkan compute, Metal on Apple, and WebGPU in browsers.

7. AI Accelerators

Why specialized hardware wins for AI: neural networks are dominated by matrix multiplication in low precision, which is highly parallel and predictable. Large model inference is often memory-bandwidth bound (every token reads all the weights), which is why HBM bandwidth and quantization (8-bit and 4-bit weights) matter so much.

8. NUMA and Multi-Socket Systems

In a NUMA (non-uniform memory access) system, each CPU socket (or chiplet group) has its own local memory. Accessing another socket's memory crosses an interconnect and is slower.

9. Heterogeneous Chips, Chiplets, and Power

Power and heat now limit performance more than transistor count. Dynamic power is roughly proportional to capacitance x voltage^2 x frequency, so higher clocks need higher voltage and power grows much faster than speed. Chips boost when cool and throttle when hot, which is why sustained benchmarks differ from short bursts, and why laptops slow down under long loads.

10. Buses and I/O

I/O devices talk to the CPU via memory-mapped I/O (device registers appear at memory addresses), interrupts, and DMA (the device writes directly to RAM). See OS Fundamentals for how the kernel handles them.

11. A Performance Mindset

1. Measure first. Profile with perf, flame graphs, async-profiler, pprof, or Instruments; guesses about bottlenecks are usually wrong.
2. Find the bound. Is the program limited by compute, memory bandwidth, memory latency, I/O, or locks?
3. Fix the algorithm and data layout before micro-optimizing. An O(n log n) algorithm with good locality beats a tuned O(n^2) one.
4. Parallelize the right amount. Match threads to cores for CPU work; use async or many cheap threads for I/O.
5. Reduce data movement. Moving data costs more energy and time than computing on it.

Roofline model: a kernel's attainable performance is the lower of the chip's peak compute and (memory bandwidth x arithmetic intensity), where arithmetic intensity is operations per byte moved. Low-intensity code (adding two arrays: 1 operation per 12 bytes) is memory bound no matter how many cores you add; high-intensity code (large matrix multiplication) can reach peak compute.

12. Questions

Next: Computer Fundamentals Interview Playbook.