Maia 300 Is a Memory Machine Wearing a Compute Chip's Clothes

01The problem Maia 300 is built to solve

When a language model generates text, it runs in two phases. Prefill reads your whole prompt at once, which keeps arithmetic units busy. Decode then produces one token at a time, and for each step the chip has to stream the model weights and the growing KV cache (the stored keys and values for every earlier token) out of memory. During decode, the arithmetic units spend much of their time waiting for data.

Microsoft says this plainly in its announcement: "Moving data efficiently, rather than raw compute power, is increasingly the bottleneck in large inference systems" (Official Microsoft Blog[2]). The Azure architecture post frames agentic workloads, meaning multi-step execution, long-context decoding, and tool use, as changing "the computational profile and arithmetic intensity of inference" (Azure Blog[1]).

The spec sheet confirms the bet. Compared with Maia 200, peak FP4 compute rises from 10.1 to 13.8 petaops, about 37 percent. HBM bandwidth rises from 7.1 to 18 TB/s, about 2.5 times (StorageReview[3]). Microsoft's own post puts the compute gain at 36 percent. Either way, this generation spent its budget on feeding the math, not on more math.

02Mechanism 1: software decides where every byte goes

Maia's core idea predates this chip. Maia 200 introduced what Microsoft calls a Software Defined Locally Accessed Dataflow Architecture, or SDLA. It "separates control path programming from data path execution and allows software to explicitly coordinate data placement, movement, conversion and synchronization across specialized local memories, DMA engines, compute engines, and the network" (Azure Blog[1]).

Compare that with a conventional GPU, where a large share of data movement is handled by hardware caches that guess what you will need next. On Maia, the compiler and kernels say explicitly what moves where and when. That costs programming effort, and it is only practical because Microsoft controls the workload end to end. The payoff is that far less energy goes to fetching data that was never needed.

The compute side is organized as four clusters of 12 tiles. Each tile has a tensor unit, a vector processor, a reshaper, a DMA engine, and local SRAM. Each cluster shares a larger SRAM, and a network-on-chip ties the clusters together (StorageReview[3]). Kernels can use on-die SRAM as a software-managed scratchpad to keep traffic off HBM.

03Mechanism 2: chiplets and NUMA

Maia 200 was a single monolithic die on TSMC N3P with six HBM3E stacks. Maia 300 splits the package into one main compute die on TSMC N2P, two I/O dies, and four memory dies, with eight HBM4 stacks (StorageReview[3]).

The logic is economic. Leading-edge silicon is expensive, and memory controllers and I/O circuits do not benefit much from it. Moving those functions off the compute die "increases the silicon area dedicated to compute performance and allows each function to use the technology best suited to its role" (Azure Blog[1]).

Splitting the chip has a side effect: not all memory is equally close to all compute anymore. Microsoft leans into this instead of hiding it. Maia 300 moves from uniform memory access (UMA), where every memory address costs the same to reach, to a non-uniform (NUMA) layout where "each compute region has nearby memory" and SDLA "places data next to the compute that uses it. Data travels shorter distances, which reduces the energy spent moving it" (Azure Blog[1]). On a hardware-cached design, NUMA is a performance trap. On a software-placed design, it is an optimization target.

This is a chip sized by someone who has measured their own decode traffic.

04Mechanism 3: fewer bits per number

The third lever is precision. Maia 300 introduces "enhanced MXFP4 and sub-3-bit storage formats," plus hardware-accelerated data type conversion and vector-engine instructions for softmax and low-precision casting (Azure Blog[1]).

MXFP4 is an Open Compute Project standard. In it, every block of 32 values shares one 8-bit scale (a power of two, in a format called E8M0), and each value is a 4-bit float with one sign bit, two exponent bits, and one mantissa bit (E2M1). The largest magnitude an element can hold is 6.0; the shared scale stretches the block to the range it needs (OCP MX spec v1.0[5], Rouhani et al., arXiv:2310.10537[6]). A block is 32 × 4 + 8 = 136 bits, or 4.25 bits per value.

Microsoft has not published what "enhanced" MXFP4 changes or how its sub-3-bit formats are encoded. What it has said is the purpose: narrower numbers "reduce memory footprint and data movement," and StorageReview reports the formats target both weights and KV cache storage. When the bottleneck is bytes, cutting bytes per parameter is a direct speedup.

05Mechanism 4: one network from rack to superpod

Each Maia 300 exposes 32 ports of 400 Gb/s Ethernet through an on-die AI network controller (StorageReview[3]). Microsoft runs a custom transport over that Ethernet in two tiers. Inside a rack, two accelerators in a tray share direct links, and switches connect all 96 into a pod using electrical links. Optical second-tier switches connect up to 128 racks into a superpod of 12,288 accelerators (Azure Blog[1]).

The design choice that matters is that there is no separate scale-out network. Most GPU clusters have a fast scale-up fabric inside a node or rack and a slower, host-mediated network between them. Maia uses "the same communication semantics inside the pod and across the superpod," and any accelerator can talk to any other "without involving the host or a discrete scale-out NIC." There are three latency tiers: a directly linked pair, one switch hop inside the pod, and three hops across pods.

That enables two things that matter for inference. First, prefill and decode can run on different accelerators, and after prefill "KV state moves directly to the selected decode workers over the same scale-up fabric." Second, a failed tray in one rack can be covered by a spare in another rack without leaving the fabric. Tensor-parallel traffic, which is latency sensitive, can stay in the closest tier, while expert and pipeline parallelism span racks.

06A worked example: why bandwidth beats FLOPs here

The numbers below are our arithmetic on the published specs, for a hypothetical model, ignoring KV cache and overheads. Treat them as ceilings, not benchmarks.

Step 1: the chip's balance point. Divide peak compute by memory bandwidth. Maia 300: 13.8 × 10¹⁵ ops ÷ 18 × 10¹² bytes/s ≈ 767 operations per byte. Maia 200: 10.1 × 10¹⁵ ÷ 7.1 × 10¹² ≈ 1,423. A workload doing fewer operations per byte than that is limited by memory, not math. Maia 300 needs roughly half the arithmetic intensity to keep its units busy.

Step 2: what decode actually does. Take a model with 100 billion active parameters stored in MXFP4. At 4.25 bits each, the weights are about 53 GB. Generating one token for one user does about 2 operations per parameter (a multiply and an add), so 200 billion operations against 53 GB read: under 4 operations per byte. That is two orders of magnitude below the balance point.

Step 3: the time per step. Reading 53 GB at 18 TB/s takes about 2.95 ms, a ceiling of about 340 decode steps per second. On Maia 200's 7.1 TB/s, the same read takes about 7.5 ms, or about 134 steps per second. The math itself, 200 billion operations at 13.8 petaops, takes about 0.015 ms. The tensor units sit idle for more than 99 percent of the step.

Step 4: the precision lever. Store the same weights in BF16 at 2 bytes each and the read is 200 GB, about 11 ms per step at 18 TB/s. MXFP4 alone cuts the bytes by about 3.8 times. Microsoft's sub-3-bit formats would push further, though it has not given exact sizes.

Step 5: batching. Serving many users at once reuses each weight byte across the batch. At 4.25 bits per weight, each extra user adds about 3.8 operations per byte, so it takes a batch of roughly 200 concurrent sequences to reach the 767 balance point. In practice, every one of those users also brings a KV cache that must be read each step, and for long-context agent sessions that cache can rival the weights. That is why 288 GB of capacity and KV-cache compression matter as much as raw bandwidth.

07What the claims rest on

The two headline numbers deserve scrutiny. "Up to 7x the rack-scale token throughput of Maia 200" compares a 96-accelerator, 167 kW rack with a 48-accelerator, 64 kW rack (StorageReview[3]). Per rack it is a big jump, but the rack also draws about 2.6 times the power. Microsoft separately claims more than 2.5 times the performance per watt, which is the more meaningful figure.

The "40 percent more tokens per megawatt than leading GPU-based systems" claim, as StorageReview notes, does not name the GPU, the model, or the serving setup. Maia is not offered as a rentable Azure instance, so nobody outside Microsoft can check it. Production deployment starts in 2027, by which point the GPU comparison will have moved.

What can be checked is the shape of the design, and it is coherent. Per-accelerator bandwidth rose 2.5 times while compute rose about a third. Data placement is explicit, precision is narrower, and the network uses one set of communication rules across a 12,288-chip superpod, so KV state can move between any two chips without a separate network. This is a chip sized by someone who has measured their own decode traffic. For everyone else, the lesson is that the next round of inference cost cuts will come from moving fewer bytes, not from adding FLOPs.