Route-Block Membership Selects Packed-AWQ Arithmetic
A controlled mechanism study showing that route membership in padded MoE blocks selected exact packed-AWQ arithmetic trajectories in one pinned fixture.
Plain-language summary
Mixture-of-experts inference groups routed rows into padded blocks before GPU matrix multiplication. In one controlled fixture, changing which block two rows entered changed the exact calculation path, while changing their order inside one block did not.
Technical summary
A 70-process prospective mechanism matrix on one pinned Qwen3-Coder AWQ layer-6 fixture, testing cross-order, within-block, block-boundary, and canonical controls across four exact surfaces, plus a W13 schedule control.
Why it matters
Alignment often looks like dispatch bookkeeping. This result identifies a boundary where alignment becomes part of exact arithmetic control flow, making route construction relevant to reproducibility in the tested packed kernel.
The result and its boundary
In the controlled fixture, mixture-of-experts alignment acted as arithmetic control flow. Moving two selected routes across a padded-block boundary transferred the complete exact arithmetic trajectory. Permuting two routes inside one block preserved it.
The boundary matters just as much: one Qwen3-Coder AWQ layer-6 fixture, one pinned vLLM/Marlin build, and one RTX 3090 runtime.
The bookkeeping before the matrix multiplication
An MoE layer routes each token to a small set of experts. Before the expert matrix multiplications run, the system groups those routed rows by expert and pads them into fixed-size blocks.
It is easy to treat that alignment as dispatch bookkeeping. The packed kernel, however, schedules blocks onto different reduction paths. A row's exact floating-point trajectory can therefore depend on which padded block receives it, even when the routed inputs, expert assignments, routing weights, and packed model weights stay fixed.
The paper captures that distinction with route-block membership. It records which valid flattened route identifiers occupy each eight-row expert block and ignores their order inside the block.
The boundary that mattered
Two fixed preconstruction histories, H0 and H1, selected different native alignments and different exact trajectories. The histories were composite treatments; they did not isolate an allocator primitive or a named prewarming effect.
The causal controls acted on the alignments themselves.
- Injecting the opposite history's alignment transferred the first expert projection (W13), activation, routed second projection, and final output to the opposite exact trajectory.
- Permuting prior-data-selected routes 22 and 28 inside expert-106 block 40 preserved each native trajectory.
- Exchanging prior-data-selected routes 51 and 86 across expert-106 blocks 40 and 41 transferred the complete opposite trajectory.
For the pinned Marlin geometry, those neighboring blocks belonged to different schedule classes. Block 40 used direct, full-K execution. Block 41 split the K dimension and used a global FP32 reduction. The boundary exchange changed which selected routed rows entered which reduction schedule.
Within one block
routes 22 ↔ 28Across the boundary
routes 51 ↔ 86Removing the split reduction
The schedule control forced a 200-block CUDA grid. Under that geometry, every column tile used a single slice and bypassed the global reduction. H0 and H1 then produced bitwise-equal W13 tensors.
That result supports the schedule-class explanation at W13. It does not show that one instruction caused the difference, and it does not claim final-output collapse under the grid intervention.
Stable construction selected a third trajectory
The paper also evaluates stable construction: valid routes are ordered by expert identifier and flattened route identifier rather than by the native atomic construction order.
The evaluated implementation was disabled by default and scoped to AutoAWQ-Marlin. The paper does not claim that upstream vLLM has adopted it.
Both histories then converged to the same four-surface trajectory. That result was not a recovery of H0 or H1. Stable construction selected a third exact trajectory and removed the observed history dependence in the tested operator scope.
The distinction is useful. This test did not require declaring one historical trajectory correct. Stable construction made the same inputs select the same trajectory under both histories.
The evidence package
The prospective mechanism matrix contains 70 valid cold processes: 20 for qualification, 20 for causal transfer, and 30 for ablation. Every cell and history has five exact replicates. Seven required mutations to routes, markers, hashes, raw tensors, and software identity were rejected for their declared reasons.
Those repetitions establish repeatability inside one independent problem instance. They do not estimate external validity.
The public recorded-evidence artifact contains the claim ledger, frozen plans, execution records, schedule geometry, software-build linkage, implementation patch, and a CPU-only verifier. The verifier checks the released evidence and trajectory relations. It does not re-execute the GPU tensors.
The artifact also keeps the canonical-implementation evaluation separate from the 70-process mechanism cohort. The two builds share exact anchors but use different complete diffs, binaries, and process populations.
What the paper does not establish
The native difference was sparse. At the final surface, it was one FP16 element. That is scientifically diagnostic; it is not evidence of practical impact.
The paper does not estimate prevalence across prompts, layers, checkpoints, models, GPUs, runtimes, or vLLM versions. It does not identify allocator, fragmentation, address residue, or prewarming as the cause. It makes no claim about model-token divergence, task quality, serving impact, portability, serving-scale performance, paging, or prefetch behavior.
The public package verifies the recorded evidence but omits the private prompt, checkpoint-derived fixture tensors, binary extensions, and raw tensor captures. It therefore supports evidence audit and methodological replication, not exact independent reproduction of the reported bit patterns.
The bounded result still changes how I read this path:
In this controlled fixture, alignment was part of the arithmetic mechanism, not merely preparation for it.
What I am looking for feedback on
The useful review question is whether route-block membership is the right abstraction for adjacent packed kernels. If another scheduling dimension is needed to explain exact-output changes elsewhere, that is the boundary I would most like to correct.