GLM-5.2 QuantTrio on 4x ASUS GX10 — stable baseline and failure notes

Date: 2026-07-14 Hardware: 4x ASUS Ascent GX10 / NVIDIA GB10, 128 GiB unified memory each, ConnectX-7 200 GbE fabric Container image: vllm-node-tf5-glm52-b12x:probe-modded-cutlass453 vLLM: 0.23.1rc1.dev190+gab6660699.d20260713 Model: local GLM-5.2 QuantTrio mixed int4/int8 weights at /cache/huggingface/hub/glm52-int4-int8mix Serving name: glm-5.2 Working launch: TP=4, eager mode, fp8_ds_mla KV, 160k context, no MTP, no async scheduling API: http://<head-node>:8210

Field notes

This one did not match the launch guide cleanly. The model can be made useful on the 4-node GX10 cluster, but the advertised high-context + MTP path is not yet a drop-in recipe on this stack.

The stable baseline that actually came up was deliberately conservative. The first benchmarked run used 128k:

--tensor-parallel-size 4
--max-model-len 128000
--max-num-seqs 6
--max-num-batched-tokens 8192
--kv-cache-dtype fp8_ds_mla
--kv-cache-memory-bytes 7000000000
--enforce-eager

The launch also required the local RoCE fabric settings, including NCCL_IB_HCA=rocep1s0f1, NCCL_SOCKET_IFNAME=enp1s0f1np1, and NCCL_IB_GID_INDEX=2. The GID index is cluster-specific; the public examples using 3 were not correct here.

The successful boot reported:

GPU KV cache size: 129,728 tokens
DSA indexer decode path: use_flattening=False (next_n=1, use_fp4_indexer_cache=False)
init engine (profile, create kv cache, warmup model) took 22.00 s
Application startup complete.

After benchmarking, the same no-MTP/no-async shape was relaunched at 160k and served successfully:

GPU KV cache size: 161,280 tokens
DSA indexer decode path: use_flattening=False (next_n=1, use_fp4_indexer_cache=False)
Application startup complete.

That 160k service is the current operational baseline. A 200k no-MTP/no-async retry still did not finish serving, even with extra temporary swap available.

What failed

The original MTP-heavy path is the dangerous one. With MTP num_speculative_tokens=4, vLLM derived next_n=5 and entered the flattened DSA indexer path:

DSA indexer decode path: use_flattening=True (next_n=5, use_fp4_indexer_cache=False)

That path repeatedly wedged one GX10 hard enough that SSH banner exchange would time out until the unit was reset.

Reducing MTP to one speculative token avoided the flattened path in theory, but the run still stalled during KV/indexer setup. Removing MTP entirely at 200k context got farther:

GPU KV cache size: 202,944 tokens
DSA indexer decode path: use_flattening=False (next_n=1, use_fp4_indexer_cache=False)
No available shared memory broadcast block found in 60 seconds.

That wait repeated indefinitely. The service never bound the API. During the stable 128k run, the nodes were already tight on host memory: around 114-116 GiB used, only 3-7 GiB available, and a 10 GiB swapfile per node with several GiB in use on three ranks. Larger-context investigation should start by separating host-memory/swap pressure from vLLM indexer behavior.

Benchmarks

Sampling: temperature=0.0. The concurrency sweep uses a 10,086-token prompt and requests 1,024 output tokens. This run is no-MTP and eager, so decode is much slower than the MTP-enabled Qwen3.5 baseline, but it is stable.

The N=8 and N=16 prefill numbers include queueing because this stable launch uses --max-num-seqs 6. Once more than six requests are in flight, TTFT comes in waves and aggregate prefill is no longer a pure prefill measurement.

Needle-in-a-haystack

The first raw completions NIAH attempt at 126,579 input tokens returned only an empty/EOS token, so it was not a useful retrieval result. Re-running through the chat endpoint with more headroom under the 128k limit succeeded.

Needle:

IMPORTANT: The secret access code for Project Aurora is BANANA-MOOSE-7421. Remember this exactly.

Question:

Based on the text above, what is the secret access code for Project Aurora? Reply with just the code itself, nothing else.

Final visible answer:

BANANA-MOOSE-7421

The model produced a short reasoning trace before the visible answer, but the answer itself was exact.

Next investigation

The main unresolved feature is larger context. The next runs should walk upward from the stable baseline instead of jumping back to the full recipe:

1. 128k baseline with larger swap and the same no-MTP/no-async config.
2. 160k no-MTP/no-async, keeping use_flattening=False.
3. 200k no-MTP/no-async after host memory pressure is removed.
4. Only then reintroduce async scheduling.
5. Only after the 200k no-MTP path is stable, retry MTP with num_speculative_tokens=1.

Do not retry MTP k=4 on this stack until the flattened DSA path is patched or proven safe on GB10.

DCP fork investigation

The newer XanuNetworks DCP repo is not just a better launch script for the same image. It depends on a different vLLM fork stack:

• local-inference-lab/vllm branch codex/dcp-globaltopk-sharddraft-defaults-20260622 at e232d262369b8c918cf478a7a96a0fcf8127cf65
• CosmicRaisins PR72 patches for draft-under-DCP plus the draft quant packed mapping patch
• b12x 9cd63a7
• B12X_MLA_SPARSE, VLLM_USE_V2_MODEL_RUNNER=1, VLLM_DCP_GLOBAL_TOPK=1, and VLLM_DCP_SHARD_DRAFT=1
• explicit index_topk_pattern and clear_thinking:false
• DCP2/DCP4 KV sharding, which is how the repo advertises 327k/655k context

That is the primary reason this cluster could not reproduce the Xanu numbers from the earlier non-DCP image. The current image is a different vLLM commit and does not contain the DCP machinery.

The DCP audit found several cluster-specific blockers before any launch attempt:

The patch story also needed care. Each patch applies individually to the pinned fork, but the full set does not apply blindly because Xanu's b12x-prewarm-tolerance.patch duplicates a topk_scores_buffer hunk already provided by CosmicRaisins' pr72-2 patch. The safe sequence is:

pr72-1-draft-dcp-config-propagation.patch
pr72-2-glm-dcp-draft-path.patch
draft-quant-packed-mapping.patch
xanu-b12x-prewarm-only.patch

The cleaned xanu-b12x-prewarm-only.patch keeps only the prewarm try/except logic and validates cleanly after the three CosmicRaisins patches. A guarded DCP build harness and dry-run launch adaptation now live outside the public blog tree under dcp-audit/; they refuse to build on x86 and refuse to launch while the current 160k service is running.

2026-07-14 DCP build and launch attempts

The DCP image build completed successfully on the head GX10:

image: vllm-node-eldritch-dcp:e232d26-modded
vLLM: 0.23.1rc1.dev248+ge232d2623.d20260714
vLLM commit: e232d262369b8c918cf478a7a96a0fcf8127cf65
FlashInfer commit: a92fce7a
torch: 2.11.0+cu130
nvidia-cutlass-dsl: 4.6.0
nvidia-cutlass-dsl-libs-cu13: 4.6.0
b12x: 0.23.0 from lukealonso/b12x@9cd63a7
build time: 47m25s

The build warnings were not fatal, but are worth tracking: DeepGEMM and FlashMLA reported unsupported CUDA arch 12.0, and the final image uses the DCP fork with FlashAttention and vLLM CUDA objects compiled for 12.1a. The image was copied to all three workers with docker save | docker load; no old baseline containers were removed.

The first real DCP2 launch used the repo's 327,680-token intent:

--max-model-len 327680
--decode-context-parallel-size 2
--gpu-memory-utilization 0.89
--attention-backend B12X_MLA_SPARSE
--speculative-config {"method":"mtp","num_speculative_tokens":4,...}

That run did not wedge the cluster. It loaded all 128 shards in 334.66 seconds, compiled, profiled, and then failed cleanly at KV allocation:

Available KV cache memory: 7.52 GiB
327680 tokens need 8.35 GiB
estimated maximum model length: 294912

A second 327,680-token attempt with --gpu-memory-utilization 0.90 also failed cleanly. The log first reported Available KV cache memory: 9.19 GiB, but the final allocation check still failed against an effective 7.22 GiB minimum and estimated only 283,392 tokens. That mismatch suggests the limiting rank or DCP KV layout check is not the same value printed by the head worker's early profiling line.

A conservative retry at 262,144 tokens with the same DCP/MTP stack and --gpu-memory-utilization 0.90 came up successfully on port 8211:

max_model_len: 262144
decode_context_parallel_size: 2
Available KV cache memory: 9.14 GiB
GPU KV cache size: 274,304 tokens
Maximum concurrency for 262,144 tokens per request: 1.05x
init engine (profile, create kv cache, warmup model) took 92.37 s
Application startup complete.

The full launch loaded the 128 main shards in 303.82 seconds, loaded the filtered four-shard draft path in 8.24 seconds, and reported total model loading of 96.81 GiB in 329.88 seconds. This run used DCP2, B12X sparse MLA, async scheduling, and MTP with num_speculative_tokens=4. The API advertises the served model as glm-5.2 with max_model_len 262,144.

This suggests the failed 327,680-token recipe was mostly a KV headroom issue on this hardware, not a general DCP boot failure. The remaining gap to the public 327k/655k claims is finding where the extra KV budget comes from: lower memory fragmentation, a different rank layout, a different driver/runtime envelope, or another launch flag in the upstream environment.

2026-07-21 larger-context cleanup pass

After disabling unused desktop and appliance services on the Spark nodes including Bluetooth, CUPS, Avahi, snapd, ModemManager, display-manager/gdm, and other GUI-adjacent daemons, the cluster still did not have enough KV headroom for the full DCP2 target. The useful result was a higher stable ceiling than the previous 262k service:

The working 275,840-token launch kept the same DCP2/MTP shape:

max_model_len: 275840
decode_context_parallel_size: 2
gpu_memory_utilization: 0.90
kv_cache_dtype: fp8_ds_mla
MTP speculative tokens: 4
GPU KV cache size: 277,120 tokens
Maximum concurrency for 275,840 tokens per request: 1.00x
Application startup complete.

The API smoke test returned normally. Runtime memory remains tight: three nodes kept a few GiB of swap allocated from launch-time pressure, but the extra 64 GiB swap files were unused and short idle vmstat checks did not show sustained swap-in/swap-out churn. This is still not a comfortable production margin; it is a practical larger-context bring-up point for further testing.

DCP2 262k benchmark pass

The classic 10k concurrency sweep was rerun against the DCP2 service on port 8211. This run uses async scheduling, B12X sparse MLA, DCP2, and MTP with four speculative tokens.

The DCP/MTP path roughly doubles single-request decode versus the old eager no-MTP baseline, but it did not increase aggregate decode with concurrency in this harness. Requests are served in visible waves, so TTFT grows quickly at higher concurrency.

Needle-in-a-haystack was also rerun near the new context ceiling:

Final answer contained the exact code, with extra preamble:

If the text does not contain the code, reply with "NOT FOUND".

BANANA-MOOSE-7421

Runtime swap behavior was acceptable but not literally zero-touch. Three nodes kept swap pages allocated from launch-time pressure: about 3.8 GiB on the head and about 2.0 GiB on two workers. During the 10k concurrency sweep, pswpout stayed flat on every node, while the head and one worker showed only small pswpin increases. During the 240k NIAH prefill, the head added roughly 45 MiB of swap-out; the workers stayed flat. So the service is not relying on swap for the steady inference working set, but the current 262k configuration still runs close enough to host-memory pressure that the head can touch swap under a near-ceiling long-context request.

Artifacts

• results-run6.json — full per-request baseline benchmark output
• niah-chat-120k.json — successful baseline NIAH result
• results-dcp262k.json — DCP2 262k benchmark output

The benchmark harnesses and raw console logs are kept in the private source tree because they include operator-local endpoints and filesystem details.