GLM-5.2 benchmark journey — what worked, what did not, and what to test next
Date: 2026-07-15 Hardware: 4x ASUS Ascent GX10 / NVIDIA GB10, 128 GiB unified memory each, 200 GbE fabric Model: GLM-5.2 QuantTrio mixed int4/int8 Best serving mode so far: DCP2, 262k context, B12X sparse MLA, async scheduling, MTP with four speculative tokens
Short version
GLM-5.2 is usable on the 4-node GX10 cluster, but the path there was not a one-command deployment. The plain non-DCP launch was stable at lower context, but it could not reproduce the long-context performance advertised by the newer DCP repo. The DCP fork changed that: it brought the service up at 262k context and kept the cluster stable through long-context benchmarks and a full SWE-bench Lite run.
The main surprise was not context length. The service handled prompts in the 30k, 40k, 60k, and 75k token range during SWE-bench without falling over. The main issue was answer formatting. For code-patch benchmarks, too many responses spent their output budget on reasoning or malformed diffs instead of producing a clean patch.
Deployment lessons
The safer path was incremental:
- Get a conservative baseline running first.
- Add DCP only after checking the exact vLLM fork, patches, image, and network assumptions.
- Avoid retrying a risky launch shape after a hard wedge; change one variable at a time.
- Treat swap as launch-time cushion, not a runtime dependency.
- Run a small benchmark before trusting a full run.
The original high-context, MTP-heavy path repeatedly ran into either hard cluster failure or KV-cache headroom limits. The DCP2 262k launch was the first configuration that combined high context, MTP, and operational stability.
The 327k target still failed on this hardware because there was not enough KV headroom. That failure was clean, not a wedge. It points to memory budget and rank layout, not a basic DCP boot problem.
User-facing benchmark results
The 262k DCP2 service returned exact needle-in-a-haystack retrieval near the top of the context window:
| Test | Result |
|---|---|
| Context tested | 240k input tokens |
| Retrieved needle | yes, exact match |
| Prefill speed | about 626 tokens/s |
| Decode speed | about 29 tokens/s |
The classic 10k prompt concurrency sweep showed better single-request decode than the old no-MTP baseline, but requests still arrived in visible waves at higher concurrency:
| Concurrency | Median per-request decode |
|---|---|
| 1 | 26.0 tokens/s |
| 2 | 24.9 tokens/s |
| 4 | 25.8 tokens/s |
| 8 | 25.9 tokens/s |
| 16 | 26.7 tokens/s |
That is good interactive decode for one request, but not a magic throughput multiplier. If the app queues many long requests, time-to-first-token grows.
SWE-bench Lite
We ran the full 300-instance SWE-bench Lite set against the live 262k DCP service.
| Metric | Result |
|---|---|
| Submitted instances | 300 |
| Resolved instances | 94 |
| Full Lite score | 31.3% |
| Non-empty patches evaluated | 219 |
| Resolved among non-empty patches | 42.9% |
| Empty or unusable generations | 81 |
| Patch/eval errors | 52 |
| Serialized generation time | 9.35 hours |
| Evaluation time | 51m 48s |
This does not read to me as "the model cannot code." It reads as "the current serving and prompting path is not strict enough for patch-only benchmarks." When the model emitted a clean patch, it solved a meaningful number of tasks. The failure mode was often verbose reasoning, capped output, or malformed patch syntax.
For anyone testing coding models locally, this is the part worth copying:
- validate patch shape before evaluation;
- retry malformed outputs with a shorter, stricter patch-only prompt;
- track empty patches separately from failed tests;
- do not compare a raw first-pass run against systems that use repair loops.
Runtime behavior
The DCP service stayed up during the benchmark work. The long-context tests did not wedge the cluster. Swap was still present from launch-time pressure, and one near-ceiling long-context request touched a small amount of swap on the head node, but the steady inference path did not look like it was relying on NVMe swap for normal operation.
For production-style use, I would still keep the operational rule simple: swap can help a large model load, but the running service should fit in memory.
What we would change next
The next improvement is not another bigger launch. It is stricter output control:
- Run a patch-only generation path for SWE-bench.
- Stop generation as soon as
</patch>appears. - Ignore reasoning fields when extracting the final answer.
- Validate diffs before evaluation.
- Retry only the malformed or empty cases.
The cluster and DCP path are now good enough to test the model. The benchmark harness needs to be stricter before drawing a hard conclusion about model quality.