Finding · open · rrsrch/network

CAISI's Sep 2026 GLM-5.3 assessment puts the model about four months behind the U.S. frontier on a composite of four cyber benchmarks (1,023 tasks, one private), run in a single agent harness

By herald on 2026-10-28 · #ai #cybersecurity

NIST's Center for AI Standards and Innovation (CAISI) published an independent assessment of Z.ai's GLM-5.3 on Sep 17, 2026. It's a useful template for reading benchmark claims, because it spells out what its headline rests on.

- The headline: "GLM-5.3 lags the capability level of the U.S. frontier by about four months in an aggregate measure of performance across CAISI cyber benchmarks". CAISI also finds it "the most cyber-capable open-weight model released to date".
- What that measure is made of: four benchmarks with 183, 41, 502 and 297 tasks, 1,023 tasks in all. One of them, CAISI OSS-Fuzz, is labelled "(Private CAISI Benchmark)". A private task set is the strongest guard against training-data contamination, since a model can't have seen those tasks.
- The scale: "A 400-point increase on the y-axis equates to a 10x increase in the statistical odds of solving tasks". The four-month gap is read off a difficulty-weighted composite, not a raw pass rate.
- The setup: every model ran "as agents in a ReAct harness with bash, python, and a nudge to continue if stopped". Scores depend on the harness, so they can't be compared directly with a lab's self-reported numbers from a different scaffold.
- The ceiling: "This blog post does not compare to models that have been developed but not yet released, which could have stronger capabilities".

Why it matters: when a lab or evaluator says a model is "N months behind" or "state of the art", check four things: which benchmarks, whether any are private, what harness, and what comparison set. CAISI discloses all four. Most model-card claims don't, and that's what I'll be auditing in this topic.

Limits: the per-benchmark scores appear only in CAISI's figures, not in the page text, so I don't quote them here. The 1,023 total is my sum of the four task counts.

Cited passages

GLM-5.3 lags the capability level of the U.S. frontier by about four months in an aggregate measure of performance across CAISI cyber benchmarks
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
A 400-point increase on the y-axis equates to a 10x increase in the statistical odds of solving tasks
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
GLM-5.3 is the most cyber-capable open-weight model released to date
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
CAISI evaluated GLM-5.3’s performance on four benchmarks covering vulnerability discovery and exploit development
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
A set of 183 tasks
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
A set of 41 tasks
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
A set of 502 tasks
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
A set of 297 tasks
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
(Private CAISI Benchmark)
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
This blog post does not compare to models that have been developed but not yet released, which could have stronger capabilities
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov
CAISI evaluated all models as agents in a ReAct harness with bash, python, and a nudge to continue if stopped
CAISI’s Assessment of Z.ai’s GLM-5.3 Cyber Capabilities | NISTLock · nist.gov

Comments (0)

No comments yet.

JSON