Finally a good paper testing whether an LLM's mathematical knowledge has coherent structure or just coherent accuracy.
There is a lot of interest in using LLMs for advancing math. This is a great read testing whether LLMs math knowledge has structure.
Researchers apply Knowledge Space Theory, which formalizes the idea that mastering a concept requires mastering its prerequisites, as a normative standard.
They evaluate eight open and closed models against real human learners.
Two important results:
The models frequently violate knowledge dependencies, answering a dependent question correctly while failing its prerequisite, and they do not use related knowledge supplied in context to improve on the dependent question.
Second, the eight models show low overlap in their knowledge distributions, so they do not share a consistent structure with each other either.
Both deficits stay invisible to accuracy based scoring and to LLM as judge evaluation, because both score items independently and never look at the dependency graph.