FORWARD: Part of a running series. Each audit is dated and tied to the paper versions and model snapshots tested; older audits are not rewritten. Series page: https://theparallaxidentity.com/essays/model-claim-audits(opens in a new tab)
I am keeping this record as, over the past several years, I have conducted research and writing in concert with several LLM’s.I have encountered so many inconsistencies and unhelpful conclusions from both humans and LLM’s it is impossible to keep track.
Tracking matters, as it is precisely illogical and inconsistent thinking which has landed us right in the middle of the current dystopia. The distinction between truth and fiction matters desperately to every human being on this planet.
The audit below is drafted by engineered intelligence for my review. The notebook quotations are its generated answers from a Gemini notebook conversation dated 17 October 2026. They are not my words.
The notebook says: “language models represent numbers as positions along a helix (a spiral shape) in their internal high-dimensional geometric space”
What was observed: Kantamneni and Tegmark (2025) fit a generalized helix to number representations in GPT-J, Pythia-6.9B and Llama 3.1-8B, and used interventions to show it plays a role in small-integer addition. The code combines magnitude with periodic components. It is not a simple three-dimensional spring.
Where the inference outruns the result: The notebook drops the tested models and the small-number range. It also blurs the difference between a representation indexed by number and a path taken during reasoning.
Worth keeping: The geometry is real within its domain.
The open question: Does the same code predict behavior on larger numbers, unfamiliar prompts, and other models?
The notebook says: “Deeper reasoning moves through spirals and uneven stair-stepped leaps, allowing the system to make non-linear jumps across complex internal representations”
What was observed: An earlier answer in the same notebook said “The author proposes”. The later answer drops that attribution. No cited study measures exponential growth of reasoning or a universal spiral path.
Where the inference outruns the result: Nonlinear computation, a helical representation and exponential expansion are three different claims, and none of them entails the next.
Worth keeping: Exponential thinking remains my proposal, and it remains open. The unsupported step is the notebook’s promotion of it into a measured mechanism.
The open question: What grows, along which axis, and what observation would tell exponential growth apart from recurrence, branching, or a projection artifact?
The notebook says: “No true visual or topological reasoning … models do not actually ‘see’ images”
What was observed: In BlindTest (ACCV 2024), four vision-language models averaged 58.07% on seven simple visual tasks; the best scored 77.84%. In BLINK (2024), GPT-4V scored 51.26%, Gemini 45.72%, and humans 95.70%. Later BlindTest analysis found the information could be recovered inside two open models on two tasks.
Where the inference outruns the result: Unreliable answers are not the same as absent ability. A failed answer alone does not show where the information was lost.
Worth keeping: Say exactly which visual relationships were missed, and under which conditions.
The open question: Does the failure remain when overlap, spacing, resolution, and the answer format change?
The notebook says: language is “not as thought itself, but as a secondary translation layer”
What was observed: Coconut (Hao et al., 2024) computes in continuous hidden states rather than words. It scored 97.0% against 77.5% for step-by-step written reasoning on ProsQA, but 34.1% against 42.9% on GSM8k. Turpin et al. (2023) showed that written explanations can leave out what actually influenced the answer.
Where the inference outruns the result: Which route performs better depends on the task and the training. Neither result shows that words are always secondary.
Worth keeping: Internal computation, written scratchpad and explanation are three separate things.
The open question: When the intermediate representation changes, which unfamiliar tasks improve, and which get worse?
The notebook says: its framework will “Accurately Benchmark AI”
What was observed: The conversation supplies categories, but no scoring rules, tested sample, or reliability study.
Where the inference outruns the result: Naming a dimension does not show that it can be measured.
Worth keeping: The framework is worth keeping as a research proposal.
The open question: Would its scores predict held-out performance better than existing tests do?
Kantamneni & Tegmark (2025):
https://arxiv.org/abs/2502.00873(opens in a new tab)Engels et al. (2024/ICLR 2025):
https://arxiv.org/abs/2405.14860(opens in a new tab)Rahmanzadehgervi et al. (ACCV 2024), BlindTest:
https://openaccess.thecvf.com/content/ACCV2024/papers/Rahmanzadehgervi_Vision_language_models_are_blind_ACCV_2024_paper.pdf(opens in a new tab)BlindTest revision v6:
https://arxiv.org/html/2407.06581v6(opens in a new tab)Fu et al. (2024), BLINK:
https://arxiv.org/html/2404.12390v3(opens in a new tab)Hao et al. (2024), Coconut:
https://arxiv.org/html/2412.06769v1(opens in a new tab)Turpin et al. (2023):
https://arxiv.org/abs/2305.04388(opens in a new tab)
Full audit with figure: https://theparallaxidentity.com/essays/notebook-and-the-results
I look forward to your comments and questions in the space below.
