Adobe paper tests if LLM chain of thought explanations show real influences, and finds they often hide them.

LLMs often notice hidden hints in a question but leave them out of their step by step explanation.

So the point they are making is that LLMs often notice hidden hints in a question but leave them out of their step by step explanation.

People use step by step reasoning as a safety and debugging tool, assuming the explanation matches what drove the answer.

The author plants misleading hints inside multiple choice questions and checks both the final answer and what the model says.

Across 9,154 trials and 11 major LLMs, the study tried 3 instructions, normal, "you are monitored", and "report hints".

Under normal use, models mentioned hints only 20.7% of the time, yet 99.4% admitted seeing them when asked later.

Telling models they were monitored barely changed this, and forcing hint reporting caused 68.2% false alarms when no hints existed.

User pleasing hints, where the question says a confident user believes an option, were followed 45.5% but often unmentioned, making monitoring that only reads explanations unreliable.

----

Paper Link – arxiv. org/abs/2601.00830

Paper Title: "Can They Trust AI Explanations? Evidence of Systematic Underreporting in Chain-of-Thought Reasoning"