A Language Model Can Be Honest in Prose and Still Fabricate in JSON

Ask a language model a question it can't answer, and it will usually tell you so. This is trained behavior, and the industry measures it — there are benchmarks scoring models on their willingness to say "I don't know." Every one of those…