Definition
Chain-of-thought is a prompting technique that encourages a model to articulate intermediate reasoning steps, not just the final answer, to improve performance on multi-step problems. It can raise accuracy but may increase output length and risk.
"If you have 5 apples, eat 3, and receive 2 more, how many do you have?" -- Even problems humans can solve mentally can trip up LLMs. But simply adding "please think step by step" dramatically improves accuracy. Chain-of-Thought (CoT) is a prompting technique that improves answer accuracy on complex problems by having the LLM explicitly output intermediate reasoning steps.
Why "Making It Think" Improves Accuracy
LLMs typically try to jump straight to a final answer upon receiving a question. For problems requiring multiple steps, the intermediate reasoning gets crammed into "internal processing," making it easy to leap to incorrect conclusions. With CoT, the reasoning process is output as text, so accurate intermediate results are generated at each step, and subsequent steps build on them. It's the same effect as "thinking on paper" rather than "thinking in your head."
Two Approaches
CoT has two main implementation methods. Zero-shot CoT is a simple technique that just adds "Let's think step by step" to the end of the prompt. No special examples need to be prepared, making it easy to try.
Few-shot CoT includes input-output examples with reasoning processes in the prompt. By providing 2-3 examples of the sequence "problem, reasoning steps, answer," the model learns to develop reasoning in the same format. It tends to be more accurate than zero-shot CoT but requires the effort of preparing quality examples.
Where It's Most Effective
CoT is particularly effective for arithmetic, logical reasoning, commonsense reasoning, and code generation. Google's research (2022) showed that using CoT significantly improved large models' accuracy on arithmetic reasoning. On the other hand, for tasks that don't require step decomposition, such as simple fact retrieval ("What is the capital of Japan?") or translation, the benefits of CoT are minimal.
Trade-offs
CoT has clear trade-offs. Because it outputs the reasoning process, output token count increases significantly, raising response time and API costs. Longer reasoning chains also increase the risk of establishing incorrect premises along the way. "Accuracy vs. cost" is a judgment that must be made based on the use case. In production environments, an "internal CoT" pattern is sometimes used where the model reasons with CoT but only the final answer is displayed to the user. Recently, "reasoning models" that embed CoT mechanisms within the model itself -- such as OpenAI's o1 and o3, and Claude's extended thinking feature -- have emerged, enabling high-accuracy reasoning without explicit prompting.