Repository navigation
cot-prompting: make the standard-vs-CoT contrast real again on current models - #59
Conversation
… contrast shows again On current models "answer concisely" no longer stops the model from writing out its steps, so the standard prompt was quietly doing chain of thought too. The baseline now says "Reply with only the final answer, no explanation". The notebook was re-run on gpt-5.4-mini with reasoning_effort="none" and temperature=0, and the new markdown describes what that run shows: on the water-tank problem the baseline answers 1 hour 15 minutes (wrong) and the CoT prompt answers 15 hours 42 minutes (right). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Model setup and basic prompt example all_prompt_engineering_techniques/cot-prompting.ipynb |
The setup adds temperature=0 alongside reasoning_effort="none". The basic example uses a final-answer-only standard prompt and contrasts it with the CoT prompt. |
Calculation examples and comparison all_prompt_engineering_techniques/cot-prompting.ipynb |
The advanced example records a five-step average-speed calculation. The tank example records standard and CoT answers, followed by an explanation of their difference. |
Logic-puzzle output all_prompt_engineering_techniques/cot-prompting.ipynb |
The cell records the role assignments, eliminated cases, and conclusion for the logic puzzle. |
Estimated code review effort: 2 (Simple) | ~10 minutes
Change: Bug fix
Possibly related PRs
- NirDiamant/Prompt_Engineering#55: Updates the same notebook’s model setup, which PR
#59uses for its refreshed examples and outputs.
Suggested reviewers: adornes
Merge Risk: 
c0a18
-
The updated notebook has no identified merge-blocking issue.
Pre-merge checks: 
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title is in the right area: it names the notebook (cot-prompting) and the goal of restoring a standard-vs-CoT contrast on current models. The diff does show that contrast change: the baseline prom… Docstring Coverage No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Full details: Title check
-
Explanation
The title is in the right area: it names the notebook (cot-prompting) and the goal of restoring a standard-vs-CoT contrast on current models. The diff does show that contrast change: the baseline prompt now asks for only the final answer, and the notebook sets temperature=0 and reasoning_effort="none" for gpt-5.4-mini. The title is specific and short, and it names the main change. It is a single sentence of 10 words and avoids vague terms. Its wording is a little informal ("make ... real again"), but a teammate scanning history would understand the primary change.
Full details: Docstring Coverage
-
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
-
Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can keep summaries lean with Findings visible per group, which tucks the rest behind a View link Powered by Qodo |
What this fixes
cot-prompting.ipynbis meant to show that Chain of Thought beats a standard prompt on multi-step problems. On current models the standard prompt ("Answer the following question concisely") no longer suppresses reasoning: in 4 of 4 probe calls ongpt-5.4-miniit wrote out the tank volume, the remaining third and the conversion anyway (about 200 output tokens each). Both prompts were doing chain of thought, so the contrast was gone.Changes (1 notebook)
temperature=0on thegpt-5.4-minimodel, so the run is repeatable. GPT-5.x accepts a non-default temperature only withreasoning_effort="none"(verified: withreasoning_effort="low"the API returns 400 for temperature 0), which this notebook already sets.The water-tank problem still shows the contrast, so no problem was swapped.
Verification
gpt-5.4-miniwithreasoning_effort="none", andgpt-5.4for the logic puzzle). The logic puzzle resolved correctly (Amy liar, Bob truth-teller, Charlie alternator). Before editing, I probed the water-tank problem 4 times per prompt at temperature 0: "concisely" always reasoned on the page, and "final answer only" always answered 1 hour 15 minutes.Left for Nir
🤖 Generated with Claude Code
PR Summary by Qodo
Restore the standard-versus-CoT contrast in the prompting notebook
🐞 Bug fix📝 Documentation🕐 10-20 MinutesAI Description
Diagram
graph TD Q["Example question"] --> B["Final-only prompt"] --> M("Mini model") --> O["Saved comparison"] Q --> C["Step-by-step prompt"] --> MHigh-Level Assessment
Files changed (1)
+524 / -61Bug fix (1)
+524 / -61cot-prompting.ipynb
Restore and explain the final-only baseline versus CoT comparison+524/-61Summary by CodeRabbit