Past midnight, coffee gone cold, mid-scroll through GitHub. And one line stopped me: “Do not teach anything until the diagnostic is done.”
That’s from the socratic-tutor SKILL.md, pinned at commit 7c6715c2, one skill inside Jane Street’s agent_skills repository. The repo README describes the whole collection plainly — “a collection of skills for AI agents and chatbots”. Which undersells what’s in this particular file. Load the skill, say “teach me X,” and you don’t get the usual essay. You get 3 to 5 questions before a single lesson, a skill map that scores each of 4 to 10 subskills from 0 to 5, one question at a time, spaced review. Final assessment at the end.
If you build things with LLMs, that file’s worth more than the prompt-engineering threads you’ve bookmarked.
Not because Socratic questioning is new. As it’s the cleanest example I’ve found of a prompt written as a process contract instead of a personality description. And that distinction transfers to your builds far beyond tutoring.
How Jane Street’s Socratic Tutor Runs a Session
Fire one of the trigger phrases — “teach me,” “tutor me,” “quiz me,” “drill me,” “help me master.”. And the skill wakes up. First move: it tells you it’ll ask a few questions, one at a time, to see what you already know. Then comes a short diagnostic of 3 to 5 probing questions, spanning easy to hard, to place your starting level per subskill.
Decomposition happens before any of that, though. The prompt instructs the tutor to break your topic into 4 to 10 concrete subskills, fewer for narrow topics. Broad topic? Build a two-level tree and pick the first branch to work on. Once the diagnostic is done, you see the initial skill map with each subskill rated 0 to 5 and a proposal for where to start.
Loop holds from there. One question at a time. Spaced review of earlier material. A final assessment before the session calls you finished.
Question style is specified too. The prompt says to “prefer questions that make the user: predict an outcome, explain why, find the flaw in a plausible-but-wrong claim, apply an idea to a new case, or connect two ideas.” That’s a taxonomy of understanding. Not a vibe.
What the socratic-tutor SKILL.md Gets Right
Scope control. Most prompt writers never write it.
The SKILL.md doesn’t just say when to activate.
It says when not to. Directly under the trigger phrases sits the line “Not for one-off ‘explain X’ or ‘what is X’ questions.” One sentence. Cheap. And it’s what makes the skill safe to leave installed, since if it fired on every question it would wreck the agent for normal work. Production prompts I see usually carry an activation condition and no deactivation condition, which is how a “helpful assistant” persona ends up fighting the task you actually assigned. If you write skills for coding agents or chatbots, that not-for line is the cheapest upgrade available to you.
Strictness works the same way. Plenty of people have typed “act as a strict tutor” into a chatbot and gotten a model that flatters them anyway. Jane Street’s version behaves differently since the strictness lives in the rules, not the adjectives. Stated goal: a tutor that “takes the user from their current level to genuine mastery of a topic,” teaching “primarily by asking questions that force the user to reason, predict. And explain” rather than lecturing. Mastery is gated by procedure. No teaching before the diagnostic, a level per subskill, spaced review, a final assessment.
Model can’t skip a gate that IS the instruction.
This is the transferable insight. When a prompt underperforms, the fix is rarely more personality. Fix is explicit sequencing, explicit scoring, explicit refusals. My own automation work keeps proving it: the workflows that survive contact with real use are the ones with hard gates written into the instructions. The ones that drift are the ones where I described an attitude instead of a procedure.
Where You’d Actually Use the Socratic Pattern
You probably don’t need a Socratic tutor for its own sake.
You need the pattern: diagnose before delivering, score explicitly, gate progression on demonstrated performance.
And revisit old material on a schedule.
Fits more small-business work than you’d expect. Onboarding a contractor onto your stack. Diagnostic first, skill map second, drills third. Training yourself on a framework before you bet a project on it, identical shape. And the 0 to 5 map tells you exactly which corner of the topic you’ve been bluffing about.
Sleeper use case is internal documentation: converting reference material into a quiz-style skill turns a doc people skim once into something that checks whether they retained any of it.
Running it is deliberately simple, per the repo README. With a coding agent that supports the common SKILL.md format, you copy the skill directory into your agent’s skills folder, then ask the agent to use it (“teach me X”). With a plain chatbot, you paste the body of the skill’s SKILL.md into the chat or into a custom/system prompt, followed by what you want it to teach. Whole integration is a copy or a paste, with tools you already pay for.
The skeleton, if you want to steal it. Six decisions, written down:
– Activation triggers. The five phrases that wake the skill
– A not-for clause. One-off “explain X” questions stay out
– A diagnostic gate. No teaching until 3 to 5 questions are answered
– A scoring scale. Every subskill rated 0 to 5
– A review schedule. Spaced review of earlier material
– An exit exam — final assessment before the session ends
None of it needed fine-tuning. None of it needed a new tool. Open the socratic-tutor SKILL.md and read it top to bottom, since it reads faster than the threads written about it. Then, before your next prompt, write its opposite: one line describing what your prompt must not do. That line separates a skill you can install and forget from one you have to babysit.
FAQ
What is Jane Street’s Socratic tutor?
It’s `socratic-tutor`, one skill inside Jane Street’s agent_skills repository on GitHub. Load it and say “teach me X,” and it runs a diagnostic of 3 to 5 questions, scores your 4 to 10 subskills from 0 to 5, then teaches through one question at a time with spaced review and a final assessment.
How do you install it?
With a coding agent that supports the common SKILL.md format, copy the skill directory into your agent’s skills folder and ask the agent to use it.
With a plain chatbot, paste the body of the SKILL.md into the chat or a custom/system prompt.
What is in the SKILL.md file?
Activation triggers, the not-for clause, the diagnostic gate, the 0 to 5 scoring scale, spaced review rules, and a final assessment. A complete methodology in one plain text file.
Sources
– Jane Street agent_skills repository: https://github.com/janestreet/agent_skills
– socratic-tutor SKILL.md: https://github.com/janestreet/agent_skills/blob/7c6715c29548a3b39d4a9545c698ee4fd17a7213/socratic-tutor/SKILL.md
– agent_skills README: https://github.com/janestreet/agent_skills/blob/master/README.md
