← All workshops · Slides of workshop 3

Workshop 3 - Let it search, give it know-how: facilitator sheet

For the person who runs the workshop. The participants' page is the README.md of the repository of the workshop, yal-workshop-search (source: workshops/3-search/ of the facilitator's repository), and their one-page sheet sheet.md.

⚠️ Important. From this workshop on, in modes C, D and E, the model can read every file of the repository, including its README.md, and replay does not flag it. The participants' pages therefore never quote an expected answer. Keep it that way when you adapt them, and keep this sheet out of the repository.

Question of the day: what if the model searched by itself? Message to leave with: the model decides what it reads, and you can check each step; a skill makes it do a task you would otherwise repeat by hand, the same way every time; a person decides what enters the knowledge, and signs what the chatbot says about it.


Before the workshop (about 1 hour)

Time What How you check it
15 min One Anthropic API key per pair, all in one workspace for the room, capped at $3 × the number of pairs (see Who pays). workshop-login says Key OK
25 min Your own run: a new Codespace from https://codespaces.new/Youth-AI-Lab/yal-workshop-search, workshop-start 3, openwebui start in a second terminal, workshop C with question 3; the Benevento command of inbox/README.md; summarize corpus/80-benevento-aib-2026.pdf, check two rows and say "I checked summary 80" to see the refusal, then check every row and sign (validate-summary): summaries/index.md says "(human-reviewed)"; the rumour-check skill; replay C 3 11; openwebui chatbot --name "<your chatbot>" (your fallback for the closing) each step as the page says
10 min Ask each pair, a few days before, for the address of the official page of its question 12. Try it with corpus-add on your own run: some pages are refused or nearly empty (see below) the readable copy holds the fact of question 12
10 min Print the one-page sheet (one per pair; translate it first if your room needs another language) and the exit cards (one per person: the three questions of Exit card) -

How to create a key and set a limit: Open WebUI guide, Your Anthropic API key.

Or OpenRouter keys (sk-or-): same commands, no Claude model, and the whole room on one model so that totals compare (OpenRouter guide). The figures of this sheet come from Claude and do not hold for OpenRouter (its measures). Cost per pair with the default model, DeepSeek V4 Flash: about $0.03 for the terminal commands of this workshop and about $0.12 for workshops 1 to 5, played once on 2026-11-05 in a container of the image 2026.11.4 (OpenCode 2.1.4), not in a Codespace and not with a group (measures). Not counted: replay RAG and the Open WebUI chatbots. A pair that asks more questions spends more: take it as a minimum. The workshops are for adults: no young person types in the Codespace or holds a key, and OpenRouter asks for users of 18 or older. Young people only use a chatbot in Open WebUI, in their browser. When what they wrote goes through OpenRouter, leave out names and personal details, and type workshop-login --europe first (SETUP.md, Young people and OpenRouter; OpenRouter guide, Part 8).

ℹ️ Note. Who pays. Each partner organisation pays for its own sessions: its own Anthropic account, workspace and keys, on its project budget. One workspace for the room, capped at $3 × the number of pairs: 8 pairs, $24. At Antwerp, the most advanced pairs used less than $2 in a 2-hour workshop. The limit is on the workspace, not on each key, so one pair can use up the share of another: keep the Anthropic console open during the workshop and watch the spending. Not checked: if the console limit is per month, the workshops of the same month share it. Then count $3 × pairs × workshops of the month, or raise the limit before each workshop. Indicative cost per pair for the whole series: TO BE COMPLETED at the rehearsal; this workshop's measured cost is under "Cost per pair".


Timing

Time Part What you do What the room should have at the end
0:00 1. Welcome (20) Setup (SETUP.md, steps 1 to 5), workshop-start 3 --from my-work.zip, openwebui start in terminal 2. Ready for workshop 3
0:20 2. Let it search (20) workshop C, questions 3 and 7; make everyone press Ctrl+O to unfold the steps (again to come back); compare the gauge with mode B; each pair checks the answer to question 3 in corpus/text/43-… (Ctrl+F 5 h). each pair can tell which files the model opened, and has read the sentence of the source
0:40 3. Let a document in (35) Show summarize once on the big screen with Benevento. Then each pair adds Benevento, then its own page, and curates its own page. corpus/80 and 81; one summary "(human-reviewed)" per pair
1:15 4. Your own skill (30) Read summarize; each pair uses source-note (a fact of its territory, then the brush rumour: the skill must ask for a source and name summary 22); each pair writes one skill (rumour-check if no idea); restart workshop C; test the trigger. one draft summary per pair from source-note, or none after the rumour; one skill per pair that triggers on one request and not on "Hello"
1:45 5. Closing (10) Prediction, replay C, score C 3 8 11 12 (8 for workshop 4, which compares it with column D), openwebui chatbot, workshop-save, exit cards. column C for 3, 8, 11, 12; my-work.zip downloaded
1:55 Margin (5) - -

⚠️ Important. Part 3 is the longest and the most variable: a page can be refused, nearly empty, or long to check. Benevento is the fallback: a pair whose page fails curates Benevento instead, and keeps its own page for after the session.


What we measured

Without a key (2026-10-30, container of image 2026.10.33, copy of the repository, fake key):

With a key, at the rehearsal of 2026-10-20 (older images, 2026.10.4 to 2026.10.7):

What Measure
Question 3 in mode C right and sourced (the full order in corpus/text/43); gauge about 30,100 tokens, against 100,700 in mode B (measured after question 10: in mode B the whole kit is in the window, whatever the question)
Question 7 in mode C "I don't know", after a search on "campania"; gauge about 32,100 tokens
summarize of Benevento 38 s, 10 sentences, a contradiction with summary 71 pointed out without choosing
validate-summary refuses a general "I checked summary 80", validates after "I verified every row"
replay C (11 questions, before question 12) 1 min 29, $0.29

rumour-check, local test of 2026-10-22 (workshop settings, Sonnet 5.5, not in a Codespace): a rumour about a barbecue near La Rochelle used the skill and got an answer with its source ($0.13); "Hello, who are you?" used no skill ($0.04). The participants' page uses another rumour, about the clearing of brush: the barbecue near La Rochelle is the topic of test question 6, and the page must not hint at its answer.

On image 2026.10.33, with a key (2026-10-30, real Codespace, spending limit $2, about $1.0 spent in all; details in facilitator/ATELIER-3-OUTILS.md):

What Measure
First start of Claude Code theme screen, "Security notes", then the trust screen "Quick safety check" with No, exit proposed: Enter alone closes Claude Code; pairs must go down to Yes, I trust this folder (now in SETUP.md)
Write permission (Claude Code 2.1.303) Do you want to create …? / 1. Yes / 2. Yes, and switch to accept edits (…) for this session (shift+tab) / 3. No / Esc to cancel · Tab to amend
Question 3 in mode C grep in summaries/ and corpus/, then summaries/40 and corpus/text/43; complete answer; no permission asked; about 23,000 tokens; 11 s; $0.07
Question 7 in mode C "I don't know", no 2025 figure found; about 10,000 tokens; $0.02
summarize of Benevento 39 s, $0.12; 14 sentences (draft) and a table of 14 rows; row 9 notes itself "count to be checked" (the list of municipalities); a contradiction with summary 71 pointed out
validate-summary "I checked summary 80 … it is correct" with the OK? cells empty: refused, nothing changed. After filling the cells, "I filled in the check table…": the skill did not trigger. "Validate summary 80. I checked every row… My name is …": validated (status: stable, verified: … human:…, index "(human-reviewed)"). Hence the sentence of the page.
rumour-check (barbecue rumour) with -p: triggered. In a conversation, with this image: 0 out of 2 (answered directly, without the format of the skill). With the skills bundled with Claude Code switched off (disableBundledSkills): 2 out of 2. "Hello, who are you?" never triggered it.
replay C (11 questions, no question 12) 1 min 42, $0.30; judge: question 3 = 2, question 11 = 2; no question triggered rumour-check
OpenCode, replay C 3 21 s, $0.09, complete answer; searches without shell ("searched for … in summaries", "listed the files corpus/text/40*")

On image 2026.10.34, with a key (2026-10-30, real Codespace, workshop-start 3, interactive screen; bundled skills switched off):

Request Measure
Brush rumour of the page (I heard that in France, clearing the brush around your house is only a piece of advice.) 2 out of 2: ● Skill(rumour-check), then FALSE. In France, clearing brush around your house is a legal obligation, not just advice.
Barbecue rumour (I heard that you can always have a barbecue near La Rochelle.) 2 out of 2 (against 0 out of 2 on 2026.10.33): ● Skill(rumour-check), Searched for 1 pattern, read 2 files, then FALSE. You can't always have a barbecue near La Rochelle.
Hello, who are you? no skill (1 try): Hello! I'm Feurisson, a friendly hedgehog. 🦔
Context after one question 11,000 to 12,000 tokens out of 200,000 (8,000 for Hello)
Cost not shown in a conversation; estimated about $0.03 per try

Token estimates. corpus-add prints characters / 4 (Benevento: about 5,400 characters, printed as 1,346 tokens). On our documents, the gauge counts more: about 2.3 to 2.7 characters per token (mode B, counted by Anthropic on 2026-11-02: 234,096 characters, 99,621 tokens). We did not measure why. If someone asks, say that the printed number is a low estimate. The "8,000" of the page is the Hello on the interactive screen of image 2026.10.34; the 23,000 is a -p run on image 2026.10.33. So the 15,000 of the page is a subtraction across two setups, not a measure.

The expected answer to the brush rumour (FALSE: a legal obligation, summary 22) is written here and not on the participants' page, which gives only the rumour: in Part 4, Step 2, it says only that source-note names the summary that says otherwise, not which one. The model can read that page in mode C, but not this sheet, which is outside the workshop's repository. This sheet is public on the slides site, though: a participant can find the answer, so do not count on it staying secret.

⚠️ Important. Since image 2026.10.34, the skills bundled with Claude Code are switched off (disableBundledSkills), so that the skill of Part 4 triggers in a conversation. Run this workshop only with that image or a later one.

Cost per pair: about $0.55 for what was measured (replay C $0.30, questions 3 and 7 $0.09, summarize $0.12, plus the judge); the skill tests about $0.03 each (estimated, about $0.15 for five); validate-summary and the other questions of the pair: TO BE COMPLETED. Each pair's $3 share of the room's limit leaves a margin.


Leading the discussions

After Part 2. "Who decided what the model read?" Make the room press Ctrl+O and name the files. "In mode B, nobody picked; in the RAG, a search engine picked; here the model picked, and you can see each step." Question 7: the search finds documents about Campania, but not the figure; the model sees that the answer is not there.

During Part 3. Watch three moments:

  1. The permission prompts: pairs must accept with 1. Yes and never the option that stops asking.
  2. The check: pairs tend to trust the draft. Insist: every row, against the original, with Ctrl+F.
  3. A general "it's fine" is refused by validate-summary: this is on purpose.

After Part 3. The "Think about it": "Why is it a person's name in the verified line?" Expected answer: a person is responsible for what the chatbot tells a young person; the model proposes, the person decides.

After Part 4. "What goes in the system prompt, and what in a skill?" Rules for every answer in the system prompt; a procedure for one kind of request in a skill. Then: "What did the skill add to the search, and what was left for you to do?" Expected answer: the search brings the facts and shows where they are; the skill does the same task the same way every time (four verdicts, a quoted sentence); the pair opens the document and decides.

At the closing, in Open WebUI (image 2026.11.4). openwebui chatbot takes the pair's own skills (never summarize, validate-summary, source-note: they write files) and attaches them to the chatbot, with Open WebUI's skill tools switched off. Open WebUI then puts each attached skill's whole text in the system prompt, with every message, Hello included: about 204 tokens for rumour-check (cl100k_base estimate). So in Open WebUI the description triggers nothing, and the skill behaves like a part of the instructions; the model cannot read, list or write a skill. Bring it out against the exit card's third question: "when does the model open a skill?" has one answer in Claude Code and another in Open WebUI. First measured with a fake model, without network, on 2026-11-04 (facilitator/openwebui-prototypes/skills-rag/); then with real models on 2026-11-05 (qwen3:8b 3 times, Claude Sonnet 5.5 and DeepSeek V4 Flash one try each), see below.

With a small local model, the rumour got an empty answer. On 2026-11-05, qwen3:8b through Ollama on a laptop, in an Open WebUI set up as the kit (facilitator/openwebui-prototypes/real-model-local/results-thinkpad/): Hello got no verdict, but the closing's rumour (I heard that in France, clearing the brush…) got an empty bubble, 3 times out of 3, with no error. Replayed directly to Ollama: the model takes the skill for a tool and calls it; the tool does not exist, so nothing comes back. One sentence ("they are not tools: never call a skill as a tool") in the instructions, in the skill, or both, changed nothing (3 tries each). Without the YAML header of SKILL.md (name, description), the empty bubble goes away, but the model still does not search the documents: it quotes a summary named in Feurisson's instructions, or invents. The kit is not changed. With Claude Sonnet 5.5, the same chatbot (one try each, 2026-11-05, facilitator/openwebui-prototypes/real-model-local/results-claude/): the rumour got 4 searches, then "Verdict: PARTLY TRUE." with quotes from document 22 ($0.06); Hello got no verdict. With DeepSeek V4 Flash through OpenRouter, the same chatbot (one try each, 2026-11-05, Open WebUI v0.12.4 on a laptop, measures): the rumour got 5 calls to the knowledge tools (2 searches, one of which found nothing; document 22 opened twice, document 42 once), then "PARTIELLEMENT VRAI" with the true part, the false part and quotes from document 22, but in French, to a question asked in English ($0.0014); Hello got no verdict. If a pair sees an empty bubble on its skill's request, say what it is: a model that mistook the skill for a tool, not a broken chatbot.


Exit card

  1. In mode C, who decides what the model reads, and how can you check it?
  2. Why is a person's name, and not the model's, in the verified line of a summary?
  3. When does the model open a skill?

What a good answer contains: (1) the model, step by step; Ctrl+O shows every file it opened and every search; (2) a person checked it against the source and is responsible for it; (3) when a request matches the skill's description: the model knows only the names and descriptions until it opens one.


Fallbacks

Problem What to do
The page of a pair is refused, or its readable copy is empty The pair curates Benevento instead (summarize corpus/80-benevento-aib-2026.pdf).
A site refuses the download (HTTP 403) Save the document on the computer, drag it into the Codespace, then corpus-add "<file>" --url "<address>".
A pair does not get Skill(...) Check the place and the name (.claude/skills/<name>/SKILL.md, capitals); leave and start workshop C again; check that the request matches the description.
A pair has no question 12 (no save, or never written) It writes one at the start of Part 3 (test-questions.md), or curates Benevento and scores questions 3, 8 and 11 only (score C 3 8 11).
Open WebUI is not ready at the closing Skip openwebui chatbot; show your own chatbot on the big screen (built in your own run).
A pair has no save Its columns B and RAG are empty: it compares column C with a neighbouring pair.
Rows 11 and 12 empty in columns B or RAG (scored only if there was time at workshop 2) The answers came back with the save: score B 11 12 and score RAG 11 12 before comparing (no question 12: score B 11, score RAG 11).
A pair restored an old save (saved on is not the end of workshop 2) workshop-start 3 --from the newest zip, its exact name between quotes.
A pair uses OpenCode It is asked before every shell command and every file it writes: same steps, more "allow" answers. An OpenRouter key takes the same steps, judge and Open WebUI included. With this key, openwebui chatbot and a skill in Open WebUI: tried once, on a laptop, on 2026-11-05 (see Part 4 above). The skills of the terminal with the default model, in a conversation: on 2026-11-02, summarize wrote the summary and its check table; on 2026-11-05, it wrote the summary without its check table, and validate-summary and source-note wrote nothing (one try each; measures, section d, and measures). Check that the files are there.

After the workshop

Reuse with young people. Part 3 is the Data Box of WikiYouthBot: a group brings local documents, and an adult checks what the chatbot will say about them before it goes in. To build a chatbot with checked documents for a group: Open WebUI guide, Step 1 - Gather your documents, then Part 3 - Test before you open. The pair's skill (for example rumour-check, for I Heard a Rumor) reaches a group's chatbot with the file that chatbot-export makes at workshop 5: openwebui import brings it back attached to the chatbot (Open WebUI guide, Part 6, Step 2 - Import a chatbot), and openwebui share open gives it to the participants. What an attached skill does in Open WebUI: Step 3 - Create the chatbot, paragraph "Skills, if you attach one" (attaching one by hand in the editor was not tried). Read each SKILL.md before you open it to young people.