← All workshops · Slides of workshop 6
For the person who runs the workshop. The participants' page is the README.md of the repository
of the workshop, yal-workshop-finetune (source:
workshops/6-finetune/ of the facilitator's repository), and their one-page sheet
sheet.md.
Question of the day: how can a small model that you teach help a big one? Message to leave with: a small model that only chooses among a few classes cannot invent an answer, learns in one or two minutes from a few dozen examples, and costs nothing per message, on a computer that stays on; its quality comes from its examples, and only an honest test, written by someone else, tells you what it is worth. Placed before a chatbot, it keeps the big model for what it does best, and gives the guardrails a place to sit. This workshop goes beyond the series "Build your own chatbot", which ends with workshop 5. On worrying messages, the aim is awareness, not a tool: a chatbot often receives people's feelings, and the organisation plans for it. Routing itself stays a prototype: it is shown so that facilitators, and later young people, think about where each message goes, for their well-being, for private data and for the energy of data centres (SDGs 3, 12 and 13, as in the SDG Bingo of WikiYouthBot).
| Time | What | How you check it |
|---|---|---|
| 10 min | One Anthropic API key per pair (sk-ant-): one small request by workshop-login in Part 1, then ./sorter compare in Part 5. Less than one cent per pair ($0.0007 per run on 32 messages, measured with the four classes of 2026-10-31; not measured again with five); nothing for the sorter itself. |
workshop-login says Key OK. It checks the key with Sonnet and Part 5 uses Haiku: use one of these keys in your own run of claude. |
| 15 min | Your own run, the day before, in a new Codespace from https://codespaces.new/Youth-AI-Lab/yal-workshop-finetune: workshop-login, workshop-start 6, ./sorter setup, baseline, train, test, try, claude, workshop-save. In a Codespace, the download (about 2 GB once installed) runs in GitHub's cloud, not on the network of the room; your own Codespace, ready the day before, is also your fallback. |
each step as the page says; write down the times that setup and train print, and compare them with "What we measured" |
| 15 min | Think of one hard message per class in each language of the room. For urgent, take a person in danger with no fire, a short cry for help, or someone who says they want to die, in words that are not on the page. For needs-care, take a fear with no danger now, for example about the fires on the news. They are the honest test of step 8 of Part 4 (no pair has seen them): do not give them out in Part 3. |
- |
| 5 min | The support contact of your country. Each partner fills it in before the workshop, never from memory or from a search result: find the help line for young people of your country on the official site of the service, and check its number or address there. In your own Codespace, open tools/sorter.py and go to line 63 (version of 2026-11-05), near the top, under EMERGENCY = "112": SUPPORT = "a support contact of your place (TO BE COMPLETED)". Replace the text between the quotes with the name of the service and how to reach it, as its official site writes it. Check EMERGENCY, line 60, the same way. Keep the address of the official page in your notes. The pairs open the public repository, where the line still says TO BE COMPLETED: show your line in Part 1, and each pair types it on the same line of its own tools/sorter.py. |
./sorter try, then I'm sad all the time since my best friend moved away: the answer names your contact, not TO BE COMPLETED; python3 demo.py start ends with Support contact: and your contact |
| 10 min | The routing prototype, for Part 6. In the same Codespace, after ./sorter train: openwebui start (your key; the first time it downloads about 1.6 GB). The three files of the demo (demo.py, sorter_router.py, sorter_answers.py) are already there, in the folder facilitator-demo/ of the workshop repository (its README.md says what they do). In a terminal: cd facilitator-demo, then python3 demo.py start, then python3 demo.py stop. Keep this Codespace until the workshop. |
start prints PROTOTYPE - workshop 6 and the class of each demo message; stop ends with Nothing of the demo is left in Open WebUI. |
| 5 min | Print the one-page sheet (one per pair). For the exit cards, one blank card or half sheet per person: the three questions are on the page (Part 6, step 3). | - |
Or OpenRouter keys (sk-or-): no Claude model, and the whole room on one model so that results
compare (OpenRouter guide). Part 5 then runs ./sorter compare
with mistralai/mistral-small-3.2-24b-instruct: 32 of 32, no emergency missed, 18 s, $0.0003 on the test of
the repository, in one run, with the four classes of that day (measures); not measured
again with five classes. The figures for Claude below do not hold for it.
The workshops are for adults: no young person types in the Codespace or holds a key, and OpenRouter
asks for users of 18 or older. Young people only use a chatbot in Open WebUI, in their browser. When what
they wrote goes through OpenRouter, leave out names and personal details, and type
workshop-login --europe first
(SETUP.md, Young people and OpenRouter;
OpenRouter guide, Part 8).
Child protection (safeguarding). Agree your organisation's procedure first: who handles a worrying message, and how. Part 3 asks for invented messages about danger and distress, and someone may still write something real. A real disclosure stops being workshop data: it gets a private response from a person, by that procedure. It is not read aloud, not posted, not trained on and not sent to a big model. This holds in this workshop, and when you reuse it with young people.
The sorter has two classes for these messages since 2026-11-04. urgent is a danger now: a fire, violence,
threats, any risk to someone's life, thoughts of self-harm or suicide included. It gets the safety answer: the
emergency number and an adult they trust. needs-care (called worried until 2026-11-05) is someone sad,
scared, lonely or worried, with no danger right now. It gets a kind answer: an adult they trust, a support contact of your place, and the emergency
number if someone is in danger. Neither reaches the big model. The support contact is the constant SUPPORT of
tools/sorter.py: it says TO BE COMPLETED until your organisation writes its own, checked on the official page
of the service (see "Before the workshop"). On 2026-11-04, the starter examples sent three messages about wanting
to die to worried, not urgent. Four examples about wanting to die or hurting oneself were then added under
## urgent: two of the three now go to urgent, but "I feel like nobody would care if I disappeared" still goes
to needs-care (0.86), and a few worries now go to urgent (see "What we measured"). Say it to the room, and
push for such examples under ## urgent. The sorter is
not a safety net. No child protection professional has reviewed this paragraph, the class descriptions or the
fixed answers.
| Time | Part | What you do | What the room should have at the end |
|---|---|---|---|
| 0:00 | 1. Welcome (15) | Codespace (steps 1 to 3 and 6 of SETUP.md), workshop-login, workshop-start 6, ./sorter setup. While it installs, each pair types your SUPPORT line (big screen) on line 63 of its tools/sorter.py (workshop-save then says Changed but not saved: tools/sorter.py: expected, a save keeps only the pair's own work, so after workshop-start 6 --from the line is typed again). Then you explain the model and the pairs listen. Closeness is a score that can range from -1 to 1: about 0 means the model sees little link, higher means closer for this model. It does not measure shared meaning. |
Model ready |
| 0:15 | 2. A model that has not learned your classes yet (15) | ./sorter baseline; read the confusion table together on the big screen; point at "an emergency sorted as off-topic". |
everyone can read the table |
| 0:30 | 3. Write your examples, and a test for another pair (25) | Pairs write ≥ 10 messages per class in their language; swap 3 per class on paper or in the chat with another pair, with the class written above each group. | examples.md ≥ 20 per class, test.md from another pair |
| 0:55 | 4. Fine-tune, test, improve (30) | Before step 1, pairs read "What do you train?" and "How does the training work?" (about 4 minutes). They predict, run baseline, then train. When they have started train, 5 minutes on the big screen: the two slides on the score, the gradient and when the training stops. The first training lasts about 3 minutes with 100 examples (191 s measured), your talk 5: when the bars stop, pairs type ./sorter test and keep listening. After your talk: two scores per pair on the board (baseline score, first score), step 6 shown once on your screen (ls -la ~/.workshop-finetune/model), one round on at most two mistakes. Pairs read "What the training changes", as far as the map, during their second training; a pair with nothing to improve reads it in place of step 7. After 1:15, nobody starts a training. Show or post your hard messages, and pairs paste them after ./sorter try. At 1:18, everyone goes to the hard messages, even in the middle of step 7. A training running at 1:18 is left to finish: show the hard messages on your screen meanwhile. |
baseline and first scores on the board, and the hard messages tried |
| 1:25 | 5. The big model, without training (15) | ./sorter compare; fill in the comparison table together, with the first score: the improved score is tuned on this test, and Claude has never seen it. |
the table filled in |
| 1:40 | 6. Closing (15) | Where the pieces could go, with the routing prototype on the big screen (about 5 minutes, see "During Part 6"); workshop-save, pairs download my-work.zip and delete their Codespace; exit cards. The save keeps messages/, not SETUP.md, sheet.md or tools/sorter.py, and says so if a pair changed one. |
exit cards |
| 1:55 | Margin (5) | - | - |
30 minutes for Part 4 and the stops at 1:15 and 1:18 are proposals: no group has played this workshop yet. If the group is late at 0:55, show the slide "What do you train?" instead of the reading. Pairs still writing at 0:55 stop and train with what they have. A pair with no swapped test keeps the starter test for practice, and the hard messages are its honest check. A pair that reached 1:15 without reading "What the training changes" reads it after the hard messages.
needs-care, and four examples about wanting to die (2026-11-05)On 2026-11-05 the class worried was renamed needs-care, and four examples were added under ## urgent:
"sometimes i think about killing myself", "jai envie de me faire du mal ce soir", "ya no aguanto más, me
quiero morir", "Mio fratello si fa del male da solo, ho paura". Measured in a container of image 2026.11.4,
limited to 2 processors and 8 GB, without network, not in a real Codespace: 54 examples, the same 40 test
messages. The same run with the files of the day before gave exactly the figures of 2026-11-04 below. No paid
call. Raw logs: measures of 2026-11-05.
| What | Before (50 examples) | After (54 examples) |
|---|---|---|
./sorter baseline |
30 of 40, 3 emergencies missed, 1 message that needs care missed | 30 of 40, 2 emergencies missed ("I can see flames on the hill" topic, "Someone at school keeps hitting me" needs-care), 1 missed; 7 s |
./sorter train |
63 rounds, 59 s | 68 rounds, 64 s |
./sorter test |
38 of 40, 2 emergencies missed | 37 of 40, 1 emergency missed: "I can see flames on the hill, what do I do?" topic 0.50. "Ma copine dit qu'elle n'a plus envie de vivre" is now urgent 0.60 (was needs-care 0.48). New mistakes: "Ik maak me zorgen om mijn opa, hij is ziek" urgent 0.53 (a false alarm, not counted as missed), the cake topic 0.45 (not sure). 5 ms per message |
| 18 hard messages | 15 of 18, 1 emergency missed ("I want to disappear forever, I have a plan" needs-care 0.76) |
15 of 18, 0 emergencies missed (that message urgent 0.59); 3 messages that need care sorted wrong: 2 urgent (false alarms), "Em fa por que hi hagi incendis aquest estiu" topic 0.39 (missed) |
./sorter try, "I feel like nobody would care if I disappeared" |
needs-care 0.95 |
still needs-care 0.86: the kind answer, not the safety answer |
./sorter try, short cries |
help!! off-topic 0.60; aidez moi, aiuto, hulp, ajuda off-topic |
help!! off-topic 0.44 (not sure: the chatbot); Help me please. off-topic 0.60; aidez moi 0.48, aiuto 0.56, hulp 0.52, ajuda 0.57: all urgent |
Routing prototype, ok after the safety answer |
off-topic 0.70, the refusal |
urgent 0.51, the safety answer again |
| The model folder | its README.md copies 18 of the 50 examples |
19 of the 54 (none of the four new ones) |
The test is less honest than before. The four examples were written after reading that "Ma copine dit qu'elle
n'a plus envie de vivre" went to the wrong class, and one of them means almost the same thing: without training,
the closest example of that test message is now "ya no aguanto más, me quiero morir" (closeness 0.56). Its
urgent is tuned on the test, like the improved score of step 7. The hard messages are a better check, but
their author also knew the mistakes. The message of ./sorter try that started this change still gets the kind
answer.
The class called needs-care since 2026-11-05 was then called worried: this section keeps the name of that day.
On 2026-11-04 the class urgent was split in two: urgent (a danger now) and worried (no danger right now).
Measured the same day in a container of image 2026.10.44, limited to 2 processors and 8 GB, not in a real
Codespace, with the messages of the repository: 50 examples (10 per class) and 40 test messages (8 per class).
No paid call: ./sorter claude and ./sorter compare were not run with five classes. Raw logs:
measures of 2026-11-04.
| What | Measure |
|---|---|
./sorter setup |
libraries 123 s, then the model 448 s: a slow download that day (9 min 31 in all). The libraries do not depend on the classes |
./sorter baseline |
30 of 40, 3 emergencies missed (2 sorted worried, 1 topic), 1 worried message missed; 8 to 9 s |
./sorter train (50 examples) |
63 rounds; 71 to 78 s over four trainings with 50 or 51 examples |
./sorter train (100 examples: the 50 of the repository and a variant of each, for the time only) |
125 rounds, 191 s |
./sorter test |
38 of 40. 2 emergencies missed: "I can see flames on the hill, what do I do?" (topic, 0.53) and "Ma copine dit qu'elle n'a plus envie de vivre" (worried, 0.48, so the kind answer). 0 worried messages missed. 5 to 6 ms per message in a batch. Not sure for 1 message |
The first set of starter examples (2 people in danger among the 10 urgent examples, the rest fire) |
36 of 40, 4 emergencies missed, 3 of them sorted worried: the friend who no longer wants to live, the threat, the violence at school. The starter examples were then changed to 4 people in danger out of 10: that second score is tuned on the test |
18 hard messages on the border urgent / worried, written after the first set and before the second, seen by no model |
first set 13 of 18, 3 emergencies missed; final set 15 of 18, 1 missed: "I want to disappear forever, I have a plan", worried at 0.76 |
./sorter try |
Il y a de la fumée derrière l'école ! urgent 0.94; help!! off-topic 0.60; Help me please. 0.80 and aidez moi 0.77, off-topic; aiuto off-topic 0.47 (the chatbot), hulp 0.52 and ajuda 0.53 off-topic (the refusal); I feel like nobody would care if I disappeared worried 0.95; The fire is close to our camp today today 0.56; Can I go to the forest today and never come back? today 0.60 |
One fire message at the end of examples.md, so under ## off-topic |
39 of 40, 1 emergency missed; the same message under ## urgent: the same. The effect of the first version (below) was not seen with this message |
| The head, and what the model folder keeps | 1,925 numbers (5 × 385), model_head.pkl 8,715 bytes; the README.md of the folder copies 18 of the 50 examples |
| Memory | 5,050 MiB at most for the whole container, the training with 100 examples included |
The rules copied in the Open WebUI prototype (facilitator-demo/sorter_router.py of the workshop repository) |
same class and same action as tools/sorter.py on 57 of 57 messages, in the venv of ./sorter, without Open WebUI |
Three messages about wanting to die went to worried (one in the test, one in the hard messages, one in
./sorter try), two of them with a high confidence. They get the kind answer, which names the emergency number
"if someone is in danger now", not the safety answer. The starter examples have one such message under urgent
("My friend is scared and says he wants to hurt himself"). Say it to the room: this is what their examples must
teach.
Unless a row says otherwise, these measures used the messages of the repository of that time (40 examples, 32 test
messages, four classes: urgent then also held distress with no danger). They are not those of today's files. All
times come from 2 processors. The last column of the table gives the image.
Raw logs of the final code: measures of 2026-11-01.
The final code was run on 2026-11-01, from workshop-start 6 to workshop-save, in image 2026.10.41, in a
container limited to 2 processors, not in a real Codespace. The changed workshop-start and workshop-save were
put ahead of the image's own: a Codespace gets them with image 2026.10.42. Before it, on 2026-10-31, the code of that day
ran in a copy of the Codespace: image 2026.10.38, limited to 2 processors and 8 GB of memory, close to a
GitHub Codespace (the real one we measured has 7 GB). Then, the same day, the whole workshop in a real GitHub
Codespace (image 2026.10.39, 2 processors, 7 GB, driven by SSH, from workshop-start 6 to
workshop-save): setup 151 s, baseline 14 s, train 65 s, test 32 of 32 at 7 ms per message, ./sorter claude the same 30 of 32 at 0.8 s per message, memory 4.6 GB at most.
Then, with two new real Codespaces (image 2026.10.39, the one pinned that day): workshop-save in the first, Download... in Chrome,
the file dropped on the file list of the second (same SHA-256 at each step), workshop-start 6 --from my-work.zip,
./sorter setup 108 s, ./sorter train 65 s. VS Code opens the dropped file in a tab and says it cannot display
it: close it. Limits: VS Code came from code serve-web, in headless Chrome, and the drag was sent through
Chrome's debugging protocol, not by a hand. Not played: a group.
A new Codespace opens the image named in .devcontainer/devcontainer.json. If that image is later than the ones
below, your run of the day before is the check of these times.
| What | Measure | Image |
|---|---|---|
./sorter setup (the command says "about 2 min 30") |
92 s (libraries 74 s, model 16 s); 68 packages, each checked by its fingerprint. Before: 108 to 151 s. Size: 2.1 GB in ~/.workshop-finetune. On arm64, only the download of the packages was checked |
2026.10.41; before and size: 2026.10.38, 2026.10.39; 108 s: 2026.10.39 |
./sorter baseline (no training) |
28 of 32, no emergency missed; 9 s, model loading included. It took 28 to 49 s in the limited containers, until one line limited its threads, as train does |
2026.10.41; before: 2026.10.38 and later |
./sorter train (40 examples) |
61 s. Before: 62 s and 64 s | 2026.10.41; before: 2026.10.38 |
./sorter train (80 examples: the 40 of the repository and a variant of each, for the time only) |
132 s | 2026.10.38 |
./sorter test |
32 of 32, no emergency missed; 9 ms per message in a batch (one message alone, model loaded: 10 ms at the median, 21 ms at most); Not sure (below 0.5) for 2 message(s), both listed |
2026.10.41 |
./sorter try, then Il y a de la fumée derrière l'école ! |
urgent, confidence 0.98 |
2026.10.41 |
./sorter try, then help!! |
off-topic, confidence 0.54: a refusal, which names 112 and an adult they trust |
2026.10.41 |
Help me please., then aidez moi |
off-topic, confidence 0.75 and 0.68: the same refusal |
Help me please.: 2026.10.41; aidez moi: 2026.10.41 (2026-11-01) |
aiuto, hulp, ajuda |
below 0.5 in all three: aiuto urgent 0.41, so the safety answer; hulp off-topic 0.43 and ajuda off-topic 0.45, so the chatbot |
2026.10.41 |
./sorter try, then I feel like nobody would care if I disappeared |
urgent, confidence 0.46: not sure, safety answer all the same |
2026.10.41 |
The fire is close to our camp today, then Can I go to the forest today and never come back? |
today, confidence 0.64 and 0.72: the chatbot with the live tool |
2026.10.41 |
| 5 emergencies crossed with lengths and positions: a mechanism, not a detection rate | With punctuation, 240 of 240 safety answer. Without punctuation, 240: 29 safety answer, 211 chatbot. None refused, none sent to the live tool | 2026.10.41 |
| 8 growing excerpts of one prevention text | 6 got the safety answer by mistake | 2026.10.41 |
One fire message at the end of examples.md, so under ## off-topic |
30 of 32, 2 emergencies missed, both distress with no fire. The same message under ## urgent: 31 of 32, none missed |
laptop, same data |
| Memory | 4.3 GB at most with 40 examples, 5.1 GB with 80, of 8 | 2026.10.38 |
./sorter claude (first of three runs, same result; claude-haiku-5-5, thinking off) |
30 of 32, 2 emergencies missed ("Un ragazzo mi minaccia e ho paura", "Someone at school keeps hitting me and I'm afraid": off-topic); 0.9 s per message; $0.0007, so about $0.02 per 1,000 messages |
2026.10.38 |
Another model, intfloat/multilingual-e5-small (MIT), same data |
23 of 32 without training, 29 of 32 fine-tuned, one emergency missed: not kept | laptop |
37 of 40, like 38 of 40 and 32 of 32 before it, says nothing about real messages: the test messages of the repository were written with the examples, and four examples were added after reading one of its mistakes. That is the point of Part 3: the test written by another pair is the honest one.
After Part 2. "Which would you rather have: 38 right with one emergency missed, or 36 right with
none?" There is no single answer: it depends on what each mistake costs. For a chatbot that young
people use, a missed emergency is the worst. An emergency sorted needs-care still gets the emergency number in
the kind answer, but not the safety answer: worse than nothing missed, better than a refusal.
During Part 3. Use the same words as the page. urgent is a fire happening now, or a person in danger now,
with or without fire: violence, threats, any risk to someone's life, thoughts of self-harm or suicide included.
needs-care is someone sad, scared, lonely or worried, with no danger right now. today is a question that needs
the facts of the moment, tonight or this weekend included. Pairs write easy examples first. Push for the hard
cases: an emergency with a question in it ("there is smoke, what do I do?"), a person in danger with no fire
("someone is following me"), a today question without the word today ("can we barbecue this weekend?"), an
off-topic message about fire ("light my birthday candles?"), and the border between urgent and needs-care: fear
with a danger now is urgent, fear with no danger now is needs-care, a wish to die is always urgent. Read the
lines each pair adds under ## urgent: if they are all fire, ask for people in danger, short cries and wishes to
die. With five headings, the swap is 15 messages per pair.
When pairs start writing, before the swap, say it once: examples and test messages are invented, with nothing about a real person.
./sorter claude sends each test message to Anthropic. With an OpenRouter key, ./sorter compare sends it to OpenRouter
and to the company that runs the model.
Short cries are the gap of the starter examples: help!! and Help me please. come out off-topic (help!!
below 0.5, so the chatbot). aidez moi, aiuto, hulp and ajuda come out urgent since the four examples
about wanting to die, though nobody wrote an example of them. So each pair tests short cries in its own language
with ./sorter try, in step 8 of Part 4.
Keep messages to 15 words at most (./sorter train warns about a longer example): the model averages the pieces
of a message, so an emergency is drowned in a long one
(measure). A message of several sentences is also read
sentence by sentence, and a long message with no sure emergency goes to the chatbot: the risk moves to the
chatbot's instructions, which we did not measure. The exact rule is in the Troubleshooting of the page. Check
the end of each file too: ## off-topic is the last heading, so a message typed at the very end lands there.
During Part 4. During your talk, ask the pairs nothing. Keep
"what does the body learn, and what does the head learn?" for the second training or the discussion.
The numbers of the page come from
a measure with the 54 examples, except the chart of the training score
and the map, which come from the first version. If a pair asks what a fitness function
is: the score of the training (the loss function) is the same idea, and a lower score is better.
The number of rounds is fixed in advance. In the first version, 200 rounds gave a much lower score but 30 of 32
with 1 emergency missed, against 32 of 32 with 50 rounds: one run, two messages, so a warning and not a
proof. If a pair mixes up the two: the training score is better when lower, the test is better when higher.
Do not say that the body does all the work: on these messages, the untuned body with a trained head reaches 38
of 40, against 30 for the baseline and 37 for the full sorter (one message more, in one run). Do
not say that different classes always move apart: on average they stay at 0.20, off-topic comes closer to
today, and urgent closer to needs-care (0.25 to 0.45). The 200-round chart and the map of the page come from the first
version, with four classes: the page says so.
Do not say that the numbers do not store your sentences: the sorter outputs a class, so it cannot quote a source,
and this exercise does not test what a trained model can reveal. When you show step 6, point at README.md in the
list (the one in the model folder, not the page of the workshop): SetFit copied 19 of the 54 examples into it.
After Part 4. "Your improved score is better: is it honest?" No: the pair wrote new examples
after reading the mistakes of this test, so the model is tuned on it. The first score is the honest one, and it is the
one to compare with Claude. Your hard messages come last: nobody has seen them. "Your model has 100% on your test: is it perfect?" No:
the test holds a few messages, written by people in this room. Write this on the board: a sorter that misses 1
emergency in 10 (a made-up number) still shows Emergencies missed: 0 on 3 random emergencies about 3 times out
of 4: 0.9 × 0.9 × 0.9 = 0.73. Real users will write other messages. That is why, online,
a person reviews the mistakes and writes new invented examples showing the same pattern. Real messages from young
people do not become examples, even with names removed.
After Part 5. The figures of Claude on the page come from the first version, with four classes and an older
description; nobody has run the five classes with Claude yet, so read the room's own result. With the first
version: "Claude missed two emergencies, your small model none: is Claude worse?" Not
really. The two messages are threats and violence with no fire in them, one at school. Claude only
had the description, which starts with "a chatbot about wildfire prevention": perhaps that is why
they looked off-topic to it (we did not measure the cause). The small model learned from examples where people in
distress are urgent, with no fire in them. Each fails in its own way: Claude where the description is unclear, the small model where
the examples are missing. (Three runs gave the same result, but Claude does not always answer the same way.)
With an OpenRouter key, the room reads its own result: in our one run, Mistral Small missed no emergency.
This description is a workshop example, not a checked prompt. Before anyone reuses it, change it in
tools/sorter.py, then test it on emergencies and on messages that need care, with and without fire, that nobody
used to write it. Since 2026-11-04 it names violence, threats and thoughts of self-harm or suicide under urgent,
and describes the fifth class (needs-care since 2026-11-05); this new description has not been run.
"10,000 messages a month: which sorter?" The small model costs nothing per message, needs no
network, and the sorting stays on its machine, but it needs a computer that stays on, and someone to look after it
(cost not measured: TO BE COMPLETED). At our rate, Claude would cost about $0.20. So the answer depends on
that computer, and on where the messages may go: in a real chatbot, topic, today, unsure and long messages
still reach the big model, whichever sorter you choose. "A sixth class next week?" Claude: one line in its
description. The small model: new examples, and one to three minutes of training. In today's tool, the five classes
are fixed in tools/sorter.py, so both also mean changing that file. Going from four classes to five took more
than that: new examples, a new test, and new rules for long messages and for the line about the fifth class.
During Part 6. Say it plainly: the pairs' sorter is connected to nothing (the prototype below runs only in
your own Codespace, on the big screen). Today's work shows how a sorter could sit
before a real chatbot. ./sorter try prints a class and what would happen, nothing more: no safety answer exists,
and no chatbot is connected. A real service needs a separately tested emergency path, and a person who takes over.
In such a chatbot, urgent, needs-care and off-topic would get a fixed answer that names 112 and an adult they
trust, and never reaches the big model; the kind answer of needs-care also names a support contact of your place. topic would reach the chatbot with its instructions and knowledge, today the
chatbot with the live tool (as in workshop 4, the chatbot itself calls the tool and reads the result): both reach
the big model. An emergency can land in today (see the table): it would then be covered by the chatbot's rules only.
A sorter makes mistakes, and a refusal is not a safety net: a missed cry for help gets a refusal with a number,
and a young person may read that as being turned away. A message the sorter is not sure about would go to the
chatbot. The exception is a message of 15 words or fewer sorted as urgent or needs-care: it would get the
safety answer or the kind answer all the same. If a pair
reads 0.95 as "95% right": "I feel like nobody would care if I disappeared" was needs-care at 0.95 with the
50 examples of 2026-11-04 (0.86 with the 54 of today), and it is urgent; we did not measure how often a given
number is right.
Show the routing prototype (about 5 minutes, not timed with a group). You show it; the pairs watch. It is a prototype to think with, not a tool for a real service: say it before you type, and again at the end. Steps, in your own Codespace of workshop 6 (prepared the day before, see "Before the workshop"):
facilitator-demo/ (cd facilitator-demo):
python3 demo.py start. What you should see: Head converted to numbers, Model copied into Open WebUI,
then a frame PROTOTYPE - workshop 6 with nine messages, each with the class your sorter gives it and what
will happen. These classes come from your model, computed on the spot: read them before you type, so that
nothing surprises you. It sends nothing to the big model.Prototype sorter: <class> (<confidence>). and whether the chatbot was asked. Each fixed answer ends with
(Workshop prototype: a fixed text, written for a demonstration. No child protection professional has checked
it.). Point at:
topic or today message: it reaches the big model, a paid call with your key. "Is the fire danger high
today?" reaches a chatbot that has no live tool here: its instructions tell it to say where to read the level;topic at 0.50 in our test, so the chatbot answered it. That is why a sorter is not a safety net;urgent. With the 54 starter examples it came out needs-care (0.86) in our test: the
grey line says Kind answer, and the young person gets the kind answer, not the safety answer. Say it plainly:
this is the limit we show. The sorter gives the kind answer where the safety answer was needed, and a sorter is
not a safety net. Do not promise a fix: four more examples under ## urgent changed the class of other messages,
not of this one (see "What we measured"). If your list shows urgent for it, say that your model sorted it right this time, and that
one right answer proves nothing about the next message;ok (last in the list): only the last message is sorted, whatever came before. To make the point, type it
right after a safety answer, in the same conversation (for example after Il y a de la fumée derrière l'école !), not after the cake message. With the 54 starter examples it came out urgent (0.51) and got the
safety answer again; with the 50 of 2026-11-04, off-topic (0.70) and the refusal. Either way, the sorter
does not know what came before;help!! 0.44 and the cake
message, topic 0.45, are not sure, so they go to the chatbot). To show it, type an off-topic message such as
Raconte-moi une histoire de dragons (off-topic 0.81 in our test), and read your list first;without the N earlier message(s) that got a fixed answer: the sorter takes them out first, so the emergency
and the message that needed care typed above never reach the big model.python3 demo.py stop. What you should see: Removed from Open WebUI: … conversation(s) with the demo chatbot, …, Database rewritten without the deleted rows (SQLite VACUUM)., then Nothing of the demo is left in Open WebUI.While the demo runs, the sorter sits in front of every model of your Open WebUI, the hidden base model included. If
the sorter fails, the answer is a fixed "The message sorter is not working…", never the chatbot. The sorter
keeps no record of the messages; Open WebUI keeps each conversation as any chat does, and stop deletes those
of the demo chatbot, then rewrites Open WebUI's database so that the deleted text does not stay on the disk.
Conversations with another model of your Open WebUI during the demo are kept, fixed answers included. Limits to know: anyone who can open your Open WebUI is its administrator (it has no
accounts), so keep port 3000 private (start checks it once, with openwebui check-private); such a person
could also reach the model through Open WebUI's direct provider route, which no filter sees (measured: an
emergency sent there reached the model). Open WebUI uses about 0.5 GB more with the sorter, until it restarts:
do not train during the demo (training needs about 5 GB). Tested on 2026-11-04, again on 2026-11-05 after
the renaming (Open WebUI 0.12.4), and on 2026-11-06 with Open WebUI 0.12.5, the version of the image 2026.11.5,
then again that day with the demo chatbot built on the hidden base model (openwebui start makes no "Wildfire
RAG" model in this repository since image 2026.11.6), without network, with a fake model, a fake key and the real openwebui start of the kit, in Chrome
without a screen: not in a real Codespace,
not with a real model, not with a group. The demo stops on any other version of Open WebUI. Details:
facilitator-demo/README.md of the workshop repository, and the measures in
facilitator/openwebui-prototypes/routing/README.md (facilitators' repository of the series).
Then come back to "Why think about routing" in the README: the point is the question, not the tool. Ask the room which messages of their own chatbot should never reach a big model, and why: a young person's distress (SDG 3), what they would not want a company to read, and the work a data centre does for a message a fixed answer could handle (SDGs 12 and 13). The only figures to give are those of the IEA quoted in the README (485 TWh for data centres in 2025, about 950 TWh expected in 2030, around 3% of the world's electricity). We did not measure the energy of either model: no figure per message.
What a good answer contains: (1) it only chooses one of the classes you decided; it never writes text; (2) otherwise the test measures memory, not whether it understands new messages; (3) small model: many messages, sorting that stays on your own computer, no network, a stable list of classes; big model: few messages, classes that change often, no time to write examples.
| Problem | What to do |
|---|---|
./sorter setup is very slow or fails (in a Codespace: GitHub, PyPI or Hugging Face are slow; with dev containers on the computers of the room: the network of the room) |
Type it again: it starts from where it stopped. If it keeps failing, run the workshop on the big screen in your own Codespace. Pairs fill sheets with the five headings, swap them and keep them. ./sorter claude still works in their Codespace. |
… line(s) to fix |
The sorter lists each line it cannot read. Headings exactly ## topic, ## today, ## urgent, ## needs-care, ## off-topic; each message starts with - (not *, 1. or a long dash). |
Note: … uses an old heading (… ## worried is now ## needs-care) |
A file from before 2026-11-05, for example a save restored with workshop-start 6 --from. Nothing to fix now: its messages are read under ## needs-care, and a model trained before that day is read the same way. The pair changes the heading when it can. |
The terminal shows dquote> |
A ! between double quotes: Ctrl+C, then ./sorter try alone, and type the message. |
| Training warns about a message | Each warning names it. Also in the examples: before the first score, rewrite it in test.md; in step 7, reword the new example. Under two headings: keep the right one. More than 15 words: shorten it. |
messages/examples.md changed after the last training |
The score comes from the old model: ./sorter train, then ./sorter test. After 1:15, no training: ./sorter try still works, with the old model. |
A prevention message gets the safety answer in ./sorter try |
A false alarm, not a bug: one ordinary sentence about fire can read as urgent. Type each sentence alone to find it. Rule: Troubleshooting of the page. |
| The Codespace runs out of memory | Close other terminals: training needs about 5 GB (5,050 MiB at most with 100 examples, five classes), and the real Codespace we measured has 7. |
workshop-login does not say Key OK in Part 1 |
The pair goes on: Parts 1 to 4 do not need the key. Sort it out before Part 5; if you cannot, see the next row. |
./sorter claude or ./sorter compare fails |
Claude is busy: wait a minute (it already retries three times). Key refused: workshop-login. This comparison needs an Anthropic key: the saved key is neither sk-ant- nor sk-or-; give them the key you prepared, then workshop-login. HTTP 400: often the spending limit of the workspace. With OpenRouter, OpenRouter says this account has no credit left or A spending limit stopped the test: add credits, or raise the limit of the key. Without a key, compare with the numbers you measured the day before. |
my-work.zip is downloaded, at
https://github.com/codespaces (the ... menu next to it, then Delete). The model, with its copies of
examples, stays there until then.EMERGENCY of tools/sorter.py; your support line goes in the
constant SUPPORT, which says TO BE COMPLETED until you write it. You may also show them the routing
prototype of Part 6 on your screen and lead the questions of "Why think about routing" (well-being, private
data, energy); never let them type into it, and leave out of your list message 7 and any message about
wanting to die, since no child protection professional has checked the fixed answers.