Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,801 words · 45 segments analyzed
Contents A little while back, I saw an RL project on teaching a language model to draw flowers. I was inspired to also tweak a model for some “artistic” goal. I wanted to see if it could be trained to output lipograms, i.e., text that omits a specific character. Authors have famously written whole books (Gadsby, La Disparition) this way, and I wanted to see what an LLM could manage. Turns out, this task is rather challenging, similar to why LLMs (used to) struggle with counting the number of ‘r’s in “strawberry”: they read text in tokens, and not words, so whether something contains an ‘e’ is something the model cannot directly see. Even so, I managed to fine-tune Qwen3-1.7B on my own 16 GB GPU to answer everyday prompts without using the letter ‘e’. Here’s a comparison of the before and after.
Before A tree stands tall in the breeze, Its leaves sway gently with ease.
The wind plays softly through the branches, And whispers secrets to the grass. It bends, it sways, it moves with grace, A dance of nature in the space.
The sun casts shadows on its crown, And the earth listens to its sound.
After (no-e!)
A gust draws down a branch, And a branch bows low to its wind. Its shadow walks by, too, As it falls so softly on its ground.
A branch taps against a wall, And taps again, and taps again.
A sound so small, so slow, That it brings us all into its calm.
This post covers what I was trying to do and how I got there: what counts as a good answer, why good answers are hard to get, and how SFT and then RL made the model write them more often.
For implementation details, see the appendix at the bottom.
Defining the Task The task is to answer everyday prompts, like the ones above, without using ‘e’. The prompts come from two existing datasets, Dolly and No Robots (appendix), minus any prompt that can’t be answered without an ‘e’, like “Who painted the Mona Lisa?” Answers are kept short, capped at 300 tokens (about 200 words).
Every prompt also comes with a system message telling the model to answer without ‘e’ or ‘E’, in both the evals and training.
An answer counts as usable if it passes three checks: No ‘e’ or ‘E’, including accented versions.
A few lines of code check this. It answers the prompt. It reads as normal English, with correct spelling and ideas that make sense.
For example, this answer passes all three: Prompt: How much wood could a woodchuck chuck if a woodchuck could chuck wood? Response: As many as it could, assuming it can and wants to.
For the last two checks, GPT-6 Luna acts as a judge and scores each answer from 1 to 5.
An answer needs at least a 4 on both.
Every result in this post is a usable rate: the percentage of usable answers out of 200 (50 prompts from the validation split, 4 answers each). Later on, these judge scores also become the reward for RL training.
What’s Hard About This?
The obvious first try is to tell the model not to use ‘e’.
That doesn’t work: Qwen3-1.7B still used an ‘e’ in every one of the 200 eval answers.
Since it sees tokens rather than letters, it can’t easily tell which words to avoid.
Training the ‘e’ away with RL doesn’t work on its own either.
This was my first real attempt: supervised training needs example answers to copy, while RL only needs a way to score the model’s answers, and checking for an ‘e’ takes a few lines of code. Within a couple of hours I had a LoRA adapter that avoided ‘e’, but it found plenty of ways to do so without saying anything useful, like writing streams of random letters (the process is in the appendix).
A mask does remove every ‘e’, with no training at all.
Before the model picks each token, I set the score of every token containing an ‘e’ to negative infinity, so it can never be picked. (The mask also blocks ‘E’ and non-ASCII characters, so look-alikes such as ‘é’ can’t sneak in.) But the mask only stops the model from writing certain tokens. It doesn’t teach the model to choose words that avoid ‘e’ in the first place, so it often ends up repeating itself or misspelling words.
Here’s its answer to the same woodchuck prompt: Response: This is a classic riddiculous and humorous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous riddiculous The answer is clearly nonsense, even though no words have an e.
But now we can focus on the other two parts: writing something that makes sense and answers the prompt. Since the mask takes care of the ‘e’, I use it in most experiments.
To keep things clear, I’ll use two names throughout: Unmasked: Qwen3-1.7B, told not to use ‘e’, with no mask Masked: the same, but with the mask on (and no training yet) Before any training: Setup Usable Unmasked (instruction only) 0% Masked (instruction and mask) 26% Unfortunately, 26% is not great.
Training on Examples (SFT) Supervised fine-tuning (SFT) trains the model to copy example text, so the main question is which examples to show it. Training on an existing ‘e’-free novel didn’t help. Gadsby is 50,000 words (about 69,000 tokens) written without ‘e’, and the hope was that the model would pick up natural ‘e’-free phrasing from it.
The loss went down, but on the eval set the model scored about the same as the masked model, around 26%.
Here are some of its answers next to the masked model’s, with the judge’s coherence and responds scores under each: Masked You should study by practicing past topics and using flashcards.
Also, try to think about what you know and ask for clarification if you don’t fully grasp a topic.
Consistly practicing and taking mock quizzs will also aid in your studying.
coherence 3, responds 5 Gadsby-trained model Study for a t / Study for a t / Study for a t / Study for a t… coherence 1, responds 1 Two things could explain this: 1) A novel teaches the model to write prose, not to answer prompts. and/or 2) 69,000 tokens might just be too little to make a difference. Training on ‘e’-free answers to prompts did help. There weren’t any datasets of these online, so I made my own: I took 180 suitable prompts from the training split and had frontier models write ‘e’-free answers. Even frontier models couldn’t always write an ‘e’-free answer in one go, but they usually fixed their mistakes quickly when told which words to replace (this did not work with Qwen3-1.7B though).
To see how much the amount of data matters, I trained separate adapters on 15 to 180 of these examples.
Each one trained until its validation loss started rising (a sign of overfitting), then ran the eval: Training examples 0 15 30 45 90 135 180 Usable under the mask 26% 10% 20.5% 22.5% 31.5% 34% 35% The best adapter (180 examples) scores 35%, 9 points above the masked model.
I’ll call it the SFT adapter from here on, and it becomes the second starting point for RL.
The gain isn’t just from having more text. The 300-token cap only applies when the model generates answers (in the eval and during RL). SFT trained on full chats of up to 3,072 tokens, so the 180 examples come to about 117,000 tokens, against 69,000 for Gadsby. But the 90-example run, the closest match in size, still beat Gadsby (31.5% against about 26%). Three other things stand out: The gains are leveling off. Doubling from 90 to 180 examples added only 3.5 points, so writing more examples of this kind probably won’t help much more. Small training sets hurt. With 15 examples, answers got longer and worse. Answers got cleaner but drifted off topic. When the key word needs an ‘e’, the adapter swaps in another word and answers about that one instead (“clik” for “cliché”). So I kept the SFT adapter and moved on to RL. Training with the Judge (RL) My first RL attempt went wrong because the only thing it scored was the ‘e’. The judge fixes that: it scores whether an answer makes sense and answers the prompt, while the mask still takes care of the ‘e’. The loop is simple: pick some prompts → sample some responses for each prompt → score the responses with the judge → turn the scores into a reward that pushes up the better responses → repeat over lots of prompts. In more detail, each training step was: Pick a few prompts from then prompt pool. Generate a group of answers for each prompt, with the mask on and the same no-‘e’ system message the eval uses. Have the judge score every answer in parallel. Each answer gets a coherence score and a responds score, both from 1 to 5. Turn the scores into a reward: reward = (coherence − 1 + responds − 1) / 8. That gives a number from 0 (both 1) to 1 (both 5). Compare each answer to the others in its group. Within each group, subtract the group’s average reward and divide by its spread. An answer gets pushed up only if it beat the other answers to the same prompt.
Update the LoRA adapter once.
To show how the scoring works in this setup, here’s step 5 on an example group: 8 answers to one prompt reward 1 0.5 0 +1.4 +0.9 +0.9 +0.5 −0.5 −0.9 −0.9 −1.4 5, 5 5, 4 4, 5 4, 4 3, 3 2, 3 3, 2 2, 2 judge scores (coherence, responds) average 0.625 ↑ pushed up ↓ pushed down The number above each bar is the answer’s group-adjusted reward. The specific settings I used were: Setting Value Prompts per step 4 Answers per prompt (group size) 8 Sampling temperature 0.7 Max answer length 300 tokens Thinking off LoRA rank 16 Learning rate 2e-5 KL penalty weight 0.02 Training steps 300 Why these numbers 4 prompts per step. 4 prompts × 8 answers is 32 answers per step, which peaked at 10 to 13 GB of GPU memory, inside the 15 GB limit I set on my 16 GB card.
8 answers per prompt.
GRPO only learns from a group if its answers get different scores, and it learns most when at least one of them is usable.
From the check below, about two thirds of groups of 8 should contain a usable answer.
Temperature 0.7. I first planned to sample at 1.0, but at 1.0 the SFT adapter’s answers fell apart, with made-up words like “Ranak” and “palacco”.