Five Question Marks: An Encoding Bug and the Plausible Wrong Answer

Someone types äöüÄÖÜß into a text field in Netscape Navigator 3, running under System 7.6.1. What arrives on the server is ?????? . Seven characters in, seven question marks out. The explanation seems obvious — and the obvious explanation is wrong.

This is a short account of an encoding bug in ClaudeBridge 2.0, and of the detour taken while fixing it. The detour is the interesting part: the wrong fix looked like progress.

The fix and its test are in the source repository: github.com/LoetLuemmel/ClaudeBridge

The symptom

The question went into a chat form and came out the other end like this:

Can you read my text with Umlauts? ????? ?

The model at the far end, which sees nothing of the transport chain, named the failure class correctly on the first try:

This is a classic character encoding issue! Your Mac is likely using MacRoman encoding, while somewhere along the way to me, the text is getting converted (or not converted) and the special characters get lost.

Right category. Wrong encoding. And the wrong part was the part that got acted on.

Why question marks specifically

Three layers have to line up here, and it helps to keep them apart:

flowchart LR
    A["Netscape 3
form submit"] -->|"percent-encoded bytes"| B["HTTP body"]
    B --> C["parse_qs
decodes %XX"]
    C --> D["sanitize()
to ISO-8859-1"]
    D --> E["Claude API"]

The server read the request body as ISO-8859-1 — correct, and irrelevant. The body at that point is pure ASCII: the umlaut is not a byte yet, it is the three literal characters %FC.

The actual decoding happens one layer down, in parse_qs, which turns percent-sequences back into characters. And parse_qs defaults to UTF-8. A lone byte above 0x7F is not valid UTF-8, so it became U+FFFD, the replacement character. The sanitize stage, which maps anything outside ISO-8859-1 to ?, then produced the question marks.

So the bug was never in the line that mentions encoding. It was in the line that doesn’t.

The plausible wrong answer

The natural conclusion on a classic Mac: the guest displays text through the Script Manager in MacRoman, therefore it sends MacRoman. In MacRoman ä is 0x8A, ü is 0x9F. The fix writes itself:

params = parse_qs(body, encoding="mac_roman")   # wrong

The question marks disappeared. That was the trap.

What actually came over the wire

Only a real measurement settled it. ClaudeBridge stores its conversation history on disk, so the received text can be inspected as code points rather than as glyphs on a screen — and glyphs are exactly what lies to you here.

The next umlaut test, after the “fix”, was stored as:

'Gr¸sse aus M¸nchen!'    ü -> U+00B8   (CEDILLA)

U+00B8 is what you get when you decode byte 0xFC as MacRoman. So the browser had sent 0xFC — and 0xFC is ü in ISO-8859-1, not MacRoman.

The rule that was violated

A browser submits a form in the character set of the document. Not in the character set the operating system uses to draw glyphs.

ClaudeBridge serves every page as ISO-8859-1 — a deliberate decision, because that is what Netscape 3 on a Mac handles reliably. So the browser encodes the form the same way. The Script Manager governs how the Mac renders those bytes on screen; it has no say in what goes on the wire.

The correct fix is therefore the encoding that was already being used everywhere else in the project:

params = parse_qs(body, encoding="iso-8859-1")

Verified end to end through the real handler: %FC and %DF now arrive as U+00FC and U+00DF.

Why the wrong fix was worse than no fix

A broken thing announces itself. ????? is unmistakable — nobody looks at that and concludes the problem is solved.

Gr¸sse aus M¸nchen is different. It is almost right. The umlauts are gone but something umlaut-shaped is in their place, the sentence is readable, and the natural reading is “nearly there, some residual issue further down the chain”. The model at the far end read it exactly that way and suggested the remaining fault might be in the proxy.

A half-corrected bug moves the search away from the actual cause. That is the practical cost of a fix that treats the symptom with the wrong model of why.

Three encodings, three separate problems

The lasting lesson from this project is that “encoding” is not one concern:

All three are about the same character. All three fail differently. And fixing one says nothing about the other two — the escaping bug in this same codebase was fixed hours earlier, in the opposite direction, and left this one entirely untouched.

The takeaway

When a diagnosis contains a category and a detail, they carry very different weight. “This is an encoding mismatch” was worth acting on. “Your Mac is using MacRoman” was worth checking — it was plausible, it was stated with the same confidence, and it was wrong.

The check was cheap and available the whole time: the received text was on disk, in code points. Reading it took one command. It just happened after the wrong fix instead of before it.