A defect in the AppleBridge host server made the entire bridge appear to hang whenever the emulated Macintosh was not connected. The cause was a single blocking accept() call positioned ahead of the control-port service loop. Its most damaging consequence was self-referential: the one diagnostic endpoint designed to report which layer had failed was itself disabled by the failure it was meant to describe. This report documents the symptom, the root-cause analysis, the repair, and the measured verification.
1. Problem statement
The AppleBridge stack connects an AI agent to a Macintosh running System 7.6.1 under Basilisk II. Because the emulated machine sits behind MACNAT and cannot accept inbound connections, the architecture is reversed: the guest daemon dials out to a host server, which listens on two ports.
Agent --:9001--> host_server.py <--:9000-- Mac daemon --Apple Events--> ToolServer
(control port) (guest, dials out)
Port 9000 carries the daemon link; port 9001 is the local control port through which every agent-facing tool is issued. The two are served by a single process and, as it turned out, by a single thread of control.
Operational folklore had accumulated around a startup-order rule: the host server must be started before the emulator, or the bridge would not work and the guest had to be rebooted. The project documentation recorded the rule as fact:
Order matters: the host server must be listening on
:9000before Basilisk II / the daemon start, or the daemon’s early connects foul the socket.
No mechanism for the alleged socket fouling had ever been identified. The rule was empirical, and the prescribed remedy — reboot — was expensive and, as shown below, unnecessary.
2. Observed symptom
With the emulator running but the daemon not connected, the guest console reported an ordinary, honest failure: a bounded connect attempt that timed out, followed by a scheduled retry.
The daemon’s behaviour here is correct. OTConnect is issued asynchronously with a notifier, the wait loop yields to the cooperative scheduler on every pass, the attempt is bounded at CONNECT_TIMEOUT_TICKS = 600 (10 s), and failure is followed by RECONNECT_DELAY_TICKS = 1800 (30 s) of backoff before an unlimited retry. Nothing about the guest requires intervention.
The host side, however, was silent. Control-port requests received no reply at all:
$ printf 'status\n\n' | nc -w 5 localhost 9001
$ # no output; returned only on timeout
The corresponding liveness tool returned an empty payload with every field null, and the server log’s final line was Waiting for Mac daemon to (re)connect....
3. Root-cause analysis
The control loop in host/host_server.py was structured as follows:
while True:
if not server.connected:
log("Waiting for Mac daemon to (re)connect...")
server.accept_mac() # blocking: server_socket has no timeout
server.heartbeat()
server.negotiate_version()
try:
ctrl_conn, _addr = control.accept() # never reached while disconnected
except socket.timeout:
...
accept_mac() called accept() on the :9000 listening socket. That socket carried no timeout, so the call blocked indefinitely. Because it preceded control.accept() in the same loop body, the control port was never serviced while the daemon was absent.
The failure mode is subtle in an instructive way. The :9001 socket remains bound and listening throughout, so the kernel completes the TCP handshake and queues the connection in the accept backlog. A client therefore connects successfully and then waits forever for an application that will never call accept(). From the client’s perspective the server is reachable but mute — a state that resembles a hung peer far more than a refused connection, which is precisely why it resisted diagnosis.
The most consequential instance of the defect concerns the liveness endpoint. The MACSTATUS handler carries this comment:
elif cmd == "MACSTATUS":
# Liveness, answered HOST-side so it never hangs when the
# daemon is down. Reports the host's connection/heartbeat
# view, then merges the daemon's STAT counters if reachable.
The intent is explicit and correct: report host-side state without consulting the daemon, so that an operator can determine which layer has failed. The handler is nevertheless unreachable in exactly the condition it was written for, because the blocking accept() above it holds the only thread. The diagnostic channel failed together with the system it was built to diagnose.
This also disposes of the startup-order rule. The determining condition is not the order of process startup but simply whether a daemon is currently connected. Starting the client first merely guarantees an interval in which it is not. No socket was ever fouled; the server was deaf.
3.1 A masking failure
The investigation was initially misdirected by a second, independent fault. The guest could not reach the host because Basilisk II’s etherhelpertool — the child process that owns the network interface through a BPF handle — had terminated when the Thunderbolt Ethernet adapter was disconnected. Basilisk II continued running with no network interface whatsoever.
The signature is worth recording, since it is invisible from the host’s own configuration:
pgrep -fl etherhelpertoolreturns nothing while the emulator process is alive;- the interface has silently lost its
PROMISCflag, which a functioning helper sets; - packet counters are frozen. Over a 30 s interval the wired interface recorded 0 packets in either direction while the default-route interface recorded 2,836 — a promiscuous port on a live segment always observes broadcast traffic.
Reconnecting the cable does not repair this: the helper is spawned only at emulator launch, so a full quit and relaunch is required, and a guest-only reboot is insufficient. Crucially, this fault presents identically to a misconfigured host — the same timeout, the same silence — which is what made a working host-side diagnostic endpoint valuable in the first place.
4. Design and implementation
Three changes were made.
Non-blocking acquisition of the daemon link. accept_mac() gained an optional timeout. Passing None retains the blocking behaviour required by the interactive path; the control loop passes a short poll interval and proceeds regardless of the outcome:
self.server_socket.settimeout(timeout)
try:
self.client_socket, addr = self.server_socket.accept()
except socket.timeout:
return None
finally:
self.server_socket.settimeout(None)
The loop no longer gates control service on the daemon’s presence. With MAC_ACCEPT_POLL = 0.5, the control port retains approximately one-second responsiveness while the guest is down, at negligible cost. The “waiting for daemon” notice is emitted once per outage rather than once per iteration.
Explicit rejection instead of silence. Commands requiring the daemon now fail immediately with a framed error naming the probable cause and the remedy, including the etherhelpertool signature described above. MACSTATUS is exempt from this guard, since reporting the absent daemon is its purpose.
Diagnostic output on the guest. The daemon’s one-line status field cannot carry instructions, and while the bridge is down the console is the only channel available to it — there is no link over which to inform the host. On connect failure the daemon now writes an ordered checklist to its Verbose log:
*** HOST SERVER NOT REACHABLE - bridge is DOWN ***
no reply: our connect was never answered
check on the HOST, in this order:
1. is host_server.py running? host/start_stack.sh
2. is 192.168.3.154 on the default-route NIC?
3. emulator NIC alive? quit BasiliskII FULLY + relaunch
no commands can run until this link comes up.
retry 1: still no host - next attempt in 30s
On-device rendering of that output, daemon 0.8d24 on System 7.6.1, produced by stopping the host server deliberately. The address shown is read from the daemon’s preferences rather than compiled in, and the two subsequent retries are suppressed to a single line each:
The address is read from the daemon’s preferences, so the message names the endpoint actually being dialled. The full block repeats every eighth attempt — roughly every five minutes given a 10 s timeout plus 30 s backoff — so that an operator opening the window late still learns the cause, while intervening retries produce a single line. Recovery is announced explicitly.
One limitation is inherent and remains. The host’s firewall operates in stealth mode and therefore drops SYN packets addressed to a closed port rather than returning a TCP RST. “Host server not running” and “address bound to the wrong interface” consequently arrive at the guest as the same bare timeout and cannot be distinguished there. The message accordingly enumerates candidate causes in the order they are worth checking rather than asserting one.
5. Verification
Isolated reproduction. A second server instance was run on ports 19000/19001 with no daemon attached, leaving the production bridge untouched. Before the change, MACSTATUS returned zero bytes and blocked for the full 10 s client timeout; the request was never logged, confirming it was never accepted. After the change:
| Request (no daemon connected) | Before | After |
|---|---|---|
MACSTATUS | no reply, 10 s timeout | 0.13 s, host_connected=0;daemon_responding=0 |
| Daemon-dependent command | no reply, 10 s timeout | 0.50 s, framed error with remedy |
The connection path was re-tested with a synthetic client to confirm that accept, connect, disconnect detection and re-accept all continue to function.
Live end-to-end test in the disputed configuration. The guest was started first, with no host server running — the exact scenario the startup-order rule prohibits. The server was then started. The log records the outcome:
09:48:41 Waiting for Mac daemon to (re)connect... (control port stays live; mac_status reports daemon down)
09:48:45 mac_status -> host_connected=0;idle_seconds=4.2;missed_heartbeats=0;daemon_responding=0
09:48:48 Mac connected from ('192.168.3.244', 2050)
09:48:48 HELLO: negotiated protocol v2 feat=[]
The liveness query at 09:48:45 was answered accurately while the daemon was still absent — the request that previously hung. Three seconds later the daemon’s own retry established the link and negotiated protocol v2. Total elapsed time from server start to a working bridge was seven seconds, with no restart of the emulator and no reboot of the guest.
The guest console records the same sequence from the other side: two bounded timeouts while the host was down, then SYNC-OK and the HELLO:2 version negotiation.
The daemon’s retry loop had always been capable of this recovery. What was missing was a server that would listen and a channel that would report.
6. Status
Both changes are deployed and verified in production. The host-side fix was activated first; the guest-side console message was subsequently compiled, linked and swapped into the running daemon as build 0.8d24.
The swap was performed over the bridge itself. The binary was linked to a
sibling file rather than over the running daemon, the SIZE resource was
re-applied (0x5AE0, retaining isHighLevelEventAware — without it every
subsequent command would fail with -903), and the SWAPSELF verb renamed the
running daemon aside before moving the new binary into its place. Renaming an
open file is permitted by the File Manager, whereas overwriting it is not. A
reboot then brought up the new binary, which DeRez -only 'vers' confirms as
0.8d24.
The complete cycle, observed on the guest with the host server stopped and then restarted, exercises all three behaviours: the full diagnostic block once, the suppressed one-line retries, and the explicit recovery announcement.
The startup-order rule can be retired. The recommended remedy for a bridge that will not come up is no longer to reboot, but to query the liveness endpoint — which now answers.
7. Lessons
Two observations generalise beyond this system.
A diagnostic facility must not share a failure domain with the subsystem it observes. The MACSTATUS handler was correctly designed and correctly commented; it was defeated by a scheduling decision fifty lines above it. Reasoning about the handler alone would never have revealed the defect, because the handler was not wrong.
Persistent operational folklore is evidence of an undiagnosed defect. The startup-order rule was reproducible, was written into the documentation, and was wrong in its stated mechanism — no socket was fouled. It survived because the prescribed workaround, restarting in a fixed order, happened to avoid the triggering condition. A rule that works for reasons nobody can name is a defect that has not yet been found.