One hot line stops a shire

22 September 2026, re-measured 25–26 September · three cards: aifoundry2, aifoundry3 and aifoundry1 card 1 · workloads/nocbench --test hotline, a new many-to-one probe · following up a result reported in Discord by Ivan, another ET-SoC-1 programmer (first analysed, with his 6% taken at face value, in L2 mainline starvation) · part of the ET-SoC-1 measurement reports

Ivan put one global atomic counter in shire 0's scratchpad, had all 32 shires hammer it, and found that shire 0 got 6% of its fair share and finished only after the other 31. The obvious reading is that the shire cache short-changes whoever owns the line. It does not. The atomic is shared out to within half a percent (every shire between 0.998 and 1.004 of an even split), including with the shire that hosts it. What that shire loses is something else entirely: with the clock steady at 600 MHz, its own memory path stops — not slows, stops — for as long as the hammering lasts. The switch is sharp: 21 requesters in one other shire leave it at 96% of its rate, 22 stop it, fewer than the 32 minions a shire has. It reproduces on all three cards we could run it on (aifoundry2, aifoundry3 and aifoundry1 card 1), in three passes on each.

Checked on three cards (26 September 2026): this page's claims were re-measured under a pre-registered plan, in three passes on each of aifoundry2, aifoundry3 (both firmware 1.3.1; aifoundry3 held at 600 MHz) and aifoundry1 card 1 (firmware 1.2.0), and the sweeps' tables and charts now show those passes. Several held with sharper numbers: the edge is exactly where section 3's inequality puts it, 22 requesters, and the pacing knee just above its 9,704 cycles. The one that did not: 31 requesters, one per shire, saturate the line's bank but did not stop the one minion of its shire that was reading (section 7, corrected). The reading-only rate in section 6 was qualified, and the bus error of section 8 was not repeated. The cards agree throughout, except one aifoundry2 launch in section 1. Of 17 claims tested here, this page counts 14 held, 1 did not hold (corrected), 1 qualified and 1 not repeated; the hub’s scoreboard, 17 “a test behind it failed”. Record: docs/reports/data/2026-09-25-claims-v3.

1. The atomic itself is fair

Every minion in the launch hammers one 4-byte word with amoaddg.w. All of them meet at a chip-wide barrier first, so the window is the same window for everyone, and each one reports how many atomics it completed inside it. A shire's share is its count divided by what an even split would give.

2. The thing that actually starves

So we changed what the host shire does. Instead of joining the atomic, its 32 minions stream their own memory — 64-byte-strided loads over a private slice, either of their shire's scratchpad or of DRAM — while the other 31 shires hammer a line that lives in that same shire cache. The baseline is the identical loop with nobody else launched.

These are not slow-downs, they are stops. The host shire's count is the same number whether the window is 5 milliseconds or 100:

About seventeen loads per minion get through, five warm-up loads before the window opens and twelve inside it, and then nothing: none through the 120 million remote atomics of a 2-second window, in all 21 such runs of the 23 September passes on the two cards. The 32 minions of that shire, a full thirty-second of the chip, are stalled in a load instruction that does not return. Nothing reports an error; the kernel finishes when the hammering stops.

3. The threshold is the bank going saturated

It takes one other shire. Two shires run, the host and one other, each with the same number of minions n. The host's n minions read their own scratchpad, and the other shire's n hammer a scratchpad word in the host shire. The host's rate is compared with n times the per-minion rate of its 32-minion loop alone, which is why a single host minion reads 111%.

There is no gradual degradation. From 2 to 20 remote requesters the host shire stays within about a percent of untouched (98.8–99.97% of its rate alone with the same number of minions, on all three cards), because each remote atomic takes about 216 cycles to make its round trip and twenty of them in flight still ask for less than the bank can retire. Twenty-one keep the bank at 10.3 cycles per atomic and leave the host at 95.6–95.7%; by twenty-two the measured cost reaches 10.0 cycles per atomic — the bank's floor, the queue full — and the host shire's own requests stop getting resources. One inequality says when:

N × 10 cycles < P + t_{round trip} ⟹ the bank is not saturated

4. Esperanto knew: two errata describe it

Two entries in Esperanto's ET-SoC-1 errata describe it (numbered 4.1 and 4.2 in the document's body, 3.1 and 3.2 in its table of contents), and both are marked Postponed:

That bears on both questions worth sending back. Would setting l3_yield restore the host? Not certainly. Erratum 4.1 says it does not when the host and the mesh want the same address. The case measured here is different addresses, which is what the yield was built for. But erratum 4.2 says the yield has no sub-bank granularity, so a host request to the sub-bank the hot line saturates can still be skipped indefinitely, and each host minion's stream reaches that sub-bank sooner or later. It was not tested. Does it hold when the line is DRAM-backed rather than in scratchpad? Yes, and slightly worse: 192–240 of the host's loads get through instead of 384 (section 2), at every window from 5 to 100 ms and on all three cards. The erratum's title says same SCP address, but the behaviour is not specific to the scratchpad: both kinds of address tested, a scratchpad word and a DRAM line homed in that shire, do it.

5. The workaround, priced

Erratum 4.1's workaround is to slow the other shires down. It works, and near the knee it is nearly free — the bank was already saturated, so the first thing pacing takes away is queueing, not throughput:

6. What contention costs: 32× the time and 17× the energy

Spreading the same work over 32 lines, one per shire, changes everything: the same 1,024 minions running the same instruction do 32 times the work.

What it costs in energy

Contention is not just slow, it is expensive per unit of work. Board power over idle on aifoundry2, in one session, measured at 10 Hz across three back-to-back launches of each case, with the die at 72–73 °C; the last column adds the 23 September passes on aifoundry2 and aifoundry3 (the three-card check did not repeat the power runs):

A thousand minions stalled on a saturated line cost the card , which is the clearest possible statement that waiting is cheap and the loss is throughput, not power. Per operation the contended line costs times the energy of the same atomic spread over 32 lines.

7. What to do instead

• One global atomic per shire, never per minion, and never spin on the line. At 32 participants a barrier's arrivals serialise in 320 cycles, against a chip-wide barrier of about 5,000 cycles ( with one minion per shire): six percent. But 31 shires hammering one line already saturate its bank. Wait on a per-shire flag or a credit instead. The relay's first barrier, with every shire's leader polling one counter, hung in the one development run we made, on a card not recorded (Hand it to the next shire). At 1,024 participants the arrivals alone take 10,240 cycles, twice the whole barrier, with the hosting shire stopped throughout.
• Keep hot shared lines out of shires that compute. The cost is not paid by the code touching the line. It is paid by whatever else lives in that shire, and at a steady 600 MHz it is total. Make the owner a shire that does no compute: one of 32 is 3.1% of the chip. Not the master shire, whose scratchpad the firmware uses.
• If you must, pace it:
• Spread instead of sharing when you can. One line per shire is the same instruction at 32 times the rate and a seventeenth of the energy per operation.
• Measure the victim from user mode. Syscall 9, SYSCALL_PMC_SC_SAMPLE, reads the cache-bank counters of any shire, so a kernel can watch the host shire's banks without touching firmware (docs/research/counters-and-dram.md).

8. Method, and what is not established

workloads/nocbench/run_hotline.sh DATA 6000000           # 42 configurations, each well under a second
workloads/nocbench/run_hotline.sh DATA3 6000000          # the same sweep, on aifoundry3
tools/ettelem/run_hotline_power.sh DATA                  # board power and rails, three launches per case
python3 tools/ettelem/analyze_hotline_power.py DATA --out DATA/power.json
# the three-card passes: unit hl of tools/claims-v3/lat/block.sh (tools/claims-v3/lat/README.md), then
V3=docs/reports/data/2026-09-25-claims-v3/raw
python3 workloads/nocbench/analyze_hotline.py --v3 $V3 --power DATA/power.json \
    --context DATA/context.json --barrier docs/reports/data/2026-09-18-nocbench-aifoundry2/barrier-chip1.jsonl \
    --stop-runs docs/reports/data/2026-09-23-reruns-aifoundry2-warm/hotline-pass*/runs.jsonl \
                docs/reports/data/2026-09-23-reruns-aifoundry3/hotline-pass*/runs.jsonl \
    --out DATA/hotline.json
python3 tools/ettelem/analyze_reruns.py docs/reports/data/2026-09-23-reruns-aifoundry2-warm \
    docs/reports/data/2026-09-23-reruns-aifoundry3 --out reruns.json   # the passes behind the ranges in section 6

Related reports

• Hand it to the next shire — the companion: share data between shires through a slab of scratchpad instead of one hot line.
• The ET-SoC-1 energy manual, section 6 — these energies re-measured on aifoundry2 and aifoundry3, with ranges.
• On-chip communication — the chip-wide barrier on three cards (8.3 µs from global atomics and credits, 2.3 µs on the hardware tree) and the reduction trees that can replace one global counter.
• Anatomy of a memory access — how physical-address bits 10:6 pick a line's home shire, and what an L3 hit costs per mesh hop.
• Sparse compute — a divergence probe whose 2,048 harts take their work from one global atomic counter.
• Spatial temperature — the per-shire temperature sensors that could tell whether a hot line heats its shire.