Fable OSI — a redesigned online stability indicator and adaptive PID authority

Contents

1. What was wrong with the current OSI / authority update

The current runtime (m91_adaptive_margin + auth_stability_gain, replayed faithfully by run_current_osi_auth_factor_cases) fails in four structural ways, all visible in the comparison figures below:

1. The residual branch is biased low by construction. Every update it takes the maximum of ~40 noisy per-bin power ratios and compares it against a percentile-based calibration constant. The maximum of noisy ratios systematically exceeds the calibration, so the branch reports a low margin (≈0.3) on perfectly healthy data, forever. Since the reported margin is min(ST, residual), the bias wins.
2. The baseline is calibrated during the worst possible seconds. It uses the first 7.5 s of the record — which contain the −14 mm actuator homing ramp — and never revisits the choice, so the "healthy floor" is garbage in every one of these records.
3. Blind optimism when unobservable. When there is no coherent input, the ST branch reports the cap (0.70) rather than "don't know", so margin and authority decisions are made on fiction in the long command-off stretches of records 1731 and 1930.
4. The authority law can only creep back. Recovery is a fixed 0.0015/update integrator with no model of what authority would actually be safe. Combined with the biased margin, the deployed test (1930) collapsed 0.3 → 0.08 at t≈90 s and never recovered. The margin also carries ad-hoc shaping (st_power exponent, percentile games) with no physical meaning.

2. The margin being estimated

Same quantity the project already standardizes on: the chordal (normalized-coprime-factor–style) distance of the loop's Nyquist locus to the critical point, evaluated per frequency and minimized over the band [2, 100] Hz (58–62 Hz line-rejected):

with T estimated from the measured dd → dm map and S = 1 − T. Since any strictly proper loop has |L| → 0 somewhere in the band, 1/√2 is the mathematical ceiling; the estimator caps there, and the cap carries the meaning "no evidence of margin deficit", never "measured 0.707". For this rig the nominal (healthy plant tf_sp1_7000, PID P0.15/I20/D0.0015) margin is 0.588 @ 35 Hz; the degraded plant (tf_sp1_5000) gives 0.345 @ 38 Hz.

3. The fable indicator (fable_osi.m)

3.1 Two-scale rolling-STFT architecture

The estimator steps every 0.125 s. The SLOW scale (accuracy): 1 s Hann segments hopped every 0.25 s in a 5 s ring of 17 segment spectra — one new 2048-point FFT per hop (the previous runtime recomputed all 39 segment FFTs of its window at every update, then had to time-slice them across target ticks). The FAST scale (latency, §3.6): 0.25 s segments in a ~0.5 s ring — one 512-point FFT per step. Auto- and cross-spectra are aggregated by medians across segments (robust to one-segment transients), and coherence is bias-corrected using the effective number of independent averages k_eff (computed from the Hann overlap correlations; k_eff ≈ 9.1 here).

3.2 Branch A — coherent (S,T) margin, debiased

A bin enters the ST branch only if it passes a null-calibrated coherence gate — the threshold where a truly unrelated bin would pass with probability α = 1%:

and an input-power gate (within 35 dB of the in-band peak). The power gate matters: a single false-coherence bin at an undriven frequency reads |T̂| = huge and would nuke the margin to zero — this exact failure appeared in testing and is why coherence alone is not enough. The margin is the minimum of the 3-bin-median-smoothed chordal curve over gated bins (≥5 required), with a 3-sample temporal median. No percentile shaping, no exponent — the estimate is left unbiased so that its level can be compared against the target honestly.

3.3 Branch B — residual growth at undriven frequencies, evidence-based

Instability that the command doesn't excite shows up as growing output power at quiet bins (dm = S·v under noise v). The branch watches only bins the command does not drive, inside the configured hazard band (default 28–100 Hz — where this loop can actually go unstable; low-frequency harmonic distortion of a strong low-band command is deliberately outside it). Per update:

1. residual PSD ratio vs a calibrated baseline, self-normalized by the cross-bin quiet median — broadband intensity changes cancel;
2. robust z-score of the log ratio against the calibration spread (median/MAD);
3. the branch engages only after z > 3.5 persists for 3 hops at the same frequency (±3 Hz). Until then it reports the cap — zero steady-state bias, which is what the old max-of-ratios branch could not do;
4. once engaged, the full growth is mapped to margin using the |L| ≈ 1 relation near the critical point (output PSD ∝ |S|², b ∝ 1/|S|):

where b₀(ω*) is the prior margin curve at the binding bin. Disengagement is hysteretic (z < 2).

3.4 Calibration that defends itself

• Outlier-rejecting collection: baseline windows must agree with the running reference within 6 dB; if everything gets rejected the reference itself was the outlier (actuator homing) and collection restarts.
• Excitation-regime rebaseline: the set of command-driven bins changing (a test phase starting/stopping) legitimately moves the quiet-bin floor — and it is an exogenous, observable event: instability grows quiet-bin power but cannot change which bins the command drives. On a sustained excited-set change the baseline is recollected; the ST branch keeps running throughout. This is what lets one calibration-free runtime survive record 1731 (command off for 620 s, then broadband) and the phase-switching 1930 test without false alarms.
• Gated slow adaptation: when z is quiet, the baseline tracks drift with a 100 s time constant, hard-capped at ±3 dB from calibration so degradation can never be "absorbed".

3.5 Validity semantics

A strict honesty rule throughout: when nothing is measurable the margin is NaN (warm-up, no data) — never a fabricated "nominal"; when only the output floor is observable the margin is the cap with st_valid=false, and the authority law treats those states differently.

3.6 The fast path — damage to authority action in ≤ 0.5 s

The 5 s analysis window cannot flip its median in half a second, so speed comes from a second scale, not from degrading the accurate one. The estimator steps every 0.125 s and keeps a short ring of 0.25 s segments beside the slow ring. Two fast statistics, both CUSUM quickest-change detectors (accumulate max(0, Σ(z−ν)); sustained moderate shifts cross in a few steps, isolated healthy spikes decay):

1. Pooled FRF-change: d = median over observed bins of |T̂_new−T̂_slow|²/|T̂_slow|², from the NEWEST segment alone — the bin pooling supplies the averaging that a chordal margin from 0.5 s of data cannot. Log-z-scored against its own calibrated healthy spread. A trip additionally requires a margin-degrading direction (b_fast < b_slow) and an absolutely-concerning fast margin — so the change back to healthy after an event, or a benign shift at safe margins, never sheds.
2. Quiet-band burst: the newest segment's undriven sub-band powers (five fixed sub-bands of the hazard band) against calibrated levels — divergence-like growth compounds the shed every 0.125 s step.

Fast trips can only ever reduce authority (one decisive drop sized by the fast margin for FRF trips; compounding for burst trips), latch the recovery ceiling, and hand regulation to the slow path. Four gates keep the trip rate at zero on healthy data — each one was added because a real record or a closed-loop scenario demanded it: (i) stand down while the excitation regime is changing or nascent (the input-support mask moving); (ii) stand down when the newest segment's input power leaves [0.25, 4]× its ~3 s average (a fading or bursting command mimics change evidence); (iii) a trip immediately followed by a confirmed regime recalibration is retrospectively diagnosed as the transition itself and undone; (iv) the authority-motion gate (added 2026-07-02): the FRF-change statistic watches the total loop, which includes the authority factor itself, so in closed loop the caller writes the currently applied authority into state.a_cmd and the statistic arms only after the authority has been steady (<fast_amove_frac relative motion) for one full analysis window — the law can therefore never trip on its own regulation moves. Real damage arrives while the authority is static, so the gate costs nothing there; replays where the authority never acted on the plant leave a_cmd unset and keep the detector always armed. Validated result: zero fast trips across all healthy phases of all three records and all healthy synthetic scenarios.

Information limit, stated honestly: when the command has no power near the critical frequency, the only damage signature is the noise-floor growth — for this rig's plant pair that is ~1.5–2× per 0.25 s snapshot, i.e. a 2–3σ event per segment. No detector can certify that in 0.5 s at a near-zero false-alarm rate; the CUSUM accumulates it in ~1.5 s (S4). A truly diverging loop produces huge z and still trips in ~0.4 s regardless of excitation.

4. The target-tracking authority law (fable_auth.m, rewritten 2026-07-02)

Scalar-state supervisory law, updated every 0.125 s OSI step. The law regulates the reported margin onto a target OSI (parameter target, default 0.55): whenever the margin is at or above the target the authority recovers to and holds a_max = 1 — a healthy loop runs at full authority; when the margin drops below the target the authority sheds multiplicatively, a ← a·(b/target)^p, until the margin climbs back. The closed loop settles where b ≈ target (or at a = 1 when the loop is healthier than the target even at full authority). Contract, in priority order:

Every knob above — and which way to turn it for a smoother, more constant, or more aggressive authority, and for faster or slower drop/recovery — is documented in the TUNING GUIDE header of fable_auth.m. The per-branch estimator knobs (each OSI branch smoother ↔ more aggressive) are documented in the header of fable_osi.m. Tune interactively with fable_tune_explorer (§9).

4.1 The predicted-safe authority, done correctly

Because authority scales the loop linearly (L(a) = (a/a_now)·L̂), the measured loop L̂ = T̂/(1−T̂) at coherent bins yields a predicted margin curve b_pred(α) over a 25-point gain grid. One subtlety is essential and easy to get wrong:

The prediction is a bound and an accelerator, never a trust anchor: it floors confirmed sheds at 90% of the predicted-safe authority (no pointless slam to a_min while the 5 s window flushes) and caps recovery at the safe-interval boundary, linearly interpolated between grid points — the boundary already stands pred_guard (0.03) of margin short of the target crossing, and the up_deadband hysteresis supplies the standoff below it. (The first draft applied an additional ×0.9 haircut on the way up; combined with the coarse gain grid it permanently stranded the authority at ≈0.9 whenever the target sits close to the healthy margin — with target 0.55 against a 0.588 healthy margin the loop could never return to full authority. The margin slack is the principled standoff; the multiplicative haircut double-counted it and is now down-path only.) The prediction is only consulted when the binding frequency is inside the coherently observed band (st_binding); a residual-driven deficit is outside the model's sight, so there the shed proceeds to the power-law floor until the evidence clears.

5. Ground-truth validation (closed loop, computable true margin)

run_fable_synthetic_checks.m builds a sample-accurate closed loop — the project's discrete PID against the identified plants (tf_sp1_7000 healthy / tf_sp1_5000 damaged, both strictly proper, dual-state so the swap keeps states continuous), with the OSI and the authority law running in the loop at their real cadence, output noise, and the true chordal margin computed analytically at every update from the active plant and current authority. Target 0.50 (healthy true margin 0.588, damaged 0.345). All five scenarios pass:

5.1 The extended 10-scenario suite (fable_synthetic_suite.m, target 0.55) — 10/10 PASS

The same sample-accurate closed loop, generalized into a method-pluggable test bench: scenarios accept any indicator+authority method via a small struct interface (fable_synthetic_suite('template')), so future methods are validated against the identical ground truth without touching the harness. Plant changes are a continuous blend y = (1−λ)·y₇₀₀₀ + λ·y₅₀₀₀ (steps, ramps, and reversals), and the true margin is computed from the blended loop at the current authority at every update. Run at the production target 0.55 — which sits only 0.038 under the healthy margin (0.588), so these scenarios exercise the regulator where it actually has to work. All ten pass:

Cases 7 and 8 encode the recovery philosophy as a testable contrast: after a trip (hard evidence of a plant change) recovery is latched and takes minutes by design; after a deficit that was only ever regulated (no trip), an improving plant gets its authority back in seconds. Both directions are tunable (latch_decay_tau_s, up_rate, up_prop).

6. The three experimental records

Replay caveat (applies equally to every offline runner in OSI_tests): the replayed authority did not act on the plant, so the authority traces show what each law would have commanded given the observed margins. Closed-loop behavior is what Section 5 validates. The black dots are an after-the-fact reference: 20 s windows, 2 s Welch segments, coherence > 0.5 — the best margin estimate hindsight can produce from dd→dm, where the input allows one. Both laws are replayed against a 0.50 target margin (figures and scorecard refreshed 2026-07-02; earlier versions of this section used the 0.30 project setting — the margin traces and reference errors are unchanged, the authority and detection behavior are target-dependent).

6.1  20260617_1731 — command off for 620 s, then broadband; P-gain excursion to near-instability

6.2  20260617_1749 — low-band command (2–7 Hz coherent); genuine instability at end of record

Note the harmonic-distortion lesson this record taught: a strong 2–7 Hz command produces narrowband nonlinear products at 10–28 Hz that track excitation level. A residual branch that scans the full band flags them forever (that draft read 0.22 healthy). Confining the watch to the hazard band where this loop can actually go unstable (28–100 Hz; nominal binding 35–38 Hz) removes the false alarm without giving up the protection that caught the real event.

6.3  20260630_1930 — el-centro EQ phases; the live OSI runtime was active

7. Scorecard — fable vs current implementation, all three records

Healthy windows: 1731 [625,690] s (command on, pre-event), 1749 [40,210] s, 1930 [80,395] s. Target margin 0.50 (numbers and figures refreshed 2026-07-02; the earlier scorecard used the 0.30 project setting — margin medians, IQRs and reference errors are essentially unchanged, the false-low, authority and detection rows are target-dependent). "Full authority" means a ≥ 95% of the law's own initial value (fable a₀=1.0; current a₀=0.3). Note the 0.50 target sits inside this rig's healthy margin scatter on 1749 and near the EQ-transition dips on 1930, so some authority loss while healthy is the honest price of that target — see the captions.

* cap = "no evidence of deficit": during 1930's step-and-hold phases the 5 s window has no coherent bins and no quiet-band growth; the 20 s reference still resolves ≈0.55–0.62 there.  † the 1930 "event" is a 1–2 s dm transient coincident with the live runtime restoring its gain from the floor; the offline reference is undefined there and no sustained growth follows. Fable's burst CUSUM trips once on the transient itself (a defensible ×0.6 precaution with slow latched recovery); the current implementation's instant "detection" there is its ordinary bias, not skill.

8. Limitations & deployment notes

• dd/dm only, Speedgoat-shaped. The estimator honors the existing runtime contract (no fm/dc required). Step mode is fixed-size per call (hop_n samples, one FFT); state is a fixed-size numeric struct, flattenable exactly like prepare_osi_gain_runtime flattens the M9.1 config today. The dd→dm map absorbs the phase lead G_T; the margin is therefore the outer (transfer-system) margin — the same quantity the current tests target (target_outer_ncf).
• Unobservable-band optimism. Where the command has no power and the output floor shows no growth, the margin reads the cap. That is a statement of evidence, not of the true global margin (1930's step phases read 0.707 while the 20 s reference sees ≈0.6). The authority law compensates: recovery in that state is slow, and the residual latch remembers hazards the band cannot currently see.
• Common-mode blind spot. The residual z-statistic is self-normalized across bins, so a perfectly flat broadband output rise is absorbed by design (that is what makes it immune to intensity changes). Broadband divergence is covered by the calibrated sub-band burst CUSUMs and, when any input is present, by the ST branch; a physically diverging loop also concentrates power at the unstable frequency within a couple of windows, which the z-statistic then sees.
• Quiet-band regulation tolerance. With no command power at the critical frequency, a mild deficit (< ~0.08 below target for this rig) produces a growth signature beneath any detector's noise floor; the law regulates such deficits to a target−0.08 tolerance band rather than to the target itself (S4). Serious deficits and divergence produce strong signatures and are shed hard and fast. This is an information limit, not a tuning choice.
• Fast-path guarantee scope. "Authority moves ≤0.5 s after damage" holds when the command excites the critical band (S3 class, and any test with broadband/EQ excitation through the binding region). Quiet-band-only damage takes ~1.5 s (S4 class); truly diverging loops trip in ~0.4 s regardless.
• Replay ≠ closed loop. On the records, authority did not feed back into the plant. The closed-loop claims rest on Section 5, where the whole chain (estimator → law → loop) runs sample-accurately against a computable truth.
• Knobs that matter: target (0.55 production default — it must stay below the healthy-loop margin at full authority, 0.588 for this rig, or the law will permanently detune the loop chasing an unreachable margin; 0.30 in the record replays, 0.50/0.55 in the synthetic studies); residual_band_hz (set from where your loop can go unstable); window_s/hop_s (latency vs variance). The full authority tuning surface — smoothness (b_filt_tau_s), drop speed and sizing cadence (down_rate, down_settle_s, re_shed_delta), recovery speed (up_rate, up_prop, up_hold_s, up_rate_blind), anti-flicker (min_step, deadbands, confirm counts), post-event caution (latch constants) — is documented knob-by-knob in the fable_auth.m header; the per-branch estimator knobs in the fable_osi.m header. Tune against real records with fable_tune_explorer (§9).

9. Speedgoat runtime, equivalence & the tuning workflow

The step path is deployed as control_and_HS/OSI/fable_osi_runtime.m — a single %#codegen MATLAB Function body wrapped exactly like the legacy osi_gain_runtime.m: called once per 2048 Hz base-rate sample with scalar dd, dm, en; all configuration arrives as fixed-size parameter arrays built at compile time by prepare_fable_runtime.m (index maps documented there); all estimator state lives in fixed-size persistents and the authority state in scalar persistents; raw samples go into circular rings (no per-tick buffer copy). Outputs: auth, b_osi, b_st, b_res, b_fast, trip_reason, valid, decision, held between internal 0.125 s steps. The enable/pause contract matches the legacy block (en = 0 shadow, 1 active, +2 pause resets the rings but keeps calibration and authority state).

If the target overruns (TET too high), the runtime header contains the ordered playbook: (1) scheduling — move the block to a triggered subsystem or split the hop work across steps (no algorithm change); (2) fable_pred_enable = 0 (drops the prediction grid; the law falls back to power-law regulation); (3) step_s = 0.25 (halves the fast path, doubles trip latency); (4) hop_s = 0.5; (5) narrower band_hz; (6) shorter window_s; (7) fewer alpha_grid points; (8) fewer burst sub-bands. Do not shrink segment_s below 1 s or weaken the coherence/input gates — they are the false-alarm protection, not overhead.

Tuning workflow. fable_tune_explorer replays any record in saved_results/tests/important (selection dialog, index, or path) through the exact runtime function and draws linked-zoom subplots — dm, every OSI branch + the selected margin + the target line, and the authority with trip markers — with your 'osi'/'auth' override structs stamped on the figure. Because the explorer runs the very code the Speedgoat executes, a tuning settled there transfers 1:1 to the target. Re-validate any tuning with fable_synthetic_suite (closed loop) before deploying.

10. Files & how to run

addpath(fullfile(project_root, 'OSI_tests', 'fable'))
test_fable_runtime_equivalence(2);      % runtime == desktop, bitwise
checks  = run_fable_synthetic_checks;   % 5 closed-loop ground-truth scenarios
suite   = fable_synthetic_suite;        % 10-scenario suite (target 0.55)
results = run_fable_osi_tests;          % 3-record replay + current-impl comparison
fable_tune_explorer;                    % pick a record, tune interactively

Fable OSI — OSI_tests/fable — generated 2026-07-01, updated 2026-07-02 (target-tracking authority law, Speedgoat runtime + bitwise equivalence, 10-scenario suite, tune explorer). All numbers in this report are reproducible from the runner scripts; record replays are cached in results/.