# Adversarial review by Codex of Fable's research: agents and U-curve ## Metadata - Reviewer: Codex - Date: 2026-08-02 - Evaluated document: [Fable's Research on agents and U-curve](research-fable-agents-u-curve.md) - Related protocol: [Scientific protocol on agent drift](research-protocol-agent-drift.md) - Purpose: to evaluate rigor, evidence, falsifiability and utility of Fable's research for Objective Guard ## 1. Executive verdict Fable's research is conceptually very good and more ambitious than the existing experimental protocol. Its formalization via the relationship between necessary minimum context and effective context capacity is a powerful hypothesis. It also offers plausible explanations for the rewriting impulse, recovery via modularity, and the value of tests. However, the document does not yet constitute confirmatory evidence. It mixes four epistemic levels that must stay separate: 1. published results; 2. inferences compatible with those results; 3. mathematical analogies; 4. phenomenological reconstructions of model behavior. The distinction appears declared in text, but some conclusions receive more confidence than evidence cited permits. Overall evaluation is: > Excellent hypothesis-generating theory; evidence still insufficient to assert that the U-curve was demonstrated or that its mathematical mechanisms were established. ## 2. Condensed evaluation | Dimension | Assessment | | --- | ---: | | Conceptual power | 9/10 | | Epistemic honesty | 8/10 | | Hypothesis quality | 9/10 | | Figure provenance | 4/10 | | Derivation rigor | 5/10 | | Potential falsifiability | 8/10 | | Utility for Objective Guard | 9/10 | The recommended combination is: ```text Fable's Research ↓ mechanisms and explanatory theory Codex's scientific protocol ↓ operationalization, controls and falsation Objective Guard ↓ instrumentation, data and experiment ``` Replacing the scientific protocol with Fable's research is not recommended. Incorporating its most promising hypotheses into the protocol and subjecting them to controlled experiments is recommended. ## 3. Strongest aspects ### 3.1 The relationship \(K/C_{\mathrm{eff}}\) The document's most valuable conceptual contribution is defining: - \(K\): minimum founded context necessary to act correctly; - \(C_{\mathrm{eff}}\): agent's effective context capacity, less than its nominal window. It can be summarized via: \[ \rho = \frac{K}{C_{\mathrm{eff}}} \] This allows describing three regimes: | Regime | Condition | Expected behavior | | --- | --- | --- | | Sufficiency | \(\rho < 1\) | Agent can gather sufficient context. | | Instability | \(\rho \approx 1\) | Small task or recovery differences change the result. | | Insufficiency | \(\rho > 1\) | Decisions increasingly depend on unfounded assumptions. | In a modular repository, necessary context might approximate as: \[ K_{\text{modular}} \approx K_{\text{module}} + K_{\text{interfaces}} \] If interfaces correctly encapsulate internal behavior, \(K\) can stop growing with total repository size. The formalization is elegant because it explains why performance could recover in a larger system: the model's capacity doesn't increase; the amount of information needed for a correct local decision decreases. Nevertheless, \(K\) and \(C_{\mathrm{eff}}\) are latent variables. To become experimental science they must be operationalized via observable proxies and subjected to validation. ### 3.2 Rewriting as locally rational decision The document proposes that the agent compares perceived cost of understanding the system with perceived cost of regenerating it: \[ \text{rewrite if } \widehat R < \widehat U \] Where: - \(\widehat R\): estimated cost of rewriting; - \(\widehat U\): estimated cost of understanding and modifying. In the complexity valley: - \(\widehat U\) increases because dependencies and constraints are scattered; - \(\widehat R\) is biased downward because it omits historical behavior, edge cases, compatibility and unknown consumers; - generating new code is more straightforward than conditioning every decision on idiosyncratic existing code. The consequence is a decision that can be locally coherent and globally destructive. This hypothesis is especially useful for Objective Guard because it allows searching for signals prior to rewrite attempt: - abrupt surface growth; - increase in deleted and recreated files; - creation of substitute abstractions; - expansion of architectural radius; - drop in evidence per unit of change. ### 3.3 Ignorance becomes irrelevant The most precise phrase in the document is: > Sealed compartments don't make the agent smarter; they make its ignorance irrelevant. This captures the value of information hiding. The agent doesn't need to know every internal detail if it can justifiably trust a verifiable boundary. From Objective Guard's viewpoint, interfaces, contracts and tests are not merely documentation: they function as evidence-backed compression. ### 3.4 Honesty about introspection The document correctly declares that it has no direct access to weights, activations or attention patterns, and that its introspective explanation is phenomenological. This precaution is necessary. A model's self-reports can generate hypotheses about observable behavior, but don't prove internal mechanisms. ### 3.5 Pre-registered attackable points The section enumerating incomplete provenance, unmeasured analogies and unobservable variables elevates the document's intellectual quality. It exposes in advance how it could be refuted. This practice should be preserved and expanded in future documents. ## 4. Main scientific problems ### 4.1 C1's figures are not backed by the stated source The document reports approximately: ```text repos <10K LOC: ~91 % pass rate repos 10–50K: ~67 % repos >50K: ~15 % ``` However, the cited source as primary support, ProjDevBench, does not present that curve. ProjDevBench studies end-to-end project construction from specifications, using 20 problems. Human solutions average around ten files and the work is not a controlled study of performance by repository size. ProjDevBench does support more general claims: - agents fail at specification alignment; - prolonged interactions negatively correlate with performance; - integration deficiencies, edge cases and systemic decisions appear. It doesn't support, by itself, the LOC→pass rate table. Verified source: [ProjDevBench](https://arxiv.org/html/2602.01655v1). #### Recommended correction Change C1 from `[MEDIUM]` to: ```text [UNVERIFIED — do not use quantitatively] ``` until identifying the exact primary source, reproducing the table and verifying: - pass rate definition; - number of repositories per stratum; - evaluated models and agents; - task difficulty; - confidence intervals; - possible confounding variables. ### 4.2 A U-curve doesn't rule out size causality The summary claims that U-shape rules out repository size as a causal variable. That conclusion is too strong. A non-monotonic relationship can include size as a cause interacting with modularity and observability: \[ Y = \beta_0+ \beta_1 Size+ \beta_2 Modularity+ \beta_3 Observability+ \beta_4 Size\times Modularity \] Survivor bias can also exist: repositories that grow and remain maintainable probably have better teams, tests, processes and architecture. #### Recommended correction Replace: > U-curve rules out size as a causal variable. With: > U-curve would suggest that gross size is insufficient as explanation and that its effect is moderated by structure, observability and contracts. ### 4.3 The mathematics of horizon uses unproven assumptions The formula: \[ P(\text{success})=(1-\epsilon)^T \] is valid under restrictive conditions: - constant error probability; - independence, or specifically modeled dependence; - any error is terminal; - steps have comparable difficulty; - no recovery exists. Real trajectories normally violate these conditions. Errors can be correctable, correlated or increase probability of later failures. More adequate representation uses dynamic risk: \[ h_t = \sigma( \beta_0+ \beta_1S_t+ \beta_2U_t- \beta_3E_t+ \beta_4H_t ) \] Where: - \(S_t\): accumulated surface; - \(U_t\): uncertainty; - \(E_t\): evidence; - \(H_t\): horizon or remaining dependencies. Sufficient direct support was also not found for: \[ \epsilon(t)\approx\epsilon_0+\alpha\log t \] nor for a universal transition around 50 steps. HORIZON shows horizon-dependent degradation, but emphasizes that the breaking point depends on domain. In embodied tasks it may appear with few steps, while web tasks tolerate different horizons. Verified source: [The Long-Horizon Task Mirage / HORIZON](https://arxiv.org/html/2604.11978v1). #### Recommended correction Separate C3 into: - an illustrative identity under independence; - a dynamic hazard hypothesis; - domain-specific empirical results. Don't present a constant or universal threshold without direct evidence. ### 4.4 Transfer from DAgger to tests is overasserted The document asserts with high confidence that tests convert error accumulation \(O(\epsilon T^2)\) to \(O(\epsilon T)\), by analogy with DAgger. DAgger studies imitation learning and covariate shift. It compares policies trained on expert-visited states with policies incorporating their own induced states, using expert corrections. Tests executed during a programming session are not equivalent to that mechanism: - they don't always indicate correct action; - can be binary or little diagnostic; - don't equal consulting an expert policy; - don't update model weights; - don't guarantee relevant state coverage; - may validate an incomplete objective interpretation. The closed-loop control analogy is useful, but the asymptotic result can't be transferred as a theorem. #### Recommended correction Reclassify M4 from `[HIGH]` to: ```text [SPECULATIVE — analogy with control and online learning] ``` And formulate a measurable hypothesis: \[ \text{greater diagnostic density of feedback} \rightarrow \text{lesser error propagation} \] Must be measured by varying experimentally: - feedback frequency; - diagnostic precision; - relevant coverage; - latency; - ability to localize the correction. ### 4.5 "Constraint decay" was interpreted imprecisely The cited paper uses constraint decay to describe performance drop as structural and persistence requirements are added: architecture, database and ORM. It reports that capable configurations lose approximately 30 assertion pass-rate points between simple and fully specified conditions. It doesn't necessarily demonstrate that an initial instruction progressively forgets due to contextual distance. Verified source: [Constraint Decay](https://arxiv.org/html/2605.06445). The result introduces productive tension with Fable's thesis: ```text Fable's hypothesis: more structure → lower K → better performance Constraint Decay: more structural constraints → greater difficulty → worse performance ``` It's not definitive contradiction. It requires distinguishing four phenomena: 1. existing, explicit and navigable structure; 2. structure added as requirement that agent must implement; 3. implicit structure based on conventions; 4. verifiable interfaces that allow ignoring internal details. A more complete formalization would be: \[ K = K_{\text{business}}+ K_{\text{constraints}}+ K_{\text{integration}}- K_{\text{modular compression}} \] Structure can increase or decrease \(K\) depending on how it's represented and whether it already exists or must be built. ### 4.6 Percolation is analogy, not validated explanation The comparison with giant component emergence in Erdős–Rényi is creative, but software graphs are not simple random graphs. They typically have: - hierarchies; - direction and edge types; - hubs; - heavy-tailed degree distribution; - communities; - intentional evolution; - organizational constraints. Direct transfer of mean-degree-one threshold doesn't apply. The usable intuition is that transitive impact closure can experience abrupt growth. For node \(v\), you can measure: \[ B(v)= \frac{|\operatorname{reachable}(v)|}{|V|} \] Also measurable: - affected component size; - conductance between modules; - edge density between communities; - propagation cost; - change of blast-radius percentile across versions. The falsifiable hypothesis would be: > Probability of drift increases non-linearly when task impact radius crosses certain repository percentiles. ## 5. Evidence that does support parts of the thesis ### 5.1 Iterative degradation SlopCodeBench offers direct evidence of degradation when agents must repeatedly extend their own code: - 20 problems; - 93 checkpoints; - 11 models; - no agent completes a problem end-to-end; - structural erosion increases in 80% of trajectories; - verbosity increases in 89.8%; - agent code is roughly 2.2 times more verbose than compared human repositories; - prompt interventions improve initial quality but don't stop degradation slope. Verified source: [SlopCodeBench](https://arxiv.org/html/2603.24755v1). This supports: > Agent-produced code negatively conditions its future iterations and final benchmarks may hide accumulated erosion. It doesn't yet demonstrate: - the U-curve; - recovery by modularization; - the \(K/C_{\mathrm{eff}}\) formula; - a universal transition; - that tests change error accumulation from quadratic to linear. ### 5.2 Degradation with horizon HORIZON gathers over 3,100 trajectories across four domains and shows degradation as horizon increases. It also finds a change in failure composition: planning and memory gain importance in long horizons. This supports that the problem isn't only local correction. Trajectory and accumulated state matter. However, the work itself warns that no single breaking point applies to all domains. ### 5.3 Difficulty respecting structure Constraint Decay demonstrates that adding architectural and persistence requirements can strongly reduce performance, even holding functional contract fixed. This strengthens the need to measure effective complexity rather than use repository size alone. ## 6. Salvageable hypotheses for the scientific protocol ### 6.1 Contextual sufficiency ratio \[ \rho = \frac{K}{C_{\mathrm{eff}}} \] #### Possible operationalization of \(K\) It can't be observed directly, but can approximate via: - size of minimal file set needed to solve the task; - minimum count of relevant symbols; - tokens of minimal programmatic slice; - count of needed contracts and invariants; - size of relevant dependency closure; - entropy between plausible locations; - performance when providing progressive context subsets. #### Possible operationalization of \(C_{\mathrm{eff}}\) Can estimate experimentally via: - correct retrieval of relevant facts at different positions; - ability to respect distributed constraints; - performance against distracting context; - ablation and reordering tests; - amount of founded context correctly used in decisions. The hypothesis doesn't require both variables be perfectly identifiable. It does require proxies with convergent validity. ### 6.2 Probability of rewriting \[ P(\text{rewrite})= \sigma( \widehat U- \widehat R ) \] Can test by manipulating: - code navigability; - availability of architectural maps; - amount of hidden contracts revealed; - allowed inspection cost; - ease of generating a replacement; - warnings about blast radius. The prediction would be that reducing understanding cost or revealing actual regeneration cost decreases rewrite attempts. ### 6.3 Structure as reduction of necessary context \[ \Delta K= K_{\text{before}}-K_{\text{after}} \] Must evaluate keeping functionality constant and varying: - modularity; - interface explicitness; - tests; - implicit conventions; - architectural documentation. The protocol's factorial design is appropriate: ```text A: weak structure / weak tests B: strong structure / weak tests C: weak structure / strong tests D: strong structure / strong tests ``` ### 6.4 Evidence relative to surface Fable's intuition fits the metric proposed for Objective Guard: \[ J_t = \frac{E_t+\epsilon}{S_t+\epsilon} \] The risk case isn't simply a large diff. It's a trajectory whose surface grows faster than its evidence: \[ \Delta S_t \gg \Delta E_t \] This metric can detect expansion before determining code is incorrect. ## 7. Revised claims | Original claim | Verdict | Recommended reformulation | | --- | --- | --- | | Collapse by LOC is measured with cited figures | Unverified | Find primary source; don't use figures yet. | | U-curve rules out size as cause | Excessive | Size is insufficient and likely interacts with structure. | | Iterative degradation is measured | Supported | SlopCodeBench provides strong erosion and verbosity evidence. | | Success universally follows \((1-\epsilon)^T\) | Illustrative | Use as idealized case; model dynamic hazard. | | \(\epsilon(t)\approx\epsilon_0+\alpha\log t\) | Unverified | Treat as hypothesis and compare functional forms. | | Universal transition exists near 50 steps | Unsupported | Breaking point depends on domain and task. | | Tests convert \(O(\epsilon T^2)\) to \(O(\epsilon T)\) | Invalid transfer as theorem | Treat as hypothesis about diagnostic density. | | Constraint decay is progressive constraint forgetting | Imprecise interpretation | It's performance drop from constraint accumulation. | | Percolation explains the jump | Fertile analogy | Measure transitive closure and blast radius directly. | | \(K/C_{\mathrm{eff}}\) explains the three regimes | Promising hypothesis | Operationalize and validate out of sample. | ## 8. Implications for Objective Guard ### 8.1 Natural market of the product The idea that Objective Guard has greater value in the valley is convincing: ```text small prototype → agent can cover almost everything organic growth → implicit complexity and insufficient feedback mature system → modules, contracts, ownership and tests already limit the problem ``` Objective Guard must offer provisional boundaries and feedback during the phase when the repository doesn't yet produce them internally. ### 8.2 Measure before intervening The product must begin as passive observer: ```text observe → operationalize → measure → calibrate → validate → intervene ``` Alerting from the beginning modifies the behavior it tries to study and makes it hard to calibrate the detector. ### 8.3 Priority signals The first instrument should record: - surface vector per patch; - static and dynamic blast radius; - distance from objective set; - entropy between modules; - transient churn, not just final diff; - accumulated evidence; - executed tests and diagnostic density; - hypothesis changes; - complete replacement attempts; - time and tokens until correction or failure. ### 8.4 Facts, inferences and decisions The runtime must keep separate: ```text Fact: three public interfaces were modified Inference: trajectory is in 99th percentile of expansion Decision: policy requires justifying or approving that change ``` A statistical anomaly doesn't prove incorrectness. It serves to assign verifications and control autonomy. ## 9. Next recommended investigation ### Step 1: document correction - Remove or suspend LOC→pass rate figures until verifying provenance. - Lower confidence of M4. - Separate constraint accumulation from constraint forgetting. - Present percolation as hypothesis. - Replace universal 50-step threshold with domain-dependent effects. ### Step 2: K/C_eff pre-registration Define at least three independent proxies for each variable and specify what results would refute the hypothesis. ### Step 3: factorial benchmark Build functionally equivalent variants: ```text A: weak structure / weak tests B: strong structure / weak tests C: weak structure / strong tests D: strong structure / strong tests ``` ### Step 4: longitudinal observation Record complete trajectories, including reverted patches. Final surface doesn't capture cost of an attempted rewrite later abandoned. ### Step 5: predictive validation Calibrate on some repositories and evaluate on completely different repositories. Compare against baselines: - lines modified; - files modified; - time; - tokens; - failed tests; - gross repository size. ### Step 6: intervention trial Compare: - agent without alert; - agent with placebo pause; - agent with statistical alert; - agent with deterministic block. This allows separating the value of information from the simple effect of interrupting the trajectory. ## 10. Conclusion Fable's research should be preserved because it offers compact, intuitive and fertile theory. Its main value isn't in proving the U-curve, but in proposing mechanisms that can be tested: 1. relative contextual insufficiency \(K/C_{\mathrm{eff}}\); 2. rewriting as biased cost comparison; 3. modularity as reduction of necessary context; 4. feedback as error propagation reduction; 5. iterative degradation caused by coexisting with own previous decisions. Before treating it as probative research, C1, C3, M4 and constraint decay interpretation must be corrected. The final verdict is: > As a piece of thinking, it's excellent. As a causal explanation, it's promising. As scientific evidence, it still requires operationalization, controls, source verification and out-of-sample validation. ## 11. Sources verified during review - [ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development](https://arxiv.org/html/2602.01655v1) - [SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks](https://arxiv.org/html/2603.24755v1) - [Constraint Decay: The Fragility of LLM Agents in Backend Code Generation](https://arxiv.org/html/2605.06445) - [The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break](https://arxiv.org/html/2604.11978v1)