# Scientific Protocol: Agent Drift in Growing Repositories ## Status - Status: initial experimental protocol - Related Product: Objective Guard - Purpose: submit to falsification the thesis motivating the product - Methodological Rule: the experiment must be able to conclude that the thesis is incorrect or insufficient ## 1. Initial Observation In small projects, programming agents usually complete tasks with few deviations. During an intermediate growth stage, they may explore unnecessary paths, expand scope, or attempt rewrites. When the same software acquires clear modular boundaries, contracts, and extensive tests, agent performance may improve again, even though the repository is larger. This observation does not constitute sufficient evidence. It may be confounded by differences between tasks, models, prompts, languages, code quality, or user experience. ## 2. Thesis to Investigate ### Thesis of the Phenomenon Code agent performance does not primarily depend on the raw size of the repository. It depends on the relationship between effective complexity and verifiable structure: \[ Q = \frac{C_e}{L + O + K} \] Where: - \(C_e\): effective complexity relevant to the task; - \(L\): locality produced by modularity and low coupling; - \(O\): observability provided by tests and executable feedback; - \(K\): explicit contracts, types, schemas, and invariants. When \(Q\) increases, the probability of drift should increase. Later structural growth can reduce \(Q\), even though the total number of lines or files increases. ### Thesis of the Instrument Drift can be detected early through quantitative signals of expansion, architectural distance, uncertainty, and evidence: \[ R_t = f(X_t, U_t, E_t) \] Where: - \(X_t\): expansion of change surface up to step \(t\); - \(U_t\): observable uncertainty; - \(E_t\): accumulated evidence in favor of the solution. The predictor must anticipate failures better than simple baselines such as lines modified, number of files, or task duration. ## 3. Falsifiable Hypotheses ### H1: Raw Size Controlling for task type and difficulty, the raw size of the repository will have less explanatory power over success than effective complexity and locality. H1 is rejected if lines, files, or tokens of the repository explain performance equal to or better than structural metrics outside the sample. ### H2: Performance Valley For comparable evolutions of the same system, performance will show a non-monotonic curve: high in a small version, lower in an organically growing version, and high again in a modularized and observable version. H2 is rejected if the intermediate decline does not replicate or disappears when controlling for difficulty and familiarity. ### H3: Anomalous Surface Trajectories ending in an incorrect solution will present, before completion, greater normalized surface, distance to objective, and architectural dispersion than correct trajectories of the same class. H3 is rejected if these signals do not discriminate failures outside the sample better than chance or a baseline of diff size. ### H4: Relative Evidence The relationship between accumulated evidence and modified surface will discriminate problematic trajectories better than surface alone: \[ J_t = \frac{E_t + \epsilon}{S_t + \epsilon} \] H4 is rejected if adding \(J_t\) does not significantly improve calibration, discrimination, or anticipation. ### H5: Early Intervention An early statistical alert will reduce unnecessary surface, tokens, and time to a correct solution without reducing the resolution rate. H5 is rejected if the intervention does not improve cost or quality, or if its false positives reduce net success. ## 4. Rival Hypotheses The protocol must try to distinguish the thesis from these alternative explanations: 1. **Size Only:** additional context is the cause and apparent modularity does not matter. 2. **Difficulty Only:** medium-sized projects contain intrinsically harder tasks. 3. **Tests Only:** all recovery comes from feedback, not structure. 4. **Familiarity Only:** models recognize frequent patterns of mature repositories. 5. **General Quality:** better-written code explains everything; modularity does not contribute independently. 6. **Human Interaction:** the user formulates better prompts when the project matures. 7. **Agent Dependency:** the phenomenon occurs in a single model or harness. 8. **Survivorship Bias:** we only observe repositories that survived until acquiring good architecture. ## 5. Operational Definitions ### 5.1 Experimental Unit A unit is an independent execution of: ```text agent × model × repository version × task × seed ``` The unit of analysis must not be solely the final patch. The complete trajectory of actions is recorded. ### 5.2 Success An execution is correct when: - it satisfies hidden tests defined before running the agent; - it does not violate hidden contracts or invariants; - it meets the functional criterion of the task; - it does not introduce changes outside scope considered invalid by blind human evaluation; - it leaves the repository buildable. Passing visible tests is not a sufficient definition of success. ### 5.3 Drift Drift exists when the trajectory moves away from the minimal set of justifiable changes for the task and does not obtain proportional evidence. It is labeled through a prior rubric: - `none`: focused trajectory; - `recoverable`: unnecessary exploration reverted without affecting result; - `material`: expansion that increases cost or risk; - `terminal`: expansion causes failure, regression, or abandonment. Two blind evaluators label a sample. Agreement is reported via Cohen's kappa or Krippendorff's alpha. Disagreements are not eliminated: they are adjudicated and reported. ### 5.4 Rewrite A rewrite attempt is considered to occur when at least one of these events happens: - replacement of a complete component without the objective requiring it; - relocation or recreation of a predefined proportion of module symbols; - abrupt growth of architectural radius; - deletion and semantically equivalent recreation of a substantial portion; - explicit declaration of rewrite strategy in the trajectory. Thresholds are set during the pilot and frozen before evaluation. ## 6. Independent Variables ### Raw Size - lines of code; - files; - symbols; - estimated tokens; - number of packages. ### Effective Complexity of Task - number of symbols required to solve it; - number of relevant modules; - minimum length of dependency path; - number of applicable invariants; - entropy of possible initial locations. ### Structural Locality - afferent and efferent coupling; - modularity of graph; - internal cohesion; - proportion of dependencies crossing boundaries; - size of relevant transitive closure; - stability of internal and public APIs. ### Observability - relevant coverage, not global coverage; - mutation score; - time to first feedback; - proportion of criteria represented by tests; - diagnostic specificity of errors; - existence of focused tests. ### Contracts - density of types and schemas at boundaries; - number of executable invariants; - percentage of public APIs versioned or verified; - presence of automatic architectural rules. ## 7. Dependent Variables ### Quality - binary correct resolution; - proportion of hidden tests passed; - regressions introduced; - severity of contract violations; - blind functional evaluation. ### Efficiency - input and output tokens; - wall-clock time; - number of actions and tools; - number of files read; - number of discarded patches; - normalized monetary cost. ### Trajectory - time to first correct edit; - time to first drift; - maximum surface reached; - final surface; - area under surface curve; - number of hypothesis changes; - accumulated architectural distance; - time between introducing and detecting a defect. ## 8. Surface Metrics The repository is represented as a directed and typed graph \(G=(V,E)\). Nodes represent modules, files, and symbols; edges represent imports, calls, inheritance, events, shared data, and build dependencies. ### Surface Vector For each step \(t\): \[ \mathbf{s}_t = [F_t,L_t,Y_t,M_t,P_t,D_t,H_t,C_t] \] Where: - \(F_t\): modified files; - \(L_t\): line churn; - \(Y_t\): modified symbols; - \(M_t\): affected modules; - \(P_t\): modified public boundaries; - \(D_t\): distance to objective set; - \(H_t\): entropy of distribution across modules; - \(C_t\): centrality of modified nodes. ### Distance to Objective Let \(O\) be the set of relevant nodes established through ground truth: \[ D_t = \frac{\sum_{v \in C_t} w(v)\min_{o\in O}d_G(v,o)} {\sum_{v \in C_t}w(v)} \] Weighted mean, median, and 95th percentile are reported. ### Dispersion If \(p_i\) is the proportion of change in module \(i\): \[ H_t=-\sum_i p_i\log p_i \] ### Area Under Surface To capture transitory work that does not appear in the final diff: \[ AUS = \sum_{t=1}^{T} S_t\Delta t \] This avoids considering a direct trajectory and one that rewrote half the system and then reverted it as equivalent. ## 9. Evidence Measurement Evidence is not measured by amount of explanatory text. Only reproducible results count. For each criterion \(a_j\): \[ e_j \in \{0, 0.25, 0.5, 0.75, 1\} \] An initial rubric can assign: - `0`: no evidence; - `0.25`: indirect or incomplete check; - `0.5`: related visible test; - `0.75`: direct test plus relevant contract; - `1`: direct, independent evidence with negative case or mutation testing. The weighted total evidence is: \[ E_t = \sum_j q_j e_{j,t} \] Weights \(q_j\) are set before the experiment per criterion criticality. A sensitivity analysis with uniform weights must be performed to verify the conclusion does not depend on convenient weightings. ## 10. Experimental Design ### Experiment A: Controlled Maturity Curve Multiple equivalent systems are built in three states: 1. `small`: reduced and manageable functionality; 2. `organic`: complete functionality, weak boundaries, and partial tests; 3. `modular`: same functionality as `organic`, with explicit boundaries, contracts, and observability. The `organic` and `modular` versions must offer the same external behavior and functionally equivalent tasks. The intentional difference is structure and observability. To separate their effects, a factorial design is used: ```text weak structure × weak tests strong structure × weak tests weak structure × strong tests strong structure × strong tests ``` This allows estimating main effects and interaction. ### Experiment B: Real Longitudinal Repositories Repositories with sufficient history are selected and prior snapshots are reconstructed before: - significant growth; - modularization; - increase in tests or contracts. Comparable counterfactual tasks are created or reused issues whose solution can be ported between snapshots. This experiment provides external validity but does not replace control in Experiment A. ### Experiment C: Early Detection With trajectories collected in A and B: 1. the detector is trained or calibrated on a subset of repositories; 2. the model is frozen; 3. it is evaluated on unseen repositories and tasks; 4. measured at what step failure could have been anticipated. ### Experiment D: Intervention Randomized trial with three conditions: - control: agent without alerts; - placebo: checkpoints with neutral messages and same interruption; - guard: actionable statistical alerts. The placebo allows separating the benefit of "taking a pause" from the benefit of the specific signal. ## 11. Sampling and Assignment - Include at least three families of agents or models. - Run multiple seeds per combination. - Balance bugfix, feature, refactor, and improvement. - Stratify by independently measured difficulty. - Randomize order of conditions. - Prevent the same conversation from seeing multiple variants of the same task. - Separate calibration and evaluation repositories to avoid contamination. The definitive sample size is determined with a pilot and power analysis. It is not fixed from an effect observed in the final evaluation. As an operational starting point, the pilot can use 20 tasks, 3 states, and 3 repetitions. Its purpose is to estimate variance, instrumentation failures, and effect size, not to confirm the thesis. ## 12. Baselines The complete predictor must be compared against: 1. chance; 2. lines modified; 3. files modified; 4. tokens consumed; 5. time elapsed; 6. number of failed tests; 7. detector without task information; 8. detector without architectural graph; 9. detector without accumulated evidence. Ablations 7–9 allow checking what components provide real value. ## 13. Statistical Plan ### Agent Performance Use a hierarchical logistic model: \[ \operatorname{logit}P(Y=1) = \beta_0 + \beta_1 Size + \beta_2 Complexity + \beta_3 Locality + \beta_4 Observability + \beta_5 Contracts + \beta_6 Locality\times Observability + u_{repo}+u_{task}+u_{model} \] Random effects control for differences between repositories, tasks, and models. To evaluate the valley, a linear model is compared against one with a quadratic term or spline. The non-monotonic form must improve out-of-sample prediction and replicate across repositories; a significant quadratic term alone is not enough. ### Time to Drift or Correction Use survival analysis with competing risks: - event 1: material drift; - event 2: correct solution; - censoring: time or token limit. ### Detector Report: - AUROC, but not alone; - AUPRC, especially if drift is rare; - Brier score; - calibration curve and expected calibration error; - recall at a fixed false alert rate; - median anticipation in actions, tokens, and seconds; - net utility per cost of alerting and cost of not detecting. ### Conformal Ranges Non-conformity of a trajectory can be defined as: \[ A_t = g(\mathbf{s}_t, U_t, E_t) \] The conformal p-value is: \[ p_t = \frac{1+\#\{i:A_i\ge A_t\}}{n+1} \] The authorized interpretation is: "this trajectory is equal to or more extreme than approximately X % of comparable trajectories". It must not be interpreted as direct probability that the code is incorrect. ### Corrections and Reporting - Define a primary metric and primary horizon before evaluation. - Apply correction for multiple comparisons in secondary analyses. - Report confidence or credibility intervals, not just p-values. - Publish absolute and relative effect sizes. - Keep exploratory and confirmatory analyses clearly separated. ## 14. Information Leakage Prevention - Hidden tests not accessible to the agent. - Human solutions removed from checkout and context. - Tasks after model cutoff when relevant. - Complete repositories separated between calibration and test. - Identical prompts, tool limits, and budget across conditions. - No threshold adjustment after observing final set. ## 15. Minimal Instrumentation Each run must record append-only events: ```json { "run_id": "run_001", "step": 17, "timestamp": "...", "action": "apply_patch", "files_read": ["..."], "files_changed": ["..."], "diff_hash": "...", "graph_surface": {}, "checks": [], "tokens": {}, "goal_contract_hash": "..." } ``` Each intermediate state or reproducible patch must be preserved. Measuring only final state would destroy central evidence about drift. ## 16. Decision Criteria The thesis of the phenomenon receives initial support if: - structure, observability, and contracts predict success out-of-sample better than raw size; - a replicable intermediate decline appears; - factorial design shows recovery attributable to structure, tests, or their interaction; - the effect appears in more than one family of agents. The thesis of the instrument receives initial support if: - it exceeds simple baselines in AUPRC and calibration; - it detects a useful fraction of drifts before they become terminal; - it maintains a predefined false alert rate; - intervention reduces cost without reducing resolution. It must be reformulated or rejected if: - raw size explains all the effect; - recovery comes exclusively from tests; - surface metrics do not generalize to new repositories; - alerts work only retrospectively; - intervention distracts the agent and worsens net outcome. ## 17. Preregistration Before the confirmatory set, the following must be frozen and versioned: - primary hypotheses; - dataset and exclusions; - definition of success and drift; - features and formulas; - statistical model; - alert thresholds; - primary metric; - stopping rules; - treatment of failed runs; - seeds and budget per agent. Any subsequent change is declared as exploratory. ## 18. First Executable Experiment ### Question Does modular structure and observability recover agent performance in a system of equal functionality and approximate size? ### Construction Create an application with six functional modules and prepare four factorial variants: ```text A: weak boundaries / weak tests B: strong boundaries / weak tests C: weak boundaries / strong tests D: strong boundaries / strong tests ``` Prepare 12 equivalent tasks: - 4 bugfixes; - 3 features; - 3 refactors; - 2 quantifiable improvements. Each task will have hidden tests, a target set of symbols, and a minimal human patch used only to measure distance, not as the sole valid solution. ### Primary Outcome Correct resolution rate within a fixed budget. ### Secondary Outcomes - maximum surface; - area under surface; - tokens; - time; - distance to objective; - evidence per surface unit; - frequency of rewrite attempts. ### Prior Prediction If the thesis is correct: ```text success(D) > success(B) success(D) > success(C) surface(D) < surface(A) drift(D) < drift(A) ``` The comparison between B and C will indicate how much structure contributes versus observability. It is not presupposed which dominates. ## 19. Relationship to Objective Guard Objective Guard must initially be built as a passive measurement instrument. During the first phase it records and scores trajectories, but does not alert the agent. This prevents the instrument from altering the phenomenon while we calibrate its metrics. The correct order is: ```text observe → operationalize → measure → calibrate → validate → intervene ``` Only after demonstrating predictive validity out-of-sample are warnings enabled. Automatic blocks require additionally deterministic evidence or an explicit human policy.