--- name: ingest-norm description: Use when the user wants to add an engineering norm (SIA, Eurocode, etc.) they are licensed for into THEIR Stratta workspace. A Python pre-pass (PyMuPDF) extracts the hierarchical tree and rasterizes figures; an agentic pass enriches sections (LaTeX formulas, tables, cross-references, summaries) and writes everything via the MCP `ingest_*` tools — scoped to the user's own organization. Trigger phrases: "ingère cette norme", "/ingest-norm", "ingest SIA", "ajoute la norme X à Stratta". --- # Ingest a norm into your Stratta workspace This skill turns a norm PDF **you are licensed to use** into a queryable TreeRAG inside **your own** Stratta workspace. A Python pre-pass (PyMuPDF) does the deterministic heavy lifting — TOC tree, per-section raw text, figure captions and full-page renders. Then an agentic pass enriches sections with summaries, LaTeX formulas, structured tables, and cross-references. Final writes go through the Stratta MCP `ingest_*` tools, scoped to your org. > ⚠️ **Licence**: only ingest norms your organization holds a valid licence > for. You are responsible for your usage rights (see Stratta's Terms). ## Prerequisites - Stratta is connected, either way: the remote connector (`https://stratta.ch/mcp`, authorised in the browser, no key) or the local package with a key (`npx -y @stratta/mcp login`). Both serve the same `ingest_*` tools. - **Python ≥ 3.10 with PyMuPDF**. Install once: `python -m pip install --user pymupdf` (or `uv pip install pymupdf`). - The norm PDF is available locally. ## Tools used (all scoped to your workspace) `ingest_status` · `ingest_create_document` · `ingest_create_sections` · `ingest_attach_formula` · `ingest_attach_table` · `ingest_attach_cross_ref` · `ingest_upload_figure` · `ingest_normalize_cross_refs` · `ingest_publish` · `ingest_delete`. ## Quotas — read before you start The workspace has server-enforced limits on norms, sections and figures. A `QUOTA_EXCEEDED` error names the dimension, the current count and the limit. **Do not retry it.** The limit will not move on its own. Stop the ingestion, report the numbers to the user, and offer the two ways out: delete a norm that is no longer needed (`ingest_delete`), or raise the plan. Check the budget up front on a large norm: the pre-pass output tells you how many sections and figures you are about to write. Failing at step 8 of 10 leaves a half-ingested document behind, which the user then has to delete by hand. ## Workflow ### 1. Locate the pre-pass script It ships inside this package at `scripts/ingest-prepass.py`. Resolve its path: ```bash node -e "console.log(require.resolve('@stratta/mcp/package.json'))" # → /package.json → /scripts/ingest-prepass.py ``` If the user is working in the Stratta monorepo, the script also lives at `packages/mcp/scripts/ingest-prepass.py`. If neither is on disk (the remote connector installs nothing), download the same script from the site and keep it next to the PDF: ```bash curl -fsSLO https://stratta.ch/ingest-prepass.py ``` ### 2. Check for an existing copy `ingest_status { code }` (e.g. `"SIA 261"`). To re-ingest, call `ingest_delete { documentId }` first. ### 3. Run the pre-pass ```bash python /scripts/ingest-prepass.py \ --pdf \ --output .stratta-ingest/ ``` Output under `.stratta-ingest//`: - `prepass.json` — full manifest (see below). - `figures/page-NNN.png` — one PNG per page that contains a `Figure N` caption. `prepass.json` structure: - `doc` — `pageCount`, detected `language`, `tocSource`, raw `metadata`, and `licence`: the watermark and copyright lines of the copy, kept once. The page text leaves them out; the norm keeps them and every excerpt served from it shows them. - `stats` — `sectionCount`, `byDepth`, `figureCount`, `charsPerPage`, `emptyLeaves`. `charsPerPage` under 300 means the text layer is missing (a scan without OCR): stop and tell the user. - `warnings[]` — what the pre-pass could not decide on its own: chapters missing from the sequence (an OCR'd copy often loses the big chapter number; read them off the contents page and add them), sub-sections dropped as out of sequence, headings it could not locate, leaves with no text. An empty list is the normal case on a native SIA PDF. **Read it before step 5** and look at the pages it names. - `sections[]` — full hierarchical tree (depth **0 = chapter**, 1 = X.Y, 2 = X.Y.Z, 3 = X.Y.Z.W), each with `nodeId`, `parentNodeId`, `path` (`"4.2.1"` or `"Annexe B"`), `title`, `depth`, `pageStart`, `pageEnd`, `orderIndex`, `rawText` and `textSource`. `rawText` is **the text between this heading and the next one**, on cleaned pages: no running header, no page number, hyphenation resolved, each numbered clause (`9.5.1.1 Les pieux…`) on its own line. A chapter whose text lives in its sections has an empty `rawText`; that is normal. - `figures[]` — one entry per `Figure N` caption: `figureNumber`, `caption`, `page`, `fileName`, `mimeType`. Every titled heading is a node (typically 250 to 350 on a SIA norm); the numbered clauses stay inside their heading's text. ### 4. Create the document Read `prepass.json`, then: `ingest_create_document { code, year, title, language: , totalPages: , scope, licence: }` where `scope` is one sentence on what the norm covers (from its first chapter); it is shown to every agent in the served methodology, so write it as a routing hint: "Geotechnical design: foundations, piles, anchors, retaining structures". Pass `licence` exactly as the pre-pass wrote it, never edited or shortened; omit it only when the pre-pass found none. → returns `documentId`. Keep it for every subsequent call. ### 5. Validate the tree, write the summaries **Keep every node of the pre-pass.** A clause (`9.5.2.1`) is an address an engineer cites and an agent is sent to; folding it into `9.5.2` made the citation impossible and the section too long to read. Do not aggregate, do not drop leaves. A typical SIA norm gives 250 to 800 nodes; the quota allows 1 000 per norm. What the agentic pass is for: - **Validate the tree**, not rewrite it. Read `warnings[]` and the pages it names; fix a wrong title, a missing chapter number, a wrong page range. Keep `nodeId` / `parentNodeId` / `path` / `depth` / `pageStart` / `pageEnd` / `orderIndex` from the pre-pass otherwise. - **`content` = the pre-pass `rawText`, verbatim.** It is the text of the norm between this heading and the next, already cleaned (no running header, no page number, hyphenation resolved, one clause per line). Do not paraphrase it, do not summarise it, do not put LaTeX inline: the agent that reads it later must read the norm, not a rewrite. Formulas and tables are attached separately (step 7) and rendered at their anchor. - **`rawContent`** = the same `rawText`. - **`summary`**: 1 to 3 sentences on **chapters and X.Y nodes only**, stating what the section decides and the values it holds (this is what `get_toc` shows, so it drives navigation). Leave `summary` empty (`""`) on deeper nodes: they inherit their nearest ancestor's at read time. - Note the norm's **scope** in one sentence (from chapter 1) for the document record; it will feed the methodology. ### 6. Insert sections (batched) `ingest_create_sections { documentId, sections: [...] }` in batches of **150** (the tool accepts up to 200). Parent links resolve via `parentNodeId` within the batch and across prior batches, so send the nodes in document order. The call returns a `nodeId → sectionId` map; **use those `sectionId`s** for every enrichment call below. A batch is idempotent on `nodeId`: retrying it after a network error does not duplicate anything. ### 7. Enrich Open the PDF visually for the pages the pre-pass flagged with a formula or a table (PyMuPDF mangles both), and attach what you read by its number, so that the read tools can place it at its anchor in the text: - `ingest_attach_formula { sectionId, latex, description, formulaNumber }` where `formulaNumber` is the number printed in the margin, e.g. `"(9.12)"`. - `ingest_attach_table { sectionId, data: { headers, rows }, caption, tableNumber }` where `tableNumber` is the printed label, e.g. `"Tableau 3"`. - `ingest_attach_cross_ref { sourceSectionId, targetDocumentCode, targetSectionPath?, refText, refType }` only for references the automatic pass (step 9) cannot read, such as a reference written in prose without a clause number. Manual rows survive a later `ingest_normalize_cross_refs`. Attach a formula or a table to the **deepest** node whose text names it. Both calls are idempotent on their number: re-attaching updates the row. ### 8. Upload figures For each figure in `prepass.json#figures`: - Read `.stratta-ingest//` and base64-encode the bytes. - Find the owning section: the kept section whose `pageStart..pageEnd` range includes the figure's `page`. - `ingest_upload_figure { sectionId, base64, mimeType: "image/png", caption, figureNumber }` (≤ 8 MB per image). ### 9. Auto cross-references (always) `ingest_normalize_cross_refs { documentId }` reads every node's text for what it points at: other clauses, annexes, figures and tables of the same norm (FR, DE, IT, EN wordings) and other norms (SIA / SN EN / EN / ISO / DIN …), and resolves each internal target against the tree. Idempotent; manual attachments are kept. Its result tells how many internal references resolved: on a SIA norm expect most of them to. ### 10. Publish `ingest_publish { documentId }`. The server scores the document first (text per page, pages under a chapter, empty leaves, attached formulas and tables) and **refuses below 30/100** with the reasons. A refusal means the ingestion is wrong, almost always because `content` holds summaries instead of the text of the norm: go back to step 5, do not retry and do not pass `force`. `force: true` is for a document that really is that short, and only after the user said so. Show the returned `quality` (score and flags) to the user. The norm is now queryable in your workspace via `list_norms`, `get_toc`, `get_section`, `search_in_norm`, `get_figure`, etc. ## Quality bar - Trust the pre-pass for the tree and the text: `path`, `pageStart`/`pageEnd` and `rawText` are deterministic. Correct what `warnings[]` points at, and nothing else. - Every node is kept. The read tools address clauses; a folded clause cannot be cited. - `content` is the norm's text, never a rewrite. Values live in the text and in the attached tables and formulas, never in a summary. - Summaries on chapters and X.Y drive navigation: name what the section decides and the values it carries. - If PyMuPDF mangled a formula (Greek letters, fractions, exponents broken), re-read the relevant PDF page visually and attach proper LaTeX under its printed number. - Never skip annexes: they hold key numeric values (zones, coefficients, characteristic loads). - Read the score `ingest_publish` returns and show it to the user, flags included.