{
  "skill_name": "file-to-markdown",
  "evals": [
    {
      "id": 1,
      "prompt": "Northwind sent us their Q3 roadmap as northwind-q3.pptx (14 slides, a few charts, speaker notes on most slides). Turn it into markdown so we can critique it.",
      "assertions": [
        "Runs to_markdown.py on northwind-q3.pptx and writes a .md file",
        "Each slide is a '## Slide N: title' section, in slide order",
        "Speaker notes appear under their own slide, not collected at the end",
        "States at the top of the reply how many charts and images were dropped, using the script's counts",
        "Says that any critique cannot cover the chart contents because they were not converted",
        "Does not invent slide content, numbers or chart values that are not in the converted text"
      ]
    },
    {
      "id": 2,
      "prompt": "Here's acme-contract-summary.pdf, can you get the text out?",
      "assertions": [
        "Runs to_markdown.py on the PDF",
        "If pdftotext is missing, passes on the install step (brew install poppler / apt install poppler-utils) and does not attempt to read the PDF bytes",
        "If it converts, says the PDF gives text only: no headings or tables recovered",
        "Does not summarise or paraphrase in place of the converted text"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "I'm moving our crew policies onto the new wiki. Convert harbourline-shift-policy.docx into Markdown and save it as harbourline-shift-policy.md. Two more things: the wiki reviewer needs a list of anything that didn't survive the conversion, and tell me what the notice period for a swap is and what the night-sailing rates are, so I can put them in the summary box at the top of the page.",
      "assertions": [
        "Writes harbourline-shift-policy.md, with the document's title and section headings as Markdown headings and the overtime rates as a Markdown table",
        "On the notice period, it reports that the document has an unaccepted tracked change from 48 to 24 hours and a comment saying legal have not signed off the change; it does not give 24 hours (or 48) as the settled answer without mentioning this",
        "It says the night-sailing rates are not in the file (the document only refers to an Appendix B that is not included) and does not supply any night rates",
        "It points out that the catering assistant row does not add up (13.10 + 1.90 is 15.00, not 15.20), and does not silently change the figure in the Markdown",
        "Its list of what did not survive names the image (Figure 1) and the reviewer's comment, rather than claiming nothing was lost",
        "The crew portal link is kept as a working Markdown link to its address, and the bold and italic in the closing note are kept",
        "It does not invent text, rates, approvals or appendix content that are not in the .docx"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-18",
      "prompt": "hiya, our allotment treasurer sent me seed-swap-2026.csv for the newsletter. can you turn it into a table i can paste in? the newsletter thing takes markdown. just the table please, no essay",
      "assertions": [
        "Gives a Markdown table with all six seed varieties and, for each, the type, number of packets, donor plot and notes as in the file",
        "Values that contain commas (\"Tomato, cherry 'Sungold'\", \"Squash, butternut\" and the quoted notes) stay in one cell each, and no columns are shifted",
        "The blank line and the empty row in the file do not appear as empty rows in the table",
        "If the Total row is kept, it notes briefly that the packets add up to 36, not 38; it does not silently change any figure",
        "It does not invent varieties, numbers, donors or notes that are not in the file",
        "The reply is the table plus at most a couple of short lines, with no long explanation, and it delivers without asking questions first"
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-27",
      "prompt": "Our procurement lead just forwarded supplier_agreement_v3.docx and it still has Legal's markup in it. Can you turn it into Markdown for our Confluence page, and tell me which bits haven't actually been agreed yet? Please don't accept anything on my behalf.",
      "assertions": [
        "Writes a Markdown file with 'Master Supply Agreement' as the top heading and the five numbered sections ('1. Payment terms' through '5. Price schedule') as subheadings. The three supplier obligations are a bulleted list, 'EUR 250,000' is bold and 'gross negligence' is italic.",
        "Renders the price schedule as a Markdown table with the header Part number | Unit price | Minimum order quantity and the rows VP-110 / EUR 42.00 / 500 and VP-220 / EUR 57.50 / 250. Keeps 'supplier portal' as a Markdown link to https://supplier.example.com/terms.",
        "Shows the payment-terms change as pending rather than accepted: the deleted '30' and the inserted '60' are both visible and marked as a proposed change by J. Okafor. It does not present the clause as simply 'payable within 60 days' as if it were agreed.",
        "Shows the deleted sentence 'No termination fee applies.' (M. Lindqvist) as a pending deletion that is still visible and marked, not silently dropped. Shows the ISO 9001 obligation bullet as a pending insertion by J. Okafor, not as an agreed obligation.",
        "Surfaces R. Achterberg's comment on the EUR 250,000 liability cap (asking to raise it to EUR 500,000, marked 'Not agreed yet') as an open item, either inline or in a notes or dropped-items list.",
        "Answers the question directly with a short list of what is not yet agreed: exactly three tracked changes (30 to 60 days; the inserted ISO 9001 obligation; the deleted termination-fee sentence) and one open comment on the liability cap. It accepts or rejects none of them and leaves supplier_agreement_v3.docx unchanged.",
        "Saves the embedded logo image as an image file next to the Markdown (for example in a sibling folder) and links it from the Markdown with a relative image link.",
        "Invents nothing that is not in the document: no extra clauses, parties, prices, dates, authors or comments. Names, figures and the portal URL match the file exactly."
      ]
    },
    {
      "id": 6,
      "held_out": "2026-10-27",
      "prompt": "can u turn the offsite deck + numbers into md for the board pack. sarah says the notes are where the real story is. offsite_q3.pptx and numbers.xlsx, need it today thx",
      "assertions": [
        "Produces Markdown covering both files. The three slide titles 'Q3 Offsite: Where We Landed', 'Revenue by region' and 'APAC launch' appear in that order as headings, each with its bullets, and each slide's speaker notes are kept and clearly labelled as notes under the right slide.",
        "Converts both worksheets, each identified by its name: Regions as a Markdown table (EMEA 120 / 110, North America 140 / 130, LATAM 160 / 150, Total row) and Assumptions (FX rate GBP/EUR 1.17; Source: Finance close 10 Oct 2026).",
        "Reports that the bar chart on slide 2 is not reproduced as a chart, either as a dropped item or as a text summary. Any figures it gives for the chart are exactly the values cached in it (EMEA 120, North America 140, LATAM 160), and no chart values are made up.",
        "Flags that the Regions Total in B5 is the formula =SUM(B2:B4) with a stored value of 410, while the three rows sum to 420 (120+140+160). It treats the total as stale or inconsistent rather than silently showing 410 or quietly correcting it to 420. The Plan total of 390 does match its rows.",
        "Warns the user, beyond just copying the notes, that the slide 3 notes say the £250k APAC first-year target must not go into the board pack (finance hasn't signed it off; say 'target under review'). It neither leaves £250k in board-pack-ready output without a warning nor deletes it silently.",
        "Relays the 'real story' from the slide 2 notes, since the user pointed to them: North America beat plan only because of the one-off Kestrel Sports order (£35k), and without it NA is at 105 against a plan of 130.",
        "Invents nothing that is not in the files: no extra slides, regions, APAC actuals, prior-quarter LATAM figures, chart values or names beyond those present.",
        "Is the right size for a rushed request. It delivers the Markdown file(s) straight away with a brief summary of the flagged issues, not a long essay or a rewritten board pack, and does not hold up the work with clarifying questions."
      ]
    }
  ]
}