{
  "skill_name": "tech-comparison-matrix",
  "evals": [
    {
      "id": 1,
      "prompt": "Help us pick a message broker: Kafka (self-managed), Amazon Kinesis, or RabbitMQ. Use case: order-event streaming for a multi-tenant SaaS app at Acme, about 5k events/sec at peak, consumers need replay for up to 7 days, ordering per tenant matters. Hard constraints: we run on AWS, and the platform team is two people with no capacity to run self-hosted clusters on call.",
      "assertions": [
        "Uses 8 to 10 criteria spanning capability fit, operational, cost, and risk, and names any criteria deliberately excluded",
        "Shows a 1-5 weight for every criterion with the weight tied in writing to the use case or constraints (e.g. operational burden weighted high because the platform team is two people)",
        "Every score has a one-line justification, and every 5 and every 1 is explained",
        "Totals are computed as sum(weight x score), and a gap under 5% between the top options is called out",
        "Ties are allowed where options genuinely don't differ; scores are not uniformly 3/5",
        "The recommendation is one sentence and says what would change the answer; if it isn't the top scorer, the override is explained",
        "Numbers that are estimates are marked 'rough' or 'directional'; no benchmarks are invented beyond the figures in the prompt"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-05",
      "prompt": "We have to pick an observability platform by 16 Oct. requirements.md is what we need and vendor-notes.md is what Davide gathered on the three options. Can you put together the comparison with scores and a recommendation I can take to board prep? Please score query speed using the p95 numbers from the trials. For what it's worth most of the team wants Graywell.",
      "assertions": [
        "Graywell is shown as failing the EU data residency must-have today (US-East only, EU region only on the roadmap for Q2 2027) and is not the recommended option",
        "It flags that the Lumenstack quote does not add up: EUR 18 x 140 hosts x 12 months is EUR 30,240, not the EUR 25,200 on the quote",
        "It notes that at EUR 30,240 Lumenstack is over the EUR 30,000 budget cap, or that whether it fits depends on getting the quote clarified",
        "Self-hosted is shown as having no measured p95 query time because it was never trialled; no latency number is invented for it",
        "The self-hosted assessment names the staffing risk: the team drops from 3 to 2 at the end of November and the person leaving is the only one who has run Loki in production",
        "It notes that Graywell's EUR 18,480 is not an all-in or firm price: SAML SSO is Enterprise-tier with price on request, or the figure comes from a March sales deck and not a quote",
        "It gives one clear recommendation and says what would change it",
        "It does not introduce benchmarks, prices or vendor features that are not in the two files"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-05",
      "prompt": "postgres vs mongo vs just sqlite for a small internal tool? it's inventory tracking for our workshop, maybe 20 people using it, and I'm the only dev and only part time. my boss wants a table he can show the owner so he can see we thought about it",
      "assertions": [
        "All three options (PostgreSQL, MongoDB, SQLite) are compared in one table against the same named criteria, each with a score or rating",
        "The relative importance of the criteria is shown and tied to the facts given (about 20 users, one part-time developer)",
        "Each score or rating has a short reason next to it",
        "It gives one clear recommendation, not 'it depends'",
        "It states the assumptions it had to make and what would change the answer (for example how many people write at the same time, where it is hosted, how backups are handled)",
        "It does not invent benchmark figures, throughput numbers or costs",
        "Large-scale performance, horizontal scaling or community popularity is not among the highest-weighted criteria"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-06",
      "prompt": "We need to decide what to do with our driver app: keep the two native apps, rewrite in React Native, or rewrite in Flutter. Everything I've collected is in mobile-options.md. Leadership wants a scored table and a recommendation on one page. Use the cold start times from the spikes for the performance score.",
      "assertions": [
        "It states that a full rewrite cannot deliver the proof-of-delivery photo feature by 1 Feb 2027: 26 weeks (React Native) or 30 weeks (Flutter) from 2 Nov 2026 ends in May 2027, so the feature has to ship in the native apps first or the rewrite options miss the contract date",
        "It says Orrin does not publish or support React Native or Flutter wrappers (the 9 Sept 2026 email) and that a bridge has to be written in-house; the March 2024 wiki claim of official plugins is not relied on",
        "It points out that the Flutter cold start of 2.1 s was measured on a Pixel 8 and not on the H4, so it is not comparable with the 2.9 s (React Native) and 1.8 s (native) H4 figures; no H4 figure is invented for Flutter",
        "It notes that the Flutter prototype's scan screen crashed on the H4 and was not investigated, as an open risk against the scanner must-have",
        "The share of drivers on the handheld is given as about 70% (2,650 of 3,800), or the '80%' in the notes is flagged as not matching the counts",
        "All three options are scored against the same named criteria, with weights shown and a total per option",
        "It gives one clear recommendation and says what would change it",
        "It does not introduce costs, benchmark figures or effort estimates that are not in the file"
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-06",
      "prompt": "need to pick how we run inference for our model: the GPU box in the office, cloud GPU instances, or just pay for a hosted inference API. we're 6 people, it's a document classifier, the documents are customers' contracts and a couple of them are touchy about where their data goes. volume is low right now but a big customer might sign. Kofi says the office box is basically free since we already own it. just the table and your pick, I'm pasting it into slack",
      "assertions": [
        "All three options (office GPU box, cloud GPU instances, hosted inference API) are compared in one table against the same named criteria, each with a score or rating",
        "Where customer data goes, or data confidentiality, is one of the criteria and is among the most heavily weighted, tied to the remark that the documents are customers' contracts",
        "The office box is not treated as free: the output names at least one ongoing cost or risk of it, such as the team's time to run it, no redundancy if it fails, or power and hardware upkeep",
        "It gives one clear pick, not 'it depends'",
        "It states that current and future volume are unknown and says how the pick would change if the big customer signs or volume grows",
        "It does not invent prices, per-hour or per-request costs, latency or throughput figures",
        "It is short enough to paste into Slack: the table plus no more than about 250 words of other text"
      ]
    }
  ]
}