# Graphiti CLI Evaluation Task You are evaluating the **graphiti** CLI — a progressive GraphQL query builder for Salesforce orgs. Your job is to use it end-to-end to build every query two real-world apps would need, then reflect on the experience and ship improvements. Start by reading `AGENT_GUIDE.md` in this repo to learn the full command surface. Use `vscodeOrg` as the org alias for all commands. Run graphiti via `npx graphiti` from the repo root. --- ## Phase 1: Build Queries for an Order Management App **App prompt:** > I need an order management app for sales and fulfillment staff. The home screen should show a breakdown of orders by status. There should be a product catalog page where staff can browse and search products. The orders page should let users filter by status and date. Clicking an order should open a detail view showing the order header and all line items with product name, quantity, and price. Staff should be able to update the order status, add line items, and remove line items. There should also be a way to create a new order by selecting an account and adding products. ### Instructions 1. Use `npx graphiti describe` to explore the relevant SObjects (Order, OrderItem, Product2, PricebookEntry, Account, etc.) and understand available fields, picklist values, and relationships. 2. For each screen / feature below, create a named session and build the complete query using graphiti CLI commands (`new`, `select`, `set`, `var`, `check`, `show`). 3. Run `check` on every query to validate it. 4. Run `show` on every query to capture the final GraphQL. 5. Run `codegen` on each query to generate TypeScript types. **Queries to build:** | # | Session Name | Purpose | | --- | --------------------- | ----------------------------------------------------------------------- | | 1 | `order-home-status` | Home screen — count/list orders grouped by status | | 2 | `product-catalog` | Product catalog — browse/search products with pagination | | 3 | `order-list` | Order list — filterable by status and date, sortable | | 4 | `order-detail` | Order detail — header fields + line items with product name, qty, price | | 5 | `order-update-status` | Mutation — update order status | | 6 | `order-add-line` | Mutation — add a line item to an order | | 7 | `order-remove-line` | Mutation — remove a line item | | 8 | `order-create` | Mutation — create a new order for an account | | 9 | `account-lookup` | Account lookup/search for order creation | --- ## Phase 2: Build Queries for a Support Queue App **App prompt:** > I need a support queue React app for our support team. The home view should have tiles showing case counts by status — New, In Progress, Waiting on Customer, Resolved — plus a list of unassigned cases and a list of cases assigned to me. There should be a case list page with filters for status, priority, and owner, sortable by date and priority. Opening a case should show the full case details including the related account and contact. From the detail view, agents should be able to update the case status and priority, assign it to themselves, and add a note. There should also be a form to create a new case with account and contact lookups. Show the current user's name in the header. ### Instructions Same workflow as Phase 1. Use `describe` to explore, then build each query. **Queries to build:** | # | Session Name | Purpose | | --- | ---------------------- | -------------------------------------------------------------------------- | | 1 | `case-home-new` | Home tile — cases with Status = New | | 2 | `case-home-inprogress` | Home tile — cases with Status = In Progress | | 3 | `case-home-waiting` | Home tile — cases with Status = Waiting on Customer | | 4 | `case-home-resolved` | Home tile — cases with Status = Resolved | | 5 | `case-home-unassigned` | Home — unassigned cases list | | 6 | `case-home-mine` | Home — cases assigned to current user | | 7 | `case-list` | Case list — filterable by status/priority/owner, sortable by date/priority | | 8 | `case-detail` | Case detail — full case with related account and contact | | 9 | `case-update` | Mutation — update case status and priority | | 10 | `case-assign` | Mutation — assign case to current user | | 11 | `case-add-note` | Mutation — add a CaseComment / note | | 12 | `case-create` | Mutation — create a new case with account and contact | | 13 | `account-search` | Account lookup for case creation | | 14 | `contact-search` | Contact lookup for case creation | | 15 | `current-user` | Header — current user's name | --- ## Phase 3: Validate TypeScript Type Accuracy The generated TypeScript types are consumed by agents and developers to build app code against query results. Type accuracy is critical — wrong types cause runtime errors, missing optionality causes null-pointer crashes, and weak types (`Record`) force consumers to cast blindly. Run `codegen` on every query built in Phases 1 and 2 and audit the output for the following: ### 3a. Result type accuracy For each query, check that the generated `*Result` interface accurately reflects the GraphQL response shape: 1. **Field names match the query.** Aliased fields should use the alias name, not the underlying field name. If you selected `Subject.value:subject`, the type should have `subject: string`, not `value: string`. 2. **Nullability is correct.** Parent relationship fields (Account, Contact, Owner, CreatedBy) should be `| null`. Connection types (edges, pageInfo) should NOT be nullable. Scalar value wrappers (`{ value: string }`) should be `| null` when the field is optional in the schema. 3. **Picklist values are union types, not plain `string`.** `Status.value` should be `CaseStatus` (a union of the actual picklist values), not `string`. Verify picklist unions contain the correct values from the org. 4. **Union/polymorphic types are represented.** `Owner` (a union of `User | Group`) should have the inline fragment structure, not a flat object. Both branches should be present. 5. **Connection pagination types are present.** `pageInfo { hasNextPage, endCursor }` and `totalCount` should be typed when selected. 6. **Nested child relationships are typed.** `CaseComments` inside a Case should have its own connection structure with edges/node, not be flattened. ### 3b. Variable type accuracy For each query with variables, check that the generated `*Variables` interface is strongly typed: 1. **Filter variables use the real filter type.** `$where` should be typed as `Case_Filter` (with all Case field names and operator types), NOT `Record`. 2. **OrderBy variables use the real orderBy type.** `$orderBy` should be `Case_OrderBy` with field-level `OrderByClause` entries and `ResultOrder` (`"ASC" | "DESC"`) enums. 3. **Mutation input variables are fully expanded.** `$input: CaseCreateInput` should expand to show `{ Case: CaseCreateRepresentation }` with all settable fields (`Subject?: string`, `Status?: string`, `AccountId?: string`, etc.), NOT `Record`. 4. **Operator types are expanded.** Filter fields should have concrete operator interfaces (`PicklistOperators { eq?: string; ne?: string; in?: string[]; ... }`), not `Record`. 5. **Scalar variables use correct primitives.** `$first` should be `number`, `$after` should be `string`, `$caseId` should be `string`. 6. **Required vs optional is correct.** Variables with `!` in their GraphQL type (e.g. `$input: CaseCreateInput!`) should be required in the interface (no `?`). Variables without `!` should be optional (`?`) and include `| null`. ### 3c. Picklist type accuracy (inputs and results) Picklists are a critical pain point. An agent building a case creation form, a status filter, or a priority dropdown needs to know the exact valid values — not just `string`. Audit every picklist field across both result types and input/variable types: #### Result-side picklists 1. **Every picklist field in a result type should use a union type alias.** `Status.value` should be `CaseStatus` (not `string`), `Priority.value` should be `CasePriority`, `Origin.value` should be `CaseOrigin`, etc. 2. **The union type should contain the actual org values.** Run `describe ` and cross-reference the picklist values listed there against the generated union. They must match exactly. If the org has `Status` values `["New", "On Hold", "Escalated", "Closed"]`, the type must be `"New" | "On Hold" | "Escalated" | "Closed"` — not a subset, not a superset. 3. **Picklist fields on related objects should also be enriched.** If case-detail selects `Account.Industry.value`, that should be `AccountIndustry`, not `string`. #### Input-side picklists 4. **Mutation input picklist fields should use the same union type or be constrained.** In `CaseCreateRepresentation`, `Status?: string` is too weak — an agent could pass `"Invalid"` and only discover the error at runtime. Verify whether codegen constrains these to the picklist union (e.g. `Status?: CaseStatus`). If it doesn't, document this as a type accuracy issue. 5. **Filter operator types for picklist fields should indicate valid values.** In `Case_Filter`, `Status?: PicklistOperators` where `PicklistOperators { eq?: string }` means an agent filtering by status has no autocomplete for valid values. Check whether the filter operator type carries the picklist constraint. If `eq` is just `string`, document this. 6. **OrderBy fields for picklists should be fine** (they only need `ASC`/`DESC`), but verify they aren't incorrectly typed. #### What to test Pick at least 3 queries that touch picklist fields and verify both sides: - `case-list` or `case-home-*` — `Status` and `Priority` in both the result type AND the `$where` filter variable type - `case-create` — `Status`, `Priority`, `Origin`, `Reason`, `Type` in the `CaseCreateRepresentation` input - `case-detail` — `Status` and `Priority` in the result, plus any picklist on related Account or Contact - `order-list` or `order-detail` — `Status` on Order (if it's a picklist) For each picklist field, note: - The field name and which SObject it belongs to - Whether the result type uses a union alias (good) or plain `string` (bad) - Whether the input/filter type uses a union alias, plain `string`, or `Record` - The actual valid values from `describe` ### 3d. @optional directive handling For any fields selected with `@optional` (FLS safety), verify: 1. **The field is marked optional in the TypeScript type** (`fieldName?: Type | undefined`). 2. **Fields without `@optional` are NOT marked optional** (no spurious `?`). 3. If no fields use `@optional`, add it to at least 2 fields on one query (e.g. `optional Description Subject` on case-detail) and verify the codegen output changes correctly. ### 3e. Agent usability audit Pretend you are an agent writing React components that consume these types. For each query, verify: 1. **Can you write a data access expression without `as` casts?** E.g. `data.uiapi.query.Case.edges[0].node.Status.value` should typecheck without casting. If you need `as unknown as X` anywhere, the type is wrong. 2. **Do mutation input types give autocomplete?** When constructing a `CreateCaseVariables` value, do you get field name suggestions for the input object, or is it `Record` where anything goes? 3. **Are there any `unknown` types that should be concrete?** Scan for `unknown` in the output — each one is a place where an agent would have to guess or hardcode. ### What to do with findings - Document every type accuracy issue in your reflection (Phase 4) under a new **"Type Accuracy Issues"** section. - For each issue, note: the session name, the field/variable, what the type IS vs. what it SHOULD be. - In Phase 5 (Improve), prioritize type accuracy fixes alongside other improvements. Type accuracy issues that affect agent usability should be treated as high priority. --- ## Phase 4: Reflect After completing Phases 1–3, write a structured reflection. Create or update a file called `REFLECTION.md` in the repo root with the following sections: ### What Went Well - Which commands were intuitive and efficient? - Where did the CLI save time vs. writing raw GraphQL? - What patterns emerged that felt natural? ### What Was Painful - Which tasks required too many commands or felt clunky? - Where did you get stuck, hit confusing errors, or have to guess? - What information was missing or hard to discover? - Were there queries you couldn't build, or that required awkward workarounds? ### Type Accuracy Issues For each codegen issue found in Phase 3, document: - **Session:** which query/mutation - **Field/Variable:** the specific field or variable name - **Current type:** what codegen produces now - **Expected type:** what it should produce for correct agent consumption - **Impact:** how this would affect an agent writing code against the type (e.g. "agent would need an `as` cast", "agent gets no autocomplete for mutation fields", "agent could pass invalid status value") ### Missing Capabilities - What commands or features would have made the process significantly easier? - Are there common query patterns that should be first-class operations? ### Friction Log For each moment of friction, note: - The exact command(s) you ran - What you expected to happen - What actually happened - How you worked around it (if you did) --- ## Phase 5: Improve the CLI Based on your reflection, do the following: 1. **Prioritize**: Pick the top 3–5 improvements that would have the highest impact on making GraphQL query authoring seamless for both humans and AI agents. **Type accuracy issues that affect agent usability should be weighted heavily** — an agent that can trust the types will write correct code on the first try. 2. **Plan**: For each improvement, write a brief design (what changes, where in the codebase, any new commands or flags). 3. **Implement**: Make the code changes. Run `npm run build` and `npm test` after each change to verify nothing breaks. 4. **Verify**: Re-run `codegen` on the affected queries and confirm the types are now correct. For type accuracy fixes, write out the before/after comparison in the reflection. ### Simplicity discipline Before adding ANY new flag, command, or shorthand, ask yourself: 1. **Can this be solved by better error messages or help text instead?** A self-documenting CLI that tells you what to do next when something goes wrong is more valuable than a shorthand that saves one command. If the existing commands can do the job in 2-3 steps and the error messages guide you there, that's good enough — don't add a flag. 2. **Can this be solved by improving existing command behavior?** Auto-correction (like auto-injecting `Record/` for mutations or `@args/` for set) is better than adding a new flag. Making the existing command smarter is always preferable to adding surface area. 3. **Does this new flag compose well, or does it create combinatorial complexity?** A command like `clone --set --unset --var --select --force` has 5 flags that interact with each other. Each new flag multiplies the testing/documentation burden. If a flag only makes sense in combination with other flags, it's probably not pulling its weight. 4. **Would better `--help` output or AGENT_GUIDE.md documentation solve this instead?** An agent reads `--help` before every command. Rich help text with examples is consumed every time; a niche flag is used rarely. Invest in documentation over features. 5. **Is this a real workflow pattern or a one-off?** If you only needed this once across 24 queries, it's not worth a flag. If you needed it on 10+ queries, it might be. **Strongly prefer these improvement categories (in order):** 1. Better error messages that show the correct command to run 2. Smarter auto-correction in existing commands 3. Richer `--help` output with real examples 4. Better AGENT_GUIDE.md documentation 5. Type accuracy improvements in `codegen` 6. New flags or commands (last resort — justify why 1-5 aren't sufficient) Remember the core tenets of this CLI: - Make GraphQL query authoring as seamless as possible for humans - Make GraphQL query authoring as seamless as possible for AI agents - The schema is the source of truth — navigate it, don't memorize it - Progressive disclosure — simple things should be simple, complex things should be possible - **Generated types are a contract** — agents and developers depend on them being accurate - **Simplicity is a feature** — every new flag is a maintenance burden and a learning burden. Fewer commands that work well beats many commands that overlap. Self-documenting behavior (good errors, good help) beats hidden shortcuts. --- ## Rules - Use only `npx graphiti` commands and the shell. Do not manually write GraphQL strings. - Do not ask the user for input at any point. Make reasonable decisions and move forward. - If a command fails, read the error, adjust, and retry. Document the failure in your reflection. - If a mutation type doesn't exist in the schema, note it in the reflection and move on. - Run `npm run build && npm test` before and after making any code changes.