--- name: run tests by URL attributedTo: - https://bengo.is uuid: - 4d26a319-6c1f-490a-bcbf-21b7e3780ffd status: proposed --- # run tests by URL ―https://bengo.is ## Overview [Why statement](https://medium.com/olzzio/y-statements-10eb07b5a177): * In the context of * designing how `activitypub-testing` can run tests for behaviors specified by FEPs (not only core ActivityPub behaviors), * facing * the requirement that `activitypub-testing` must be able to run tests for FEPs, * we decided * to build an affordance to invoke `activitypub-testing` with a command to run a test indicated by URL, * to achieve the ability * to run non-core tests using activitypub-testing * accepting downsides * that the invoker will be able to pass URLs to tests that the test runner cannot parse and/or run. ## Context `activitypub-testing` is a test runner for ActivityPub tests. FEPs are Fediverse Enhancement Proposals, i.e. generally proposals that specify ActivityPub-adjacent behaviors. [bumblefudge.com made a test case for FEP-521a](https://codeberg.org/fediverse/fep/pulls/342). [bengo.is implemented the test case in JavaScript](https://codeberg.org/socialweb.coop/activitypub-testing-fep-521a/). ## Concern How can we make it so `activitypub-testing` CLI can run the test cases for FEP-521a? * until now, only run tests in src/activitypub-tests, and the only tests in there are for [behaviors](https://socialweb.coop/activitypub/behaviors/) specified by [ActivityPub][], not FEPs. ## Desired Outcomes 1. testers can invoke `activitypub-testing` in a way that runs tests for FEPs, e.g. the FEP-521a test [specified here](https://codeberg.org/socialweb.coop/activitypub-testing-fep-521a/src/branch/main/fep/521a/actor-objects-must-express-signing-key-as-assertionMethod-multikey.md) ## Candidate Solutions ### Maintain FEP tests in activitypub-testing Repo Pros * maintainers could vet every single test to make sure it works well with the activitypub-testing runner Cons * the maintainer's of activitypub-testing are not the best maintainers of all FEP tests * the repo PR queue would become a bottleneck, e.g. review could be too slow, or submitters may disagree with the review criteria and results ### Run tests by URL Acceptance Criteria * I can invoke activitypub-testing-cli like ```shell activitypub-testing run test \ --url=https://codeberg.org/socialweb.coop/activitypub-testing-fep-521a/raw/branch/main/fep/521a/actor-objects-must-express-signing-key-as-assertionMethod-multikey.js --input.actor='{}' ``` * the result of this test run should have outcome `inapplicable` because `input.actor` does not satisfy applicability requirements of the FEP-521a test * I can invoke the same test with a different input and get outcome `passed` Pros * avoids work of reviewing/maintaining FEP tests in the activitypub-testing repo Cons * maintainers can't vet every single test to make sure it works well with the activitypub-testing runner. It will be possible to run a test by URL that the runner can't run, e.g. because the referent of the URL is not in the required format * it will be possible to run tests by URL that do malicious things once fetched and executed [ActivityPub]: https://www.w3.org/TR/activitypub/ ## Decisions ### Implement "Run tests by URL" Candidate Solution [see above](#run-tests-by-url-1) ## Log ### Tue Jun 25 16:55:47 PDT 2024 Now you can `activitypub-teseting run test --url=...`. This satisfies [desired outcome](#desired-outcomes) #1 above: > testers can invoke `activitypub-testing` in a way that runs tests for FEPs #### Example of actor passing FEP-521a test: ```shell ⚡ activitypub-testing run test \ --url=https://codeberg.org/socialweb.coop/activitypub-testing-fep-521a/raw/branch/main/fep/521a/actor-objects-must-express-signing-key-as-assertionMethod-multikey.js \ --input.actor='{"type":"Person","assertionMethod":[{"type":"Multikey","id": "https://example.com/#ed25519-key","controller": "https://example.com/","publicKeyMultibase": "z6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2"}]}' | jq . \ { "type": [ "Assertion" ], "result": { "outcome": "passed", "pointer": { "results": [ { "target": { "actor": { "type": "Person", "assertionMethod": [ { "type": "Multikey", "id": "https://example.com/#ed25519-key", "controller": "https://example.com/", "publicKeyMultibase": "z6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2" } ] }, "assertionMethod": { "type": "Multikey", "id": "https://example.com/#ed25519-key", "controller": "https://example.com/", "publicKeyMultibase": "z6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2" } }, "result": { "outcome": "passed" } } ] } }, "test": { "slug": "fep-521a-actor-objects-must-express-signing-key-as-assertionmethod-multikey", "description": "This rule checks whether a given Actor Object has a Multikey object in top-level assertionMethod property of the shape specified in FEP-521a.", "name": "Actor Objects must express signing key as assertionMethod Multikey", "id": "urn:uuid:36f73f6e-8c14-4606-864d-32b9a74abc87", "requirementReference": [] }, "input": { "actor": "{\"type\":\"Person\",\"assertionMethod\":[{\"type\":\"Multikey\",\"id\": \"https://example.com/#ed25519-key\",\"controller\": \"https://example.com/\",\"publicKeyMultibase\": \"z6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2\"}]}" }, "@context": [ "https://socialweb.coop/ns/testing/context.json", "https://www.w3.org/ns/activitystreams" ] } ``` #### Example with helpful error If the `input.actor` has a slightly-wrong `publicKeyMultibase`, the test reports a helpful error message with the failure. ```shell ⚡ activitypub-testing run test --url=https://codeberg.org/socialweb.coop/activitypub-testing-fep-521a/raw/branch/main/fep/521a/actor-objects-must-express-signing-key-as-assertionMethod-multikey.js --input.actor='{"type":"Person","assertionMethod":[{"type":"Multikey","id": "https://example.com/#ed25519-key","controller": "https://example.com/","publicKeyMultibase": "6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2"}]}' | jq . { "type": [ "Assertion" ], "result": { "outcome": "failed", "pointer": { "results": [ { "target": { "actor": { "type": "Person", "assertionMethod": [ { "type": "Multikey", "id": "https://example.com/#ed25519-key", "controller": "https://example.com/", "publicKeyMultibase": "6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2" } ] }, "assertionMethod": { "type": "Multikey", "id": "https://example.com/#ed25519-key", "controller": "https://example.com/", "publicKeyMultibase": "6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2" } }, "result": { "outcome": "failed", "info": "assertionMethod is not a valid fep-521a multikey", "pointer": { "error": { "message": "assertionMethod.publicKeyMultibase MUST be a string startwith with 'z' (i.e. multibase-base58btc)", "cause": { "assertionMethod": { "type": "Multikey", "id": "https://example.com/#ed25519-key", "controller": "https://example.com/", "publicKeyMultibase": "6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2" } } } } } } ] } }, "test": { "slug": "fep-521a-actor-objects-must-express-signing-key-as-assertionmethod-multikey", "description": "This rule checks whether a given Actor Object has a Multikey object in top-level assertionMethod property of the shape specified in FEP-521a.", "name": "Actor Objects must express signing key as assertionMethod Multikey", "id": "urn:uuid:36f73f6e-8c14-4606-864d-32b9a74abc87", "requirementReference": [] }, "input": { "actor": "{\"type\":\"Person\",\"assertionMethod\":[{\"type\":\"Multikey\",\"id\": \"https://example.com/#ed25519-key\",\"controller\": \"https://example.com/\",\"publicKeyMultibase\": \"6MkrJVnaZkeFzdQyMZu1cgjg7k1pZZ6pvBQ7XJPt4swbTQ2\"}]}" }, "@context": [ "https://socialweb.coop/ns/testing/context.json", "https://www.w3.org/ns/activitystreams" ] } ```