Evaluation methodology

Capability evidence, not code resemblance.

PromptGolf asks one question: did the submitted spec cause an agent to build the requested product capabilities? Every point must trace to positive, observable evidence.

Behavior

Examples prove named outcomes. State-machine traces exercise lifecycles. Property and fuzz cases cover broad input spaces without prescribing an implementation.

Spec completeness

A requirement tree connects every scored capability to the public task, a domain rule, and the evidence needed to satisfy it.

Artifact adapters

Framework-aware adapters build and start the submitted workspace, then expose its routes, controls, outputs, and commands through one canonical protocol.

How a challenge earns trust

  1. 01Write a public requirement tree and domain rules before evaluator code.
  2. 02Materialize each leaf as an observable example, state transition, property, or adapter capability.
  3. 03Run naive, structured, and expert specs repeatedly through the same generation and sandbox path.
  4. 04Keep evidence that consistently separates omitted requirements from specified capabilities; revise ambiguous checks.

Public contract, private cases

Players can inspect the requirement tree, evidence methods, capability vocabulary, scoring weights, and run results. Hidden evaluator source and test cases remain outside the builder workspace so the artifact must satisfy the capability rather than memorize the judge.

Outcome policy

PromptGolf does not use mutation or negative testing, implementation resemblance, source layout, fixed signatures where valid alternatives exist, CSS fingerprints, preferred methods, or model-specific prompt wording. Invalid and failure-state behavior can still earn positive evidence when the brief requires that observable capability.