Dwarf SocietyDwarf Fortress benchmark Public · read-only

LLM × harness early-fortress benchmark

Compare how bounded agents build after embark.

The accepted first-fortress-v1 objective asks each model and harness pairing to establish a functioning early fortress through player-equivalent typed actions.

Open comparison Inspect prior evidence
Broad runtime · unverified No published matrix runs Paired cohorts
Checked Stage 1 result showing seven of seven founders alive after 14,400 ticks, 51 food and 28 drink produced, a 2,640-tick holdout, and a stale revision rejected without mutation.
Retained Stage 1 evidence · PASS · 7/7 alive Checked predecessor evidence, not a first-fortress-v1 result · full digests

Accepted comparison

One broad objective, controlled pairings.

Each episode starts from the same pinned, pristine post-embark candidate only after its native state is verified. The model chooses the fortress strategy; the harness exposes the same bounded observation and action contract.

ObjectiveEstablish a functioning early fortress

Excavate, organize stockpiles, create shelter, farm, and produce useful goods while keeping the settlement stable.

ControlsPinned and paired

Scenario identity, model and harness revisions, action contract, budgets, repetitions, and scheduling remain cohort-bound.

OutcomeEight reported dimensions

Survival, food and drink, shelter, excavation, stockpiles, production, costs, and stability stay visible beside usage.

Read the protocol and inspect matrix evidence

Retained evidence

Checked predecessor stages

Stage 1 and Stage 2 remain valid demonstrations of typed actions and strict replay. They are not rows in the model × harness comparison.

Stage 1 · v2 ✓ PASS

Founding stability

Objective Keep seven founders alive, sheltered, fed, and hydrated; complete two meal and two drink events.

Checked horizon
14,400 ticks · objective at 11,760 · holdout 2,640
Outcome
7/7 alive · +51 food · +28 drink
Boundary proof
Stale revision rejected · applied false
Inspect the canonical episode
Stage 2 · contract 1 ✓ ACCEPTED

Farming + workshop

Objective Place and configure a directional farm, complete native planting, and produce the requested bed or barrel.

Checked horizon
4 scenarios · accepted at tick 7,200 · holdout 1,200
Outcome
2 × 2 farm · 4 seed jobs · barrel +1
Boundary proof
Unsupported query · 0 actions · failed outcome replayed
Inspect accepted and failed proof

Identity boundary Terrain survey establishes terrain conformance; the prepared Stage 1/2 founding fixture is not represented as a fresh post-embark start.

Evidence path

Observe, act, replay.

  1. 01Observe

    Paused frame + typed state

  2. 02Decide

    Versioned query or action

  3. 03Validate

    Identity, frame, revision

  4. 04Apply + receipt

    Apply once; record consequence

  5. 05Evaluate

    Outcome, holdout, cleanup

  6. 06Replay

    Verify chains + hashes

Read the comparison protocol

Verify / boundaries

Verify and inspect

Check public evidence

python3 scripts/verify-public-evidence.py

Expected final lineBoundary: static evidence only; native evaluator not started

Proves

  • Bounded typed actions in real Classic Dwarf Fortress for retained Stage 1/2 evidence
  • Checked predecessor outcomes with strict replay and cleanup

Does not prove

  • Full first-fortress-v1 gameplay availability or outcome yet
  • A stable model/harness ranking or generalization beyond matched cohort identities
  • Hosted submissions or live game control