Run a witness on Fly.io

• A mosskeys-witness container (our first-party Rust witness) running on one Fly Machine.
• Free HTTPS on a *.fly.dev hostname, with your DNSimple domain attached on top.
• A tiny Fly volume to persist keys/ and state/ across restarts and deploys.

Every step below is copy-pasteable. If you would rather understand the protocol and the hosting options first, start with the witness operator guide and come back here to deploy.

Pick a schema-less URL under a domain you control, for example witness.yourdomain.com/mosskeys. This gets baked into every cosignature and into both vkeys, so choose one you can keep permanently. Buy yourdomain.com at DNSimple now; you will point a subdomain at Fly in step 5.

This is the one step that runs on your machine; everything else runs in the container. Install the binary just for keygen, then mint the identity. This creates two independent keypairs, Ed25519 (0x04) and ML-DSA-44 (0x06), writing the secret seeds to ./keys/ed25519.seed and ./keys/mldsa44.seed (mode 0600) and printing the two public vkeys. Save both vkeys; you paste them into the application in step 7.

# Everything except keygen runs on Fly, so the binary is only needed for
# this one command. Any install option from the operator guide works:
#
#   curl -fsSL https://mosskeys.com/install.sh | sh -s -- witness
#   brew install moss-piglet/mosskeys-witness/mosskeys-witness
#   cargo install mosskeys-witness --locked

# One directory holds the build context: keys/ and witness.toml.
mkdir -p ~/mosskeys-witness && cd ~/mosskeys-witness

# Mint two independent keypairs, Ed25519 (0x04) and ML-DSA-44 (0x06).
# Seeds are written 0600 and never overwritten; only the public vkeys
# are printed.
mosskeys-witness keygen --name witness.yourdomain.com/mosskeys --out-dir ./keys

# Ed25519 vkey (public — register with logs):
#   witness.yourdomain.com/mosskeys+1a2b3c4d+BC...
#   seed file (secret, mode 0600): ./keys/ed25519.seed
#
# ML-DSA-44 vkey (public — register with logs):
#   witness.yourdomain.com/mosskeys+5e6f7a8b+Bg...
#   seed file (secret, mode 0600): ./keys/mldsa44.seed

# Save both vkeys: you paste them into the application in step 7.

A witness only cosigns logs it has been told about, so the config carries the origin and vkeys of each one. Those come from our public discovery feed. Run this once, anywhere; your laptop is fine.

curl https://mosskeys.com/api/witness/logs

{
  "logs": [
    {
      "origin": "mosskeys.com/acme",
      "vkeys": {
        "hybrid": "mosskeys.com/acme+<key id>+<base64 hybrid vkey>",
        "ed25519": "mosskeys.com/acme+<key id>+<base64 Ed25519 0x01 vkey>"
      }
    }
  ]
}

Each entry becomes one [[log]] stanza. Copy them by hand, or generate the stanzas straight from the feed:

curl -s https://mosskeys.com/api/witness/logs | jq -r \
  '.logs[] | "[[log]]\norigin = \"\(.origin)\"\nvkeys = [\n  \"\(.vkeys.hybrid)\",\n  \"\(.vkeys.ed25519)\",\n]\n"'

Save the following as witness.toml next to keys/, replacing the placeholder stanza with the real one(s) from above. Note the listen port, and that the seed and state paths are the ones inside the container, on the volume you create in step 3.

name = "witness.yourdomain.com/mosskeys"
listen = "0.0.0.0:8080"
state_file = "/data/state/state.jsonl"

[keys]
ed25519_seed = "/data/keys/ed25519.seed"
mldsa44_seed = "/data/keys/mldsa44.seed"

# One stanza per log you cosign, generated from the discovery feed
# above. Origins are exact: no wildcards, no prefixes. Anything else
# answers 404.
[[log]]
origin = "mosskeys.com/acme"
vkeys = [
  "mosskeys.com/acme+<key id>+<base64 hybrid vkey>",
  "mosskeys.com/acme+<key id>+<base64 Ed25519 0x01 vkey>",
]

Install flyctl, log in, and create the app without deploying yet. Then create a small persistent volume for keys and state, in the same region you will deploy to. Pick one that gives the network real infrastructure independence; where the machine runs is part of what the human review weighs. Every flyctl command in this guide names the app explicitly, so you can run them from any directory.

# Install flyctl and log in (https://fly.io/docs/flyctl/install).
curl -L https://fly.io/install.sh | sh
fly auth login

# Create the app without deploying yet.
fly apps create my-mosskeys-witness

# A small persistent volume for keys + state. 1 GB is the floor and far
# more than one state file will ever need. The region must match the one
# you deploy to: pick one that gives the network real infrastructure
# independence, because where the machine runs is part of what the human
# review weighs.
fly volumes create witness_data -a my-mosskeys-witness --region ams --size 1

The mosskeys image lives at ghcr.io/moss-piglet/mosskeys-witness:latest, listens on 8080, is FROM scratch, and runs non-root (uid 65532). Save this as fly.toml in the directory from step 1:

app = "my-mosskeys-witness"
primary_region = "ams"

# The wrapper Dockerfile below adds your witness.toml to the signed
# scratch image.
[build]
  dockerfile = "Dockerfile"

[http_service]
  internal_port = 8080
  force_https = true
  # A witness wants to be mostly online: scale-to-zero adds cold-start
  # latency and pointless retry churn.
  auto_stop_machines = "off"
  auto_start_machines = true
  min_machines_running = 1

# keys/ and state/ live here, so they survive restarts and deploys.
[[mounts]]
  source = "witness_data"
  destination = "/data"

# The floor: a small static binary plus one state file, pennies a month.
# Only bump memory if you see OOMs.
[[vm]]
  size = "shared-cpu-1x"
  memory = "256mb"

Because the image is scratch (no shell) and read-only-friendly, two things need handling.

Ship witness.toml in the image

Keys and state live on the volume under /data, matching the paths in your config. The config itself rides in with a tiny wrapper Dockerfile, saved next to fly.toml as Dockerfile (exactly that name, no leading dot). The [build] section above points at it instead of the image directly:

# The mosskeys-witness image is FROM scratch: no shell, read-only-friendly,
# runs as uid 65532. This wrapper only bakes in your witness.toml. The
# seeds stay out of the image and live on the volume.
FROM ghcr.io/moss-piglet/mosskeys-witness:latest
COPY witness.toml /witness.toml
CMD ["run", "--config", "/witness.toml"]

Copy the seeds onto the volume, once

Fly secrets cannot help here, because there is no shell in the image to write them out on first boot. The clean path is a throwaway machine that does have a shell, mounted on the volume from step 3:

# The scratch image has no shell, so `fly ssh console` has nothing to run
# on the witness machine itself. Borrow a throwaway alpine machine on the
# same volume, copy the seeds through it, then destroy it.
fly machine run alpine sleep 3600 -a my-mosskeys-witness --region ams --volume witness_data:/data

# Grab the throwaway machine's id.
fly machine list -a my-mosskeys-witness

# Write the seeds, base64 over ssh so nothing binary crosses the terminal.
# Each command is one full line: paste them one at a time, in order, and
# replace <id> with the machine id from the list output.
fly ssh console -a my-mosskeys-witness --machine <id> -C "mkdir -p /data/keys /data/state"
base64 < keys/ed25519.seed | fly ssh console -a my-mosskeys-witness --machine <id> -C "sh -c 'base64 -d > /data/keys/ed25519.seed'"
base64 < keys/mldsa44.seed | fly ssh console -a my-mosskeys-witness --machine <id> -C "sh -c 'base64 -d > /data/keys/mldsa44.seed'"

# The witness runs as uid 65532: it must read the seeds and write state.
# The seeds stay 0600, only the owner changes, so they remain owner-only.
fly ssh console -a my-mosskeys-witness --machine <id> -C "sh -c 'chown -R 65532:65532 /data && chmod 600 /data/keys/*.seed'"

# Confirm the seeds landed before tearing anything down: two files, mode
# 0600, owned by 65532.
fly ssh console -a my-mosskeys-witness --machine <id> -C "ls -l /data/keys"

# Tear the throwaway down. The seeds never touched the image.
fly machine destroy <id> -a my-mosskeys-witness --force

If you would rather avoid that dance entirely, bake the keys into the image build context on a private repo. That is acceptable for a personal witness, though volume-only storage is cleaner.

# From the directory holding fly.toml and the Dockerfile.
fly deploy

# Your free https endpoint is live: https://my-mosskeys-witness.fly.dev
fly status -a my-mosskeys-witness

Then attach your subdomain. mosskeys is your witness's client: it POSTs checkpoints to your endpoint over the public internet, so the app needs public IPs before a certificate can be issued. Allocate them first (a shared IPv4 is free and fine for HTTPS, and IPv6 is free), then tell Fly which hostname to serve. Fly prints the DNS records; you add them at DNSimple:

# mosskeys reaches your witness over the public internet, so the app needs
# public IPs before a certificate can be issued: the cert cannot go green
# until the IPs and the [http_service] from step 4 both exist. A shared
# IPv4 is free and fine for HTTPS; IPv6 is free.
fly ips allocate-v4 --shared -a my-mosskeys-witness
fly ips allocate-v6 -a my-mosskeys-witness

# Tell Fly which hostname to serve; it prints exactly which records to
# add at DNSimple.
fly certs create witness.yourdomain.com -a my-mosskeys-witness

# In the DNSimple control panel for yourdomain.com, add what Fly printed:
# an A record to the shared IPv4 and an AAAA record to the IPv6 (or a
# CNAME from witness to my-mosskeys-witness.fly.dev), plus the
# _acme-challenge CNAME used to validate the certificate.

# Watch the cert go green, usually a minute or two after the records
# resolve.
fly certs show witness.yourdomain.com -a my-mosskeys-witness

Once the cert goes green, your submission prefix is https://witness.yourdomain.com/mosskeys. mosskeys POSTs to <prefix>/add-checkpoint.

The correct pre-activation answer to an origin you do not cosign is a 404: it means the request parsed and only the allowlist refused it. An empty body giving a 400 proves nothing. Test it against your live domain:

# Healthy pre-activation answer to an origin you do not cosign: 404. The
# request parsed and only the allowlist refused it (an empty body is a
# 400, which proves nothing). This is exactly what the review probe looks
# for. The body is the tlog-witness wire format: "old 0", a blank line,
# the checkpoint, then its signature line, which really does start with
# an em-dash. The placeholder sig is never checked: the origin refusal is
# evaluated before any signature is even looked at.
curl -si -X POST --data-binary @- https://witness.yourdomain.com/mosskeys/add-checkpoint <<'EOF' | head -1
old 0

unknown.example/log
1
AARpcm88QZxj7jR9izc5sCygNRvIk0Ym2MCPmtKGxBk=

— unknown.example/log AAAAAAAA
EOF
# expected first line: HTTP/2 404

A 404 here is exactly what the mosskeys review probe looks for.

Fill in the application with who operates the witness, a contact email, the cosigner name (witness.yourdomain.com/mosskeys), the submission prefix, and both vkeys from step 1. For jurisdiction, give where the operator is legally based, not the Fly region; the machine's location is weighed separately as infrastructure. The form checks the vkeys cryptographically as you type; then a human reviews independence (operator, infrastructure, jurisdiction). Expect a few days.

On approval, mosskeys adds your vkeys to the submission registry and checkpoints start arriving on their own. Your listing shows on the public directory with a live cosignature count. Day to day there is nothing to do but stay reachable. Downtime is safe: submissions retry with backoff and resume from your last cosigned size. Updates are just a new image and a redeploy; your keys and state survive on the volume.

New namespaces come online over time, and the [[log]] stanzas baked into your image are the policy that decides what you cosign. Without anything further, tracking them means re-running the stanza generator from step 2 and redeploying each time. The zero-maintenance alternative is the in-process [discovery] section (mosskeys-witness 0.4.0 and later — the :latest tag this guide deploys is well past that): the witness polls the discovery feed itself, first at boot and then every interval, and a change hot-swaps the in-memory allowlist. No restart, no redeploy, no cron.

Add three lines to the witness.toml from step 2:

# Add once, then redeploy one last time. From here the witness polls the
# feed itself: first at boot, then every interval, and a change hot-swaps
# the in-memory allowlist. New origins start getting cosignatures within
# one interval, no restart and no cron. A failed poll is logged and
# non-fatal: the last known set keeps serving.
[discovery]
feed_url = "https://mosskeys.com/api/witness/logs"
interval_secs = 300   # optional; this is the default

The config rides in the image through the wrapper Dockerfile from step 4, so one last deploy picks the section up — then confirm the poller started:

# witness.toml rides in the image through the wrapper Dockerfile (step
# 4), so bake the new section in with one last deploy, from the
# directory holding fly.toml. Origin changes never need a deploy again.
fly deploy

# Confirm the poller started: it announces itself in the banner at
# boot, and every applied change logs one line. A failed poll is logged
# and non-fatal — the last known set keeps serving.
fly logs -a my-mosskeys-witness

#   discovery: polling https://mosskeys.com/api/witness/logs every 300s (hot-reload; failures keep the current set)
# mosskeys-witness: discovery update applied — allowlist now 2 origins (+1 -0, 0 rotated)

The managed discovered_logs.toml lands next to the state file — here /data/state/discovered_logs.toml, on the volume from step 3 — so the discovered set survives every deploy. The manual [[log]] stanzas from step 2 can stay or go: a manual stanza always wins over a managed entry for the same origin, so pin a log by hand if you want its vkey rotations to wait for your review. The other postures (hand-pasted stanzas, or a one-shot sync on a schedule) and the rules every posture shares are under Keep your origin set current in the operator guide.

• shared-cpu-1x / 256 MB is the floor. The witness is a small binary plus one state file, pennies a month. Only bump memory if you see OOMs.
• Keep min_machines_running = 1 and avoid auto_stop. A witness wants to be mostly online; scale-to-zero adds cold-start latency and pointless retry churn. Downtime will not break consistency, but always-on is the cheaper-in-effort default.
• One instance only. Never run two live instances off a copied state file. That is the one rule that can actually break the security property.
• Config freshness. Turn on the in-process discovery poller above and the origin set stays current on its own. Rather have every config change git-reviewed? Keep the baked witness.toml and let a scheduled GitHub Action regenerate it from the feed, running fly deploy only when the stanzas change.

• Run a witness: the protocol, the software options, and the apply-to-live flow behind this guide.
• Apply to witness: self-serve application, cryptographically self-validating vkeys, and human review for operator independence.
• Public witness directory: the live roster, jurisdictions, vkeys to pin, and observed cosignature counts.
• mosskeys-witness: the container image this guide deploys, with signed releases and the conformance checklist.
• Fly.io docs and DNSimple: the platform and registrar used throughout.