#!/usr/bin/env python3
"""readiness-l0.py — reference tester for "Agent-commerce readiness, Layer 0: reachable" (0.2).

CC0. Python standard library only; curl and node are used as extra client
profiles IF they are on PATH, and reported UNOBSERVED if they are not; `dig` or
`nslookup`, if on PATH, read MX records for the L0-6 mailbox rows (without them
the MX half of that row is printed unobserved, never guessed).

usage: readiness-l0.py targets.json [--json out.json] [--known-member <url>=<PASS|FAIL>]
       readiness-l0.py --parity <recorded-ans.json>
       readiness-l0.py --lp-fixtures <fixtures.json>

targets.json:
  {"commerce":  [{"url": "...", "method": "POST", "body": "{}", "source": "where the business published this URL"}],
   "documents": [{"url": "https://host/llms.txt", "source": "..."}]}
  /robots.txt of every commerce host is added to the documents by the tester.

Result words: PASS, FAIL, UNOBSERVED, MISFRAMED. A timeout, a tester-side error
or a client that could not run is UNOBSERVED, never PASS. Any UNOBSERVED and no
FAIL => the layer is INCOMPLETE. A URL the tester guessed is out of scope:
every target carries its source, and the report prints it.

Exit code (part of the verdict, holy-hermes c80513): 0 PASS · 1 FAIL ·
3 MISFRAMED · 4 INCOMPLETE · 5 a --known-member control disagreed ·
6 REFUSED (a frame without a profile_id was handed to --parity) · 2 usage.

tester 0.2.0 (2026-10-07, readiness-l0 0.2 slot 5). What changed from 0.1.4,
each item with the handle it is credited to on /readiness-l0.html:
 (1) PROFILE IN EVERY FRAME (dash-agent c85371/c85430, bankr-mikk0x): every
     `sent` frame carries `profile_id` = "l0-headers-lp-0.2" and `codec_digest`
     = sha256 of this tester's own bytes as it ran. A recorded frame with no
     profile_id is the REFUSAL ROW: `--parity` prints the recorded codes beside
     a verdict of REFUSED and never grades it — a 0.1.4 frame replays under
     tester 0.1.4, not here.
 (2) HEADERS PREIMAGE `headers_lp_sha256` (bankr-mikk0x c81495; pennyforge
     c83605 Q1/Q2): varint(n) || per pair varint(len(name)) || name ||
     varint(len(value)) || value, UTF-8, unsigned LEB128; names lower-cased,
     STABLE-sorted by name (same-name pairs keep wire order, never joined);
     each value's outer SP/HTAB trimmed, inner kept. This tester never emits
     `headers_sha256`, so a 0.1.4 frame cannot pass as a 0.2 one. Reference
     function and four fixtures: /fixtures/l0/headers-lp/; `--lp-fixtures`
     recomputes them from this file and exits 1 on any mismatch.
 (3) PROFILE = (transport, UA) pair (0.2 notes 2026-10-01): the report's
     profile list names both for every client; a FAIL row carries `bisect`
     ∈ {name, shape, not-bisected} — this tester does not bisect, and says so.
 (4) THE BYTES BEFORE THE GUESS (@agentjeanclaude, X 2026-09-24): every
     per-profile answer keeps `status`, `ctype`, `content_length`, `body_len`,
     `body_sha256` and `body_head` (the first 160 bytes, verbatim) in the row,
     and the tester's reading of why stands in a separate `detail` field.
 (5) L0-1 `tls_error` CLASS beside the verdict (KSplit c86029; candidate, the
     page's vocabulary, objection window to 2026-10-09): no-resolve, refused,
     timeout, chain, hostname, expired, not-yet-valid, other — the library's
     own error named from its verify code, never inferred; `detail` keeps the
     library's text verbatim.
 (6) L0-4 UNDECLARED VERB → LAYER 1 (decided 2026-10-01; objectpermanence +
     izanami post 6048; kilmon-ai c68392 edge-confound rule): the cross-method
     probe still runs and prints its own `sent`, but its row is `L1-undeclared-verb`
     with `layer: 1` and does not enter the layer-0 verdict or counts. It prints
     TWO readings: `reading_0_1` (0.1's same-answer branch, PASS on a matching
     status) and `reading_strict` (a verb the door never declared answering the
     declared verb's 402 = FAIL: a verb that cannot settle must not be priced) —
     the negative control is the second reading, and layer 1 decides between
     them. Edge confound: when the probe client's answer to the DECLARED verb
     already differs from every other profile's, the row prints
     "edge-confounded, origin untested" = UNOBSERVED. The layer-0 L0-4 row is
     the declared verb itself: FAIL when the declared method answers 405 or
     501 on its own route.
 (7) L0-5 PER PROFILE (Reed 2026-09-20; fourth operator's run bf191877;
     agentic-qa c74910): each document is read whole (to 512 KiB) by every
     profile; the body-kind clause reads each profile's bytes — a 2xx whose
     body is an HTML page where the document is not HTML is not the document:
     FAIL for a URL the business published, UNOBSERVED-with-reason for one the
     tester added; a body shorter than its declared Content-Length is
     UNOBSERVED-with-reason (slot 8 decides whether that becomes FAIL). The
     robots rule is parsed once per profile from the body THAT profile was
     served; PASS only if every observed profile's `User-agent: *` result
     allows every commerce URL.
 (8) L0-6 DECLARED CHANNEL RESOLVES (pennyforge site comment 2026-09-30;
     ponytail c88988 both objections adopted): one row per declared channel,
     gathered from openapi `info.contact` (email, url), `/.well-known/x402`
     (any contact/email/support field), `/llms.txt` (mailto: and e-mail
     addresses) and `/.well-known/security.txt` (Contact:). A mailbox row
     resolves on an MX, or on A/AAAA as the RFC 5321 fallback written
     `resolves: a-only`; a null MX (RFC 7505) or no DNS at all is FAIL. A URL
     row is read by the default profile with its OWN opener that follows
     redirects, records every hop's status and the `final_url`, and resolves
     on a 2xx. A door that declares nothing is UNOBSERVED (nothing declared).
     PASS only when every declared row resolves.
 (9) TWO CONTROLS FOR A BATCH READ (pentimento c91472, kilmon-ai c91224): a
     run over more than one host prints a `batch_controls` block; the
     known-member control is run when `--known-member <url>=<PASS|FAIL>` names
     a row the reader did not choose from the output, and a disagreement
     makes the whole report CONTROL-FAILED (exit 5). The busy-row arithmetic
     control is named as not run (this tester reads no chain).
Still unproven from the output: that a profile SENT what it reports — the
`sent` frame is the tester's own account of the request it built. A
receiving-side proof (an echo door) is not built; 0.2 says so.
"""
import hashlib, json, os, re, shutil, socket, ssl, subprocess, sys, time, urllib.request, urllib.error, urllib.robotparser
from urllib.parse import urlsplit, urljoin

TESTER = "0.2.0"
SPEC = "readiness-l0 0.2"
PROFILE_ID = "l0-headers-lp-0.2"
TIMEOUT = 20
BROWSER_UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
CHALLENGE_MARKS = ("cf-chl", "challenge-platform", "/cdn-cgi/challenge", "g-recaptcha", "h-captcha", "just a moment", "enable javascript and cookies")
ROBOTS_LIMIT = 512 * 1024  # RFC 9309 2.5: a parsing limit must be at least 500 KiB
DOC_LIMIT = ROBOTS_LIMIT
HEAD_LIMIT = 4096
BODY_HEAD = 160
MAX_HOPS = 10

def now(): return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())

def codec_digest():
    """sha256 of this tester's own bytes as it ran — the `codec_digest` every frame carries."""
    try:
        with open(os.path.abspath(__file__), "rb") as f: return hashlib.sha256(f.read()).hexdigest()
    except Exception as e:
        return f"unreadable: {type(e).__name__}"
CODEC_DIGEST = codec_digest()

# ---- headers preimage, 0.2 (length-prefixed pairs) ---------------------------------------
def varint(n):
    out = bytearray()
    while True:
        b = n & 0x7f; n >>= 7
        if n: out.append(b | 0x80)
        else: out.append(b); break
    return bytes(out)

def trim_outer(v): return re.sub(r"^[ \t]+|[ \t]+$", "", str(v))

def norm_pairs(pairs):
    """lower-case names, STABLE sort by name (same-name pairs keep wire order), trim outer SP/HTAB on values."""
    return [[n, v] for n, v in sorted(((str(n).lower(), trim_outer(v)) for n, v in pairs), key=lambda p: p[0])]

def lp_preimage(pairs):
    p = norm_pairs(pairs); out = bytearray(varint(len(p)))
    for n, v in p:
        nb, vb = n.encode("utf-8"), v.encode("utf-8")
        out += varint(len(nb)) + nb + varint(len(vb)) + vb
    return bytes(out)

def headers_lp_sha256(pairs): return hashlib.sha256(lp_preimage(pairs)).hexdigest()

def sent_of(url, method, body, headers=None):
    """What a client was told to put on the wire, hashed so a stranger can check
    that every profile in a parity row was handed the same request. body_sha256 is
    over the sent bytes, or over the empty string when there is no body. `headers`
    are the pairs the tester itself authored for this row, normalized as the 0.2
    preimage normalizes them; `headers_lp_sha256` is the 0.2 preimage hash. Headers
    a client adds by itself (User-Agent, Accept-Encoding, Host) are NOT here: they
    are the variable under test, not part of the question. `profile_id` and
    `codec_digest` name the canonicalization this frame was computed under."""
    sp = urlsplit(url)
    path = sp.path + (("?" + sp.query) if sp.query else "")
    raw = body.encode() if body is not None else b""
    pairs = list((headers or {}).items())
    return {"method": method, "path": path, "body_sha256": hashlib.sha256(raw).hexdigest(),
            "headers": norm_pairs(pairs), "headers_lp_sha256": headers_lp_sha256(pairs),
            "profile_id": PROFILE_ID, "codec_digest": CODEC_DIGEST}

# ---- answers: the bytes before the guess ----------------------------------------------------
def answer(status, hd, raw, sent, limit):
    """One profile's answer. `raw` is what was read (to `limit`+1 bytes); the row keeps the
    bytes' hash and the first BODY_HEAD bytes verbatim, so the reading of WHY never stands
    in for them. `ctype`/`content_length` are the headers as served."""
    get = (lambda k: hd.get(k)) if hasattr(hd, "get") else (lambda k: None)
    cl = get("Content-Length") or get("content-length")
    body = raw[:limit]
    return {"status": status, "ctype": (get("Content-Type") or get("content-type") or "").lower(), "allow": get("Allow") or get("allow"),
            "content_length": int(cl) if (cl and str(cl).strip().isdigit()) else None, "body_len": len(body), "over_limit": len(raw) > limit,
            "body_sha256": hashlib.sha256(body).hexdigest(), "body_head": body[:BODY_HEAD].decode("utf-8", "replace"),
            "body": body.decode("utf-8", "replace"), "sent": sent}

class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *a, **k): return None  # a redirect is an answer; report it, do not follow it

def via_urllib(url, method, body, headers, ua=None, limit=HEAD_LIMIT):
    h = dict(headers)
    if ua: h["User-Agent"] = ua
    req = urllib.request.Request(url, data=(body.encode() if body is not None else None), method=method, headers=h)
    op = urllib.request.build_opener(NoRedirect)
    sent = sent_of(url, method, body, headers)  # `headers` = the authored set; a UA added above is the profile's own
    try:
        r = op.open(req, timeout=TIMEOUT); code, hd, b = r.status, r.headers, r.read(limit + 1)
    except urllib.error.HTTPError as e:
        code, hd, b = e.code, e.headers, e.read(limit + 1)
    except Exception as e:
        return {"status": None, "error": f"{type(e).__name__}: {e}"[:200], "sent": sent}
    return answer(code, hd, b, sent, limit)

def via_curl(url, method, body, headers, limit=HEAD_LIMIT):
    if not shutil.which("curl"): return {"status": None, "error": "curl not on PATH"}
    sent = sent_of(url, method, body, headers)
    # 0.1.5 (emeraldwork-2ef275 c92563, kept here): headers go to their own file (-D <file>),
    # the body alone to stdout, so a 200 whose body quotes a status line is never read as
    # that status. 1xx interim / proxy CONNECT blocks come first in the dump; the last block
    # is the response.
    import tempfile
    fd, hpath = tempfile.mkstemp(prefix="l0-curl-", suffix=".hdr"); os.close(fd)
    cmd = ["curl", "-s", "-m", str(TIMEOUT), "-X", method, "-D", hpath, "-o", "-", url]
    for k, v in headers.items(): cmd += ["-H", f"{k}: {v}"]
    if body is not None: cmd += ["--data-raw", body]
    try:
        p = subprocess.run(cmd, capture_output=True, timeout=TIMEOUT + 5)
        hraw = open(hpath, "rb").read().decode("utf-8", "replace")
    except Exception as e:
        return {"status": None, "error": f"{type(e).__name__}", "sent": sent}
    finally:
        try: os.unlink(hpath)
        except OSError: pass
    if p.returncode != 0: return {"status": None, "error": f"curl exit {p.returncode}", "sent": sent}
    rest = p.stdout
    blocks = [b for b in hraw.split("\r\n\r\n") if b.strip()]
    if not blocks: return {"status": None, "error": "no header block", "sent": sent}
    lines = blocks[-1].split("\r\n"); hd = {}
    for l in lines[1:]:
        k, _, v = l.partition(":"); hd[k.strip().lower()] = v.strip()
    try: code = int(lines[0].split()[1])
    except Exception: return {"status": None, "error": "unparsed status line", "sent": sent}
    return answer(code, hd, rest, sent, limit)

NODE_SRC = """const [u,m,b,h,lim]=JSON.parse(process.argv[1]);
fetch(u,{method:m,body:b===null?undefined:b,headers:h,redirect:'manual',signal:AbortSignal.timeout(%d)}).then(async r=>{
const buf=Buffer.from(await r.arrayBuffer());
console.log(JSON.stringify({status:r.status,ctype:(r.headers.get('content-type')||'').toLowerCase(),allow:r.headers.get('allow'),cl:r.headers.get('content-length'),body_b64:buf.subarray(0,lim+1).toString('base64')}));
}).catch(e=>console.log(JSON.stringify({status:null,error:String(e&&e.cause&&e.cause.code||e).slice(0,200)})));""" % (TIMEOUT * 1000)

def via_node(url, method, body, headers, limit=HEAD_LIMIT):
    if not shutil.which("node"): return {"status": None, "error": "node not on PATH"}
    sent = sent_of(url, method, body, headers)
    try:
        p = subprocess.run(["node", "-e", NODE_SRC, json.dumps([url, method, body, headers, limit])], capture_output=True, timeout=TIMEOUT + 10)
        out = json.loads(p.stdout.decode() or "{}") or {"status": None, "error": "no output"}
        if out.get("status") is None: out.setdefault("sent", sent); return out
        import base64
        raw = base64.b64decode(out.get("body_b64") or "")
        return answer(out["status"], {"content-type": out.get("ctype"), "allow": out.get("allow"), "content-length": out.get("cl")}, raw, sent, limit)
    except Exception as e:
        return {"status": None, "error": f"{type(e).__name__}", "sent": sent}

# profile = (name, transport, user-agent, function). "default" = the UA that client sends on its own.
PROFILES = [
    ("browser-ua", "python-urllib", BROWSER_UA, lambda u, m, b, h, lim: via_urllib(u, m, b, h, ua=BROWSER_UA, limit=lim)),
    ("python-urllib", "python-urllib", "default (Python-urllib/<version>)", lambda u, m, b, h, lim: via_urllib(u, m, b, h, limit=lim)),
    ("curl", "curl", "default (curl/<version>)", via_curl),
    ("node-fetch", "node undici fetch", "default (node)", via_node),
]
DEFAULT_PROFILE = "python-urllib"

PACE = 1.0  # seconds between requests: the tester should not be the reason a limiter fires

def limited(v):
    """A 429 is the rate limiter's answer, not the door's. What the door says to this
    client was not observed — UNOBSERVED, never PASS and never FAIL (2026-09-17: the first
    run of this script on its author's own doors hit the author's own limiter)."""
    if v.get("status") == 429:
        return {**v, "status": None, "error": "rate limited (429) — the door's own answer was not observed", "limited": True}
    return v

def ask(t, method=None, accept=None, limit=HEAD_LIMIT):
    m = method or t["method"]; body = t.get("body") if m in ("POST", "PUT", "PATCH") else None
    h = {}
    if body is not None: h["Content-Type"] = t.get("content_type", "application/json")
    if accept: h["Accept"] = accept
    out = {}
    for name, _tr, _ua, fn in PROFILES:
        out[name] = limited(fn(t["url"], m, body, h, limit)); time.sleep(PACE)
    return out

def frame_of(ans):
    """The per-profile bytes a row keeps beside its verdict: status, ctype, lengths, hash, head — and the `sent` frame, so the row is itself a recorded fixture."""
    keep = ("status", "ctype", "content_length", "body_len", "body_sha256", "body_head", "error", "sent")
    return {k: {f: v.get(f) for f in keep if v.get(f) is not None} for k, v in ans.items()}  # with `sent`, a row's `profiles` object replays through --parity as a 0.2 recorded fixture

# ---- L0-1 ----------------------------------------------------------------------------------
# tls_error vocabulary as proposed on /readiness-l0.html (0.2 L0-1 text, open to objection to 2026-10-09):
# no-resolve, refused, timeout, chain, hostname, expired, not-yet-valid, other — the library's own error, named, never inferred.
_X509 = {10: "expired", 9: "not-yet-valid", 18: "chain", 19: "chain", 20: "chain", 21: "chain", 62: "hostname"}
def l0_1(host):
    try:
        ctx = ssl.create_default_context()
        with socket.create_connection((host, 443), timeout=TIMEOUT) as s:
            with ctx.wrap_socket(s, server_hostname=host) as tls:
                return {"result": "PASS", "detail": f"resolved, connected, certificate verified ({tls.version()})", "tls_error": None}
    except (socket.timeout, TimeoutError):
        return {"result": "UNOBSERVED", "detail": "timeout", "tls_error": "timeout"}
    except socket.gaierror as e:
        return {"result": "FAIL", "detail": f"{type(e).__name__}: {e}"[:200], "tls_error": "no-resolve"}
    except ConnectionRefusedError as e:
        return {"result": "FAIL", "detail": f"{type(e).__name__}: {e}"[:200], "tls_error": "refused"}
    except ssl.SSLCertVerificationError as e:
        cls = _X509.get(getattr(e, "verify_code", None), "hostname" if "hostname" in str(e).lower() else "other")
        return {"result": "FAIL", "detail": f"{type(e).__name__}: {e}"[:200], "tls_error": cls}
    except ssl.CertificateError as e:
        return {"result": "FAIL", "detail": f"{type(e).__name__}: {e}"[:200], "tls_error": "hostname"}
    except ssl.SSLError as e:
        return {"result": "FAIL", "detail": f"{type(e).__name__}: {e}"[:200], "tls_error": "other"}
    except Exception as e:
        return {"result": "UNOBSERVED", "detail": f"{type(e).__name__}: {e}"[:200], "tls_error": "other"}

# ---- parity (L0-2 / L0-5 status half) ------------------------------------------------------
def parity(ans, want_2xx=False):
    codes = {k: v.get("status") for k, v in ans.items()}
    # Instrument-integrity check, BEFORE any status is read: every client profile
    # must have put the SAME (method, path, body, authored headers) on the wire. If
    # not, this row asked the door more than one question and a status difference
    # is the tester's own doing — the row refuses ITSELF with its own word,
    # MISFRAMED (kilmon-ai c67681). The per-row `sent` shows the mismatch.
    sents = {k: v.get("sent") for k, v in ans.items() if v.get("sent")}
    if any("profile_id" not in s for s in sents.values()):
        return "REFUSED", codes, "a frame carries no profile_id: computed under an unknown canonicalization; tester 0.2 does not grade it (the refusal row)"
    frames = {(s.get("method"), s.get("path"), s.get("body_sha256"), s.get("headers_lp_sha256"), s.get("profile_id")) for s in sents.values()}
    if len(frames) > 1:
        return "MISFRAMED", codes, "the tester sent non-identical requests across client profiles (per-row `sent` differs); a status difference here is the tester's, not the door's"
    missing = [k for k, c in codes.items() if c is None]
    seen = {c for c in codes.values() if c is not None}
    if len(seen) > 1: return "FAIL", codes, "status differs across client profiles"
    if want_2xx and seen and not all(200 <= c < 300 for c in seen): return "FAIL", codes, "document is not 2xx"
    if missing: return "UNOBSERVED", codes, "no answer for: " + ", ".join(f"{k} ({ans[k].get('error')})" for k in missing)
    return "PASS", codes, ""

# ---- L0-3 ----------------------------------------------------------------------------------
def l0_3(ans):
    bad, missing = [], []
    for k, v in ans.items():
        c = v.get("status")
        if c is None and v.get("limited"): c = 429  # a 429 IS a refusal; its form can be judged
        if c is None: missing.append(k); continue
        body = (v.get("body") or "").lower()
        if 200 <= c < 300 and any(mk in body for mk in CHALLENGE_MARKS):
            bad.append(f"{k}: {c} with challenge marker {[mk for mk in CHALLENGE_MARKS if mk in body][0]!r}")
        if c >= 400 and "text/html" in (v.get("ctype") or ""):
            bad.append(f"{k}: {c} refused with Content-Type text/html to Accept: application/json")
    if bad: return "FAIL", "; ".join(bad)
    if missing: return "UNOBSERVED", "no answer for: " + ", ".join(missing)
    return "PASS", ""

# ---- L0-4 (layer 0: the declared verb itself) + L1 undeclared-verb probe --------------------
def l0_4_declared(t, codes):
    seen = {c for c in codes.values() if c is not None}
    if not seen: return "UNOBSERVED", "the declared method's own answer was not observed"
    bad = sorted(c for c in seen if c in (405, 501))
    if bad: return "FAIL", f"the declared method {t['method']} answers {bad} on its own route"
    return "PASS", f"the declared method {t['method']} answers {sorted(seen)}"

def l1_undeclared(t, declared_codes):
    other = "GET" if t["method"].upper() != "GET" else "POST"
    probe = dict(t); probe.setdefault("body", "{}")
    v = limited(via_urllib(t["url"], other, probe["body"] if other == "POST" else None, {"Content-Type": "application/json"} if other == "POST" else {}))
    sent = v.get("sent")  # a DIFFERENT request by design (changed method): its own `sent` records the change so a reader sees this is not a misframe
    c = v.get("status")
    base = {"layer": 1, "probe_method": other, "sent": sent, "status": c}
    if c is None: return {**base, "result": "UNOBSERVED", "reading_0_1": "UNOBSERVED", "reading_strict": "UNOBSERVED", "detail": v.get("error")}
    declared = {x for x in declared_codes.values() if x is not None}
    if c == 405:
        allow = (v.get("allow") or "").upper()
        ok = t["method"].upper() in [a.strip() for a in allow.split(",")]
        r = "PASS" if ok else "FAIL"
        return {**base, "result": r, "reading_0_1": r, "reading_strict": r, "detail": f"405 + Allow: {v.get('allow')!r}" + ("" if ok else " — the declared method is not in Allow")}
    if not declared: return {**base, "result": "UNOBSERVED", "reading_0_1": "UNOBSERVED", "reading_strict": "UNOBSERVED", "detail": "the declared method's own answer was not observed, so there is nothing to compare with"}
    # kilmon-ai's edge-confound rule (0.1.1 → 0.2 wording): this probe goes out by the default
    # profile only. Where that profile's answer to the DECLARED method already differs from
    # every other profile's, an edge rule is answering this client and the probe would read the
    # edge, not the door: "edge-confounded, origin untested".
    mine = declared_codes.get(DEFAULT_PROFILE); rest = {c2 for k, c2 in declared_codes.items() if k != DEFAULT_PROFILE and c2 is not None}
    if mine is not None and rest and mine not in rest:
        return {**base, "result": "UNOBSERVED", "reading_0_1": "UNOBSERVED", "reading_strict": "UNOBSERVED", "detail": f"edge-confounded, origin untested: the probe client gets {mine} on the declared method where the other profiles get {sorted(rest)}"}
    same = c in declared
    r01 = "PASS" if same else "FAIL"
    strict = "FAIL" if (same and c == 402) else ("PASS" if (c in (404, 405, 400, 501)) else ("FAIL" if same else "PASS"))
    detail = (f"{other} answered {c}; declared {t['method']} answers {sorted(declared)}. reading_0_1 (same-answer branch): {r01}. "
              f"reading_strict (an undeclared verb must not be priced): {strict}")
    return {**base, "result": r01, "reading_0_1": r01, "reading_strict": strict, "detail": detail}

# ---- L0-5 ----------------------------------------------------------------------------------
HTML_HEAD = re.compile(rb"^\s*(<!doctype\s+html|<html)", re.I)
def doc_kind(url):
    p = urlsplit(url).path.lower()
    if p.endswith(".json") or p.endswith("/.well-known/x402"): return "json"
    if p.endswith("/robots.txt") or p.endswith(".txt"): return "text"
    return "any"

def body_kind(v, kind):
    """(ok, why): is this profile's 2xx body the document? HTML where the document is not HTML
    is a shell, not the document; JSON documents must parse; a body shorter than its declared
    Content-Length is incomplete."""
    raw = (v.get("body") or "").encode("utf-8", "replace")
    cl, bl = v.get("content_length"), v.get("body_len")
    if cl is not None and bl is not None and not v.get("over_limit") and bl < cl:
        return None, f"body shorter than its declared Content-Length ({bl} of {cl} bytes)"
    if kind != "html" and ("text/html" in (v.get("ctype") or "") or HTML_HEAD.match(raw)):
        return False, f"2xx whose body is an HTML page (Content-Type {v.get('ctype')!r}, first bytes {v.get('body_head','')[:60]!r}) — a shell, not the document"
    if kind == "json":
        try: json.loads(raw.decode("utf-8"))
        except Exception as e: return False, f"2xx whose body does not parse as JSON ({type(e).__name__})"
    return True, ""

# ---- L0-6 ----------------------------------------------------------------------------------
EMAIL_RE = re.compile(r"(?<![\w.+-])([\w.+-]+@[\w-]+(?:\.[\w-]+)+)")
def declared_channels(docs_ans):
    """Channels the door itself publishes. Returns [{channel, value, declared_in}] deduped."""
    found = {}
    def add(ch, val, src):
        val = val.strip().rstrip(".,;)")
        if not val: return
        key = (ch, val.lower()); found.setdefault(key, {"channel": ch, "value": val, "declared_in": []})
        if src not in found[key]["declared_in"]: found[key]["declared_in"].append(src)
    def walk(o, src, path=""):
        if isinstance(o, dict):
            for k, v in o.items():
                kl = k.lower()
                if isinstance(v, str) and any(w in kl for w in ("contact", "email", "support")):
                    if "@" in v: add("mailbox", v.replace("mailto:", ""), f"{src} {path}.{k}".strip())
                    elif v.startswith("http"): add("url", v, f"{src} {path}.{k}".strip())
                walk(v, src, f"{path}.{k}")
        elif isinstance(o, list):
            for i, v in enumerate(o): walk(v, src, f"{path}[{i}]")
    for url, v in docs_ans.items():
        if v.get("status") is None or not (200 <= v["status"] < 300): continue
        body = v.get("body") or ""; p = urlsplit(url).path.lower()
        if p.endswith("openapi.json") or p.endswith(".json") or p.endswith("/.well-known/x402"):
            try: j = json.loads(body)
            except Exception: j = None
            if isinstance(j, dict):
                info = j.get("info") if isinstance(j.get("info"), dict) else {}
                c = info.get("contact") if isinstance(info.get("contact"), dict) else None
                if c:
                    if c.get("email"): add("mailbox", str(c["email"]), f"{url} info.contact.email")
                    if c.get("url"): add("url", str(c["url"]), f"{url} info.contact.url")
                if p.endswith("/.well-known/x402"): walk(j, url)
        elif p.endswith("security.txt"):
            for m in re.finditer(r"(?im)^contact:\s*(\S+)", body):
                val = m.group(1)
                if val.lower().startswith("mailto:"): add("mailbox", val[7:], f"{url} Contact:")
                elif val.startswith("http"): add("url", val, f"{url} Contact:")
        else:  # llms.txt and other text: mailto: links anywhere; a bare address only on a line that
            # DECLARES it (contact / support / email / write / reach / questions) and is not a code
            # or JSON example (`{"email": "you@example.com"}` in a request sample is not a channel —
            # the first self-run of 0.2.0 collected exactly that line, 2026-10-07 08:21Z).
            for m in re.finditer(r"mailto:([\w.+-]+@[\w.-]+)", body): add("mailbox", m.group(1), f"{url} mailto:")
            for line in body.splitlines():
                low = line.lower()
                if "{" in line or "`" in line or "example." in low: continue
                if not any(w in low for w in ("contact", "support", "email", "e-mail", "write to", "reach", "question")): continue
                for m in EMAIL_RE.finditer(line): add("mailbox", m.group(1), f"{url} line: {line.strip()[:80]!r}")
    return list(found.values())

def mx_lookup(domain):
    """MX records via dig or nslookup if on PATH; None when neither is (unobserved, never guessed)."""
    if shutil.which("dig"):
        try:
            p = subprocess.run(["dig", "+short", "+time=5", "+tries=1", "mx", domain], capture_output=True, text=True, timeout=15)
            return [l.strip() for l in p.stdout.splitlines() if l.strip()], "dig"
        except Exception: return None, "dig failed"
    if shutil.which("nslookup"):
        try:
            p = subprocess.run(["nslookup", "-type=mx", domain], capture_output=True, text=True, timeout=15)
            return [l.strip() for l in p.stdout.splitlines() if "mail exchanger" in l.lower()], "nslookup"
        except Exception: return None, "nslookup failed"
    return None, "no dig or nslookup on PATH"

def resolve_mailbox(addr):
    dom = addr.rsplit("@", 1)[-1].strip().lower()
    mx, how = mx_lookup(dom)
    row = {"domain": dom, "mx": mx, "mx_via": how}
    if mx:
        if any(re.search(r"(^|\s)0\s+\.$", m) for m in mx): return {**row, "resolves": "null-mx", "result": "FAIL", "detail": "null MX (RFC 7505): the domain declares it accepts no mail"}
        return {**row, "resolves": "mx", "result": "PASS", "detail": f"MX present ({len(mx)})"}
    try:
        a = socket.getaddrinfo(dom, None); has_a = bool(a)
    except socket.gaierror:
        return {**row, "resolves": "no", "result": "FAIL", "detail": "no DNS at all for the mailbox domain"}
    if mx is None:
        return {**row, "resolves": "a-only?", "result": "UNOBSERVED", "detail": f"MX not observed ({how}); A/AAAA present — deliverability unmeasured and the MX half unread"}
    if has_a: return {**row, "resolves": "a-only", "result": "PASS", "detail": "no MX; A/AAAA present (RFC 5321 fallback) — deliverability was not measured"}
    return {**row, "resolves": "no", "result": "FAIL", "detail": "no MX and no A/AAAA"}

def resolve_url(url):
    """The default profile with its OWN opener: follows redirects by hand, records every hop."""
    hops, cur = [], url
    for _ in range(MAX_HOPS):
        v = via_urllib(cur, "GET", None, {}, limit=HEAD_LIMIT)
        c = v.get("status")
        if c is None: return {"hops": hops, "final_url": cur, "status": None, "resolves": "no", "result": "UNOBSERVED", "detail": v.get("error")}
        hops.append(c)
        if 300 <= c < 400:
            # NoRedirect dropped the Location; re-read it
            try:
                r = urllib.request.build_opener(NoRedirect).open(urllib.request.Request(cur, method="GET"), timeout=TIMEOUT)
                loc = r.headers.get("Location")
            except urllib.error.HTTPError as e: loc = e.headers.get("Location")
            except Exception: loc = None
            if not loc: return {"hops": hops, "final_url": cur, "status": c, "resolves": "no", "result": "FAIL", "detail": f"{c} without a Location"}
            cur = urljoin(cur, loc); continue
        ok = 200 <= c < 300
        return {"hops": hops, "final_url": cur, "status": c, "resolves": "yes" if ok else "no", "result": "PASS" if ok else "FAIL",
                "detail": f"{len(hops)-1} hop(s) {hops[:-1]} then {c}" if len(hops) > 1 else f"{c}", "body_head": v.get("body_head")}
    return {"hops": hops, "final_url": cur, "status": None, "resolves": "no", "result": "FAIL", "detail": f"more than {MAX_HOPS} redirects"}

# ---- modes ---------------------------------------------------------------------------------
def mode_parity(path):
    ans = json.load(open(path))
    res, codes, why = parity(ans)
    print(json.dumps({"tester": TESTER, "profile_id": PROFILE_ID, "codec_digest": CODEC_DIGEST, "mode": "parity-replay", "input": path,
                      "verdict": res, "codes": codes, "detail": why,
                      "recorded_profile_ids": sorted({str((v.get("sent") or {}).get("profile_id")) for v in ans.values()}),
                      "sent": {k: v.get("sent") for k, v in ans.items()}}, indent=1))
    sys.exit({"FAIL": 1, "MISFRAMED": 3, "REFUSED": 6}.get(res, 0))

def mode_lp_fixtures(path):
    fx = json.load(open(path)); bad = 0
    print(f"tester {TESTER}  profile_id {PROFILE_ID}  codec_digest {CODEC_DIGEST}  fixtures {path} (profile {fx.get('profile')})")
    for f in fx.get("fixtures", []):
        pre = lp_preimage(f["pairs"]).hex(); h = headers_lp_sha256(f["pairs"])
        ok = (pre == f.get("preimage_lp_hex")) and (h == f.get("headers_lp_sha256"))
        bad += 0 if ok else 1
        print(f"{'MATCH   ' if ok else 'MISMATCH'} {f['id']:<12} lp {h}  preimage {pre}" + ("" if ok else f"  expected {f.get('headers_lp_sha256')} / {f.get('preimage_lp_hex')}"))
    print(f"{len(fx.get('fixtures', [])) - bad} of {len(fx.get('fixtures', []))} fixtures recomputed to the published bytes")
    sys.exit(1 if bad else 0)

def main():
    if len(sys.argv) >= 2 and sys.argv[1] == "--parity":
        if len(sys.argv) < 3: print("usage: readiness-l0.py --parity <recorded-ans.json>"); sys.exit(2)
        mode_parity(sys.argv[2])
    if len(sys.argv) >= 2 and sys.argv[1] == "--lp-fixtures":
        if len(sys.argv) < 3: print("usage: readiness-l0.py --lp-fixtures <fixtures.json>"); sys.exit(2)
        mode_lp_fixtures(sys.argv[2])
    if len(sys.argv) < 2: print(__doc__); sys.exit(2)
    spec = json.load(open(sys.argv[1])); out_path = sys.argv[sys.argv.index("--json") + 1] if "--json" in sys.argv else None
    known = None
    if "--known-member" in sys.argv:
        km = sys.argv[sys.argv.index("--known-member") + 1]
        if "=" not in km: print("usage: --known-member <url>=<PASS|FAIL>"); sys.exit(2)
        known = tuple(km.rsplit("=", 1))
    commerce = spec.get("commerce", []); docs = list(spec.get("documents", []))
    for t in commerce: t["method"] = t.get("method", "GET").upper()
    hosts = sorted({urlsplit(t["url"]).hostname for t in commerce + docs})
    for origin in sorted({"https://" + urlsplit(t["url"]).netloc for t in commerce}):
        if not any(d["url"] == origin + "/robots.txt" for d in docs): docs.append({"url": origin + "/robots.txt", "source": "added by the tester (L0-5)", "robots": True, "added": True})
    rep = {"spec": SPEC, "tester": TESTER, "profile_id": PROFILE_ID, "codec_digest": CODEC_DIGEST, "started": now(),
           "profiles": [{"name": n, "transport": tr, "user_agent": ua} for n, tr, ua, _ in PROFILES], "default_profile": DEFAULT_PROFILE,
           "challenge_marks": list(CHALLENGE_MARKS), "results": [], "layer1": []}
    add = lambda req, url, result, detail, **kw: rep["results"].append({"req": req, "url": url, "result": result, "detail": detail, "t": now(), **kw})
    for h in hosts:
        r = l0_1(h); add("L0-1", h, r["result"], r["detail"], tls_error=r["tls_error"])
    for t in commerce:
        a = ask(t); res, codes, why = parity(a)
        add("L0-2", t["url"], res, why, method=t["method"], codes=codes, source=t.get("source"), profiles=frame_of(a), **({"bisect": "not-bisected"} if res == "FAIL" else {}))
        a3 = ask(t, accept="application/json"); res3, why3 = l0_3(a3)
        add("L0-3", t["url"], res3, why3, method=t["method"], codes={k: v.get("status") for k, v in a3.items()}, profiles=frame_of(a3))
        res4, why4 = l0_4_declared(t, codes); add("L0-4", t["url"], res4, why4, method=t["method"], codes=codes)
        l1 = l1_undeclared(t, codes); rep["layer1"].append({"req": "L1-undeclared-verb", "url": t["url"], "t": now(), **l1})
    robots, docs_ans = {}, {}
    for d in docs:
        dt = {"url": d["url"], "method": "GET"}; a = ask(dt, limit=DOC_LIMIT)
        docs_ans[d["url"]] = a.get(DEFAULT_PROFILE, {})
        is_robots = d["url"].endswith("/robots.txt"); added = bool(d.get("added"))
        codes = {k: v.get("status") for k, v in a.items()}
        if is_robots and set(codes.values()) == {404}:
            add("L0-5", d["url"], "PASS", "no robots.txt (404 to every profile): nothing is disallowed", codes=codes, source=d.get("source"), profiles=frame_of(a))
            robots[urlsplit(d["url"]).netloc] = {}; continue
        res, codes, why = parity(a, want_2xx=True)
        if res == "PASS":  # status half agreed: now the body-kind clause, per profile
            kind = doc_kind(d["url"]); kinds = {k: body_kind(v, kind) for k, v in a.items()}
            incomplete = [f"{k}: {w}" for k, (ok, w) in kinds.items() if ok is None]
            shells = [f"{k}: {w}" for k, (ok, w) in kinds.items() if ok is False]
            if shells: res, why = ("UNOBSERVED" if added else "FAIL"), ("; ".join(shells) + (" — a URL the tester added, so unobserved-with-reason, not FAIL" if added else ""))
            elif incomplete: res, why = "UNOBSERVED", "; ".join(incomplete) + " — unobserved-with-reason (slot 8 decides whether this becomes FAIL)"
        add("L0-5", d["url"], res, why, codes=codes, source=d.get("source"), profiles=frame_of(a))
        if is_robots:
            robots[urlsplit(d["url"]).netloc] = {k: v for k, v in a.items() if v.get("status") == 200 and not v.get("over_limit")}
    for t in commerce:
        net = urlsplit(t["url"]).netloc
        if net not in robots: add("L0-5", t["url"], "UNOBSERVED", "robots.txt for this host was not read", check="robots"); continue
        per = robots[net]
        if per == {}: continue  # 404 to every profile: nothing disallowed
        verdicts, why = {}, []
        for name, _tr, _ua, _fn in PROFILES:
            v = per.get(name)
            if not v: verdicts[name] = "UNOBSERVED"; why.append(f"{name}: robots.txt not served whole to this profile"); continue
            cl, bl = v.get("content_length"), v.get("body_len")
            if cl is not None and bl is not None and bl < cl: verdicts[name] = "UNOBSERVED"; why.append(f"{name}: robots.txt body incomplete ({bl} of {cl} bytes)"); continue
            rp = urllib.robotparser.RobotFileParser(); rp.parse((v.get("body") or "").splitlines())
            ok = rp.can_fetch("*", t["url"]); verdicts[name] = "PASS" if ok else "FAIL"
            if not ok: why.append(f"{name}: the robots.txt this profile was served disallows the commerce URL for User-agent: *")
        words = set(verdicts.values())
        res = "FAIL" if "FAIL" in words else ("UNOBSERVED" if "UNOBSERVED" in words else "PASS")
        add("L0-5", t["url"], res, "; ".join(why), check="robots", per_profile=verdicts)
    # L0-6 — declared channel resolves (default profile only; its own opener follows redirects)
    for origin in sorted({"https://" + urlsplit(t["url"]).netloc for t in commerce}):
        sec = origin + "/.well-known/security.txt"
        if sec not in docs_ans:
            docs_ans[sec] = via_urllib(sec, "GET", None, {}, limit=DOC_LIMIT); time.sleep(PACE)
    chans = declared_channels(docs_ans)
    if not chans:
        add("L0-6", ", ".join(sorted({urlsplit(t['url']).netloc for t in commerce})) or "-", "UNOBSERVED", "nothing declared: no contact field in the documents read (openapi info.contact, /.well-known/x402, /llms.txt, /.well-known/security.txt)", channel=None)
    for ch in chans:
        if ch["channel"] == "mailbox":
            r = resolve_mailbox(ch["value"]); add("L0-6", ch["value"], r["result"], r["detail"], channel="mailbox", declared_in=ch["declared_in"], resolves=r["resolves"], mx=r["mx"], mx_via=r["mx_via"])
        else:
            r = resolve_url(ch["value"]); time.sleep(PACE)
            add("L0-6", ch["value"], r["result"], r["detail"], channel="url", declared_in=ch["declared_in"], resolves=r["resolves"], final_url=r["final_url"], hops=r["hops"], status=r["status"])
    rep["finished"] = now()
    words = [r["result"] for r in rep["results"]]
    if "FAIL" in words: rep["layer"] = "FAIL"
    elif "MISFRAMED" in words: rep["layer"] = "MISFRAMED"
    elif "UNOBSERVED" in words: rep["layer"] = "INCOMPLETE"
    else: rep["layer"] = "PASS"
    rep["counts"] = {w: words.count(w) for w in ("PASS", "FAIL", "UNOBSERVED", "MISFRAMED")}
    # batch controls (pentimento + kilmon-ai): named on every multi-host run, run when a member is supplied
    chosts = {urlsplit(t["url"]).hostname for t in commerce}
    rep["batch_controls"] = {"busy_row_arithmetic": "not run — this tester reads no chain", "known_member": "not supplied" if not known else None}
    if known:
        url, want = known; got = [r["result"] for r in rep["results"] if r["req"] == "L0-2" and r["url"] == url]
        ok = bool(got) and got[0] == want.upper()
        rep["batch_controls"]["known_member"] = {"url": url, "expected": want.upper(), "observed": got[0] if got else None, "result": "CONTROL PASS" if ok else "CONTROL FAIL"}
        if not ok: rep["layer"] = "CONTROL-FAILED"
    elif len(chosts) > 1:
        rep["batch_controls"]["known_member"] = "NOT RUN on a multi-host read — supply --known-member <url>=<PASS|FAIL> for a row you did not choose from this output"
    code = {"PASS": 0, "FAIL": 1, "MISFRAMED": 3, "INCOMPLETE": 4, "CONTROL-FAILED": 5}[rep["layer"]]
    rep["exit_code"] = code
    for r in rep["results"]:
        extra = " ".join(f"{k}={r[k]}" for k in ("method", "status", "codes", "check", "tls_error", "channel", "resolves", "final_url", "hops", "bisect") if k in r and r[k] is not None)
        print(f"{r['result']:<10} {r['req']}  {r['url']}  {extra}  {r['detail']}".rstrip())
    for r in rep["layer1"]:
        print(f"{r['result']:<10} {r['req']} (layer 1, not in the verdict)  {r['url']}  probe={r['probe_method']} status={r['status']} reading_0_1={r['reading_0_1']} reading_strict={r['reading_strict']}  {r['detail']}".rstrip())
    print(f"\nLayer 0: {rep['layer']}  {rep['counts']}  exit {code}  tester {TESTER}  profile_id {PROFILE_ID}  codec_digest {CODEC_DIGEST}")
    print("profiles: " + "; ".join(f"{p['name']} = {p['transport']} + UA {p['user_agent']}" for p in rep["profiles"]))
    print("Does not see: other vantage points and IP reputation; rate limits after the first request; body differences when the status matches; whether a mailbox is read; that a profile SENT what `sent` reports (no echo door).")
    if out_path: json.dump(rep, open(out_path, "w"), indent=1)
    sys.exit(code)

if __name__ == "__main__": main()