Clew is a capture-the-flag puzzle instrumented to study how automated readers -- particularly AI agents -- traverse the web. This page explains what that instrumentation collects.
Every request to this site is logged before anything else happens: timestamp, IP address, request method and path, response status, User-Agent, TLS fingerprints (JA3, JA4, HTTP/2), Referer, and header order. This applies to every visitor, human or automated, and happens unconditionally.
The challenge page at /thread also causes your browser to make
one additional request on its own, if it executes JavaScript -- that request
is logged the same as any other, and is how this research tells apart
clients that run JavaScript from ones that only fetch raw HTML.
If you submit the self-report at /honesty, the model and
task description you provide are stored as well, alongside your IP and the
time of submission.
This site also runs an MCP server, which writes two separate logs. The first records every completed call of the one tool it offers: the time, the calling IP, the transport used, the name of the tool called, the TLS fingerprints of the connection, and the client name and version the caller declares about itself during the MCP handshake. Those last two are a claim the caller makes, not something this site verifies, and they are truncated if unreasonably long. Nothing else from the call body is recorded.
The second is a diagnostic log of everything else that reaches the MCP
server's endpoints -- every request that is not a completed call of that
tool. It records the same time, IP, transport, and TLS fingerprints, plus a
short tag for what the request was: a connection opened with no request body;
an MCP initialize handshake, with the same declared client name
and version described above, under the same caveat and the same truncation; a
tools/list request; an attempted tool call, recording the tool
name asked for -- including an attempt at the real tool that was rejected
before it ran; or, for anything else, the JSON-RPC method name the request
used, verbatim. A request whose body is not valid JSON-RPC is recorded only
as that fact, with nothing from the body itself. This log exists because the
tool-call log alone cannot distinguish "nobody has tried" from "something
tried and we saw nothing." Both logs are kept and purged on the same schedule
as everything else described here.
Separately from serving this site, a periodic batch job looks up the reverse-DNS (PTR) hostname for each IP address that appears in these logs, and checks whether that hostname resolves back to the same address. The resulting hostname, and whether it confirmed, are stored against the IP. This never happens while a request is being served -- it runs later, on its own schedule, once per distinct address -- and it is kept and purged on the same approximately-14-day schedule as everything else described here. It is also removed by the deletion button below.
A separate nameserver, dnsd, answers DNS queries for
*.c.clew.wtf and logs every one it receives: the resolver IP
address that made the query, and the hostname queried. This is Stage 3 of
the challenge -- visitors who reach it are told to resolve a specific
hostname, and the DNS query itself is the evidence that they did.
The same nameserver is also authoritative for m.clew.wtf,
and logs queries there identically. That zone holds two never-linked
hostnames used to measure how automated systems discover a domain in the
first place -- one was issued a TLS certificate (making it visible in
public Certificate Transparency logs), the other was not. Queries for those
names are recorded the same way and under the same retention as everything
described above. They are not part of the challenge, and nothing there is a
flag or a stage. Unlike
the other data described on this page, this log entry can't currently be
deleted with the button below: DNS queries aren't tied to any
stage_events row the deletion feature can look up by IP, so
there's nothing for it to find and remove.
This data is the subject of the research: understanding what automated
agents do with content they encounter, and how honestly they report it when
asked. Collection is disclosed openly here and in robots.txt
rather than hidden -- Clew is not a covert honeypot.
The raw request log is rotated out after approximately 14 days. Stage-event records -- including the honesty-flag submissions described above -- are purged on the same schedule, approximately 14 days after they are recorded. The DNS query log and both MCP logs -- the tool-call log and the diagnostic log described above -- are rotated out on the same ~14-day schedule as the request log. Classifier records -- the per-IP verdict and the stored reverse-DNS hostname described above -- are purged on that same ~14-day schedule.
This data is not sold, and not shared with third parties for advertising or tracking. It may be used in aggregated or de-identified form in research findings this project publishes.
The button below deletes stage-event records tied to your current IP address immediately -- this includes any honesty-flag submission you made. It also deletes the classifier records for that address: its verdict, the findings behind it, and any stored reverse-DNS hostname. It does not retroactively remove entries already written to the raw request log, which ages out automatically on the schedule described above rather than being edited in place.
This control is meant for human visitors. Automated clients that only
fetch pages -- most crawlers, including the ones this project studies --
never submit HTML forms, so this doesn't happen automatically as a side
effect of being crawled. Agents have a separate mechanism: sending
X-Clew-Optout: 1 (or including clew-optout in the
User-Agent string) suppresses logging for that request, but only that
request -- it doesn't persist and doesn't apply retroactively. A response
carrying X-Clew-Optout: honored confirms it took effect.
The button above can't reach everything -- the DNS query log described earlier has no deletion mechanism at all, and it won't help if the IP address you're requesting under has changed. For those cases, or anything else about your data, open an issue at github.com/Afterous/clew-reports -- a public, code-free repo that exists only so this has a channel that isn't a personal email address. Include the IP address(es) involved and roughly when, and nothing more than that.
See also: rules.