Ignored Signals
Robots directives, headers, status codes and social contracts that some agents treat as optional.
field lab / crawler behaviour
I Am Not Listening
Systems that ignore the signal.
An experimental lab for observing non-compliant crawlers, adversarial automation, ignored conventions and the strange behaviour that appears when systems refuse to follow the rules.
active system
spigot-rs
spigot-rs is an experimental Rust-based system for observing, shaping and understanding crawler behaviour — especially automated agents that ignore robots.txt, rate limits, headers, signals or polite convention.
It gives this domain a controlled surface for crawler observation, protocol-boundary testing,
selective refusal, adversarial automation and operational evidence collection. Generated pages
are deliberately isolated under /spigot,
while the public homepage explains the experiment without inviting search engines into an
effectively infinite generated space.
The generator is inspired by gw1urf/spigot. This deployment is a Rust implementation using Axum, Askama templates, an EPUB-derived corpus, bounded cache behaviour and Keycloak-backed administration.
concepts
Boundary Testing
Controlled edges for seeing where crawlers respect protocol boundaries and where they press through.
Adversarial Automation
Automation that behaves less like a visitor and more like a pressure test of convention and restraint.
Behavioural Evidence
Operational traces that help distinguish compliant crawling from refusal, probing and abuse.
For broader observations about optimisation failure, incentives and irrational systems, see Daft Choices.
No invasive tracking. Generated pages are marked noindex. Admin access is private.