For publishers

The C Wire crawler

We only buy pages we have read. If our crawler cannot reach your pages we cannot read them, so blocking it removes your inventory from our plans rather than protecting it.

What the crawler sees

CWireBotreads the pagemaps every ad unitlogs to the Placement Graph

User agent

CWireBot

The request says who it is and what it accepts. Verification is the log line, not the promise.

Crawl rate

One request per second per host by default, lower on request. We honour Retry-After and back off on any 429 or 5xx.

One request a second at most. A busy page gets a quieter hour.

What the crawler does

It reads and vectorises article text to establish what the content is about, and records where ad units sit in the layout. It never collects personal data and never follows forms or logged-in areas.

Content Graph

What each page or video is about after we have read and vectorised it.

Placement Graph

What every placement has delivered, profiled and measured unit by unit.

Both feed every decision.

Allowlisting instructions

Nothing changes but this: our requests stop being refused, and the page starts being read.

Security · WAF · Custom rules. Create a rule with the Skip action.

(http.user_agent contains "CWireBot")
→ Action: Skip
→ Skip: All remaining custom rules, Bot Fight Mode, Rate limiting

Questions

The crawler, explained

Why does C Wire need to crawl my site?

Because C Wire only buys pages it has read. If the crawler cannot reach your pages, they cannot be read, so blocking it removes your inventory from every plan rather than protecting it.

What does the crawler read and record?

Article text, to understand what the content is about, and the position of ad units in the layout. It never collects personal data and never follows forms or logged-in areas.

How much load does it create?

One request per second per host by default, lower on request. It honours Retry-After and backs off on any 429 or 5xx response.

How do I allowlist it?

By the published user agent and IP ranges, by range rather than single address. Per-vendor instructions for Cloudflare, Akamai, Fastly, AWS WAF and robots.txt are on this page, without a form gate.

Ask us to crawl your site

Send the domain and we will confirm the crawl window. No form gate on any of the instructions above.

Next step

Talk to us about integration.

One tag, no rebuild, and a review of how our formats sit in your layout. Or simply allowlist the crawler and start there.

Talk to us about integration Allowlist our crawler