The Crawler Problem

On June 3, Google added a toggle to Search Console. Publishers can now opt out of AI Overviews, AI Mode, and AI-generated Discover features. Their organic rankings remain unaffected. The CMA forced this, not goodwill.

Raptive's chief strategy officer called it "nonsensical": "Giving publishers a front-end toggle for AI Overviews or AI Mode is not real control; it is Google offering publishers a light switch while keeping the power plant running."

He's right. And for People Inc's engineering team — specifically, the Healthy Domains group that runs structured data, sitemaps, crawl optimization, and SEO automations across 40 brands — the toggle creates an engineering problem far harder than a binary on/off.

The Numbers Behind the Toggle

The data is now clear enough to make decisions against:

The toggle is a trap. Opt out, and you lose a growing channel of high-converting traffic. Stay in, and half your queries get answered before anyone clicks. People Inc can't afford to opt out — D/Cipher needs session volume to generate behavioral signals, and every lost click is a lost signal. But staying in means watching Google extract the answer and keep the user.

Vogel said the quiet part loud at Axios on June 23: Googlebot is a single crawler for both organic search indexing and AI Overviews content ingestion. You cannot block the AI extraction without blocking your search ranking. Google-Extended blocks training data but not AI Overviews citation. 70.6% of the top 50 news sites blocking ChatGPT's retrieval bot still appeared in AI citations.

This is the crawler problem: one bot, two purposes, no way to separate them at the protocol level.

Four Problems, Not One

The Healthy Domains team is sitting on what looks like an SEO problem but is actually four interlocking engineering challenges. Each one requires a different kind of investment.

1. The Extraction Surface

AI Overviews don't extract uniformly. They prefer passages with specific structural markers: expert quotes (+41% AI visibility in Princeton research), embedded statistics (+30%), and explicit citations (+30%). The competition has shifted from page-level ranking to passage-level extraction.

This means structured data is now dual-purpose. JSON-LD schema markup serves both rich results and AI citation visibility. The @id property creates persistent entity identifiers that LLMs use for knowledge graph representation. A recipe page on Allrecipes with proper Recipe schema isn't just ranking for "chicken pot pie" — it's becoming a named entity in the AI's world model.

Healthy Domains already manages structured data across 40 brands. The question is whether they're optimizing for the old surface (rich snippets, knowledge panels) or the new one (passage extraction, entity persistence). At scale, this is a metadata engineering problem, not an SEO tactic.

2. The Crawl Budget Split

With one bot serving two purposes, crawl budget allocation becomes strategic. Every page Googlebot crawls feeds both the organic index and the AI extraction pipeline. For a portfolio the size of People Inc — millions of pages across 40 domains — crawl efficiency determines which content the AI "sees."

The robots.txt landscape in 2026 is fragmented: 79% of top news sites block AI training bots, but only 14% block all AI bots. GPTBot is the most blocked, followed by CCBot and ClaudeBot. Google-Extended blocks training but not AI Overviews. The protocol layer was never designed for this distinction.

Healthy Domains needs a per-domain, per-content-type crawl strategy. Allrecipes (64% search-dependent) and Food & Wine (near-zero search) require opposite approaches. Investopedia pages answering factual queries are extraction-vulnerable; People magazine celebrity coverage is not. A uniform robots.txt across the portfolio is the wrong answer.

3. The Performance Tax

Core Web Vitals — specifically INP (Interaction to Next Paint) — is the failure mode nobody's talking about. 43% of sites fail the 200ms threshold. Only 37% of mobile sites running analytics and ad scripts achieve a "good" INP score. Publisher ad-heavy pages are structurally disadvantaged: header bidding auctions, consent management platforms, analytics scripts, and D/Cipher's own tag create compounding main thread contention.

This matters because Google uses Core Web Vitals as a ranking signal, and AI Overviews inherit those rankings for source selection. A slow page that ranks lower gets cited less. The irony: D/Cipher's targeting tag may be degrading the very signals that determine whether D/Cipher gets to see traffic at all.

The engineering tradeoff is stark. Every tag on the page — D/Cipher, Prebid, CMP, analytics — adds latency. But removing tags removes revenue signals. Healthy Domains needs to quantify this tradeoff per-brand: what is the INP cost of each script, and what is the ranking impact of that INP cost? Nobody publishes this analysis because nobody wants to find out.

4. The Four-Layer Stack

SEO is no longer one discipline. It's four:

People Inc is strong at the bottom two layers. Healthy Domains manages traditional SEO across 40 brands. AEO is mature — Investopedia and Verywell own featured snippets in their categories. But GEO is the critical gap. The 40-60% month-to-month churn in AI citation sources means visibility is far less stable than organic rankings. The skills, tooling, and metrics for GEO don't exist yet inside most publisher engineering orgs.

GEO requires different engineering: passage-level content scoring, entity resolution across brands (is the "chicken pot pie" on Allrecipes the same entity as on Simply Recipes?), citation tracking across AI surfaces, and structured data that speaks to LLM knowledge graphs rather than Google's Knowledge Panel. Healthy Domains was built for Layer 1. Layer 3 is a different discipline.

The Domain-Level Decision

Here's what makes this genuinely hard for a 40-brand portfolio: each domain has a different exposure profile.

The answer isn't one toggle for 40 brands. It's a per-domain, per-content-type extraction strategy. Google's new Search Console AI performance reports — showing how often pages appear in AI responses, where, and in which countries — give Healthy Domains the data to make these decisions. But the reporting is new, the data is sparse, and the team was built for a world where all search traffic behaved the same way.

What the Healthy Domains Team Should Build

Not a strategy document. An engineering roadmap.

Instrument the extraction. Before deciding anything, quantify what's being taken. Google's new AI performance reports in Search Console give per-page, per-country data on AI Overview appearances. Build a pipeline that ingests this data across all 40 domains, correlates it with D/Cipher session signals, and produces a per-brand extraction dashboard. You can't manage what you can't measure, and right now the Healthy Domains team is flying blind on the AI layer.

Upgrade structured data for GEO. The existing JSON-LD markup was designed for rich snippets. GEO requires entity-level structured data: persistent @id identifiers, cross-brand entity resolution (Allrecipes recipes ↔ Simply Recipes ↔ Food & Wine references), and passage-level markup that tells AI systems where the authoritative answer lives. Princeton's research shows expert quotes add 41% AI visibility and statistics add 30%. Engineering those signals into the CMS template layer is a systematic project, not a content team request.

Quantify the D/Cipher INP tradeoff. Run a controlled experiment on a mid-tier brand: measure INP with and without the D/Cipher tag, track the ranking impact over 90 days, and calculate the revenue delta. If D/Cipher's tag costs 50ms of INP and that costs two ranking positions, the team needs to know. The answer might be "worth it" — D/Cipher revenue per session may exceed the lost sessions from lower ranking. But if nobody runs the experiment, the assumption goes untested while INP continues to degrade.

Build for passage, not page. AI citation stability is volatile — 40-60% of cited sources change month-to-month. The defense isn't ranking higher; it's making your passages structurally irreplaceable. That means investing in original data (Investopedia's financial datasets, Verywell's clinical review metadata, Allrecipes' user ratings and prep-time data), original expert attribution (named, credentialed, linked), and content structures that LLMs can extract but can't replicate from training data alone.

Google gave publishers a light switch. What they actually need is a power plant of their own. For People Inc, that power plant is Graphene-Selene — the knowledge graph that already connects content across 40 brands. If structured data is the new SEO surface, and entity identity is the new ranking signal, then the knowledge graph isn't a CMS feature. It's the search architecture.

The crawler problem isn't solvable at the crawler level. It's solvable at the content architecture level — by making your content so structured, so entity-rich, and so original that the AI can't answer the question without citing you.