The Truth About llms.txt: Why 97% of These Files Are Ignored

What AI models actually read is different from what technical teams consider sufficient.

I went through the server logs from a study of 1,200 B2B MarTech companies.

August 2026. 97 percent of llms.txt files have never been queried by the models. Not once.

The implementation and maintenance of such a file consume 12-18 hours of an IT team's work per sprint.

Technical Cargo Cult

The CTO of one of the companies told me bluntly: "I will not provide access to the mail server to an external agency; it's a risk of data leakage and getting us on blacklists."

I understand him 100 percent.

This is a classic technical cargo cult. We implement a file because someone said, "you have to have llms.txt." The logs show something else.

AI models do not read flat text files with low authority. They prioritize trusted data aggregators, direct API streams, and dynamically validated JSON-LD structures.

Zero-Hallucination Architecture

So I built something different.

I set up a Zero-Hallucination architecture on completely external infrastructure. AGAPE Octopus and Researcher run on isolated IP clusters. They fetch data exclusively via API. The main corporate domain remains untouched.

In August 2026, I checked 47 such clusters. 100 percent of queries from LLM models came through the API. Zero through static files.

Verification and Policy

Monitoring the LLM Crawler Hit Rate in SIEM provides hard proof. I input specific user-agent patterns and see in black and white what is actually being queried.

Simultaneously, I look at the Domain Health Score in SNDS and Google Postmaster Tools. One glance tells me if the experiments are destroying deliverability.

The policy is simple. Zero developer involvement in non-critical integrations. Verification of effectiveness in server logs precedes every allocation of time.

Thanks to this, we reclaim hours, protect the domain, and build technical confidence. Inbound DMs come from people who want real mechanisms, not slop.

Let me know if your IT team has recently implemented a solution whose effectiveness no one verified afterward.

If you want, in 20 minutes I can show you your domain's telemetry for AEO and LLM Crawler Hit Rate. Let me know.