Why Is My Site Not Indexed? Indexation Troubleshooting Guide | SEO AEO Specialist SEO & AEO Specialist

"Why is my site not indexed?" almost always has a specific, findable cause. Indexing is a chain: a bot has to discover the URL, be allowed to crawl it, be allowed to index it, be able to render the content, and judge it worth keeping. Break any link and the page vanishes from search.

Work through these causes in order. The first few are the most common and the fastest to fix.

1. Noindex tags

The single most common cause. A <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex response header tells search engines to keep the page out of the index. These often survive from staging environments or get applied site-wide by a misconfigured plugin or template.

• Check the rendered HTML and the response headers, not just the source.

• Search your templates and CMS settings for noindex.

• Remove it only where indexing is actually intended.

2. Robots.txt blocks

robots.txt controls crawling. A Disallow rule stops bots from fetching the page at all. Critically, a disallowed page cannot show its noindex tag, because the bot never reads the page, so combining the two backfires.

• Confirm important sections are not under a Disallow.

• Remember: robots.txt blocks crawling, meta-robots blocks indexing. Use the right tool.

• If you want a page fully out of the index, allow crawling and use noindex, not disallow.

3. Canonical conflicts

A canonical tag tells search engines which URL is the master version of similar content. When canonicals point to the wrong URL, or every page canonicalizes to the homepage, search engines drop the "duplicate" pages from the index.

• Each page should either self-canonicalize or point to a genuine primary version.

• Watch for CMS defaults that canonicalize paginated or filtered URLs incorrectly.

• Make sure canonical, sitemap, and internal links all agree on the same URL.

4. Crawl budget and discovery

For large sites, search engines allocate a finite crawl budget. If bots burn it on low-value URLs (faceted navigation, session parameters, endless filter combinations, redirect chains), your important pages get crawled late or not at all.

• Consolidate redirect chains into single hops.

• Block or parameter-handle low-value URL variations.

• Keep an accurate, current sitemap and link important pages internally.

• Small sites rarely have a crawl-budget problem; do not over-optimize for it.

5. JavaScript rendering

If your content only appears after JavaScript executes, search engines have to render the page to see it, which they do on a delay and not always completely. Content that depends on client-side rendering can be crawled but come back empty.

• Check what the crawler actually sees after render, not just the raw HTML.

• Prefer server-side rendering or static generation for content you need indexed.

• Make sure primary content and internal links exist in the initial HTML where possible.

6. Quality and duplication

Sometimes the page is perfectly crawlable and the engine simply chooses not to index it. Thin, duplicated, or near-empty pages get filtered as "crawled, not indexed" or "discovered, not indexed."

• Ensure each indexable page has substantial, distinct content.

• Merge or noindex thin variants rather than letting them dilute the site.

• Give the engine a reason to store the page.

AI-crawler indexing

In 2026, showing up in AI answers is a separate concern from training crawlers, and the distinction matters.

Practical takeaways:

- Blocking training crawlers (GPTBot, ClaudeBot) protects content from model training without hurting your presence in AI answers.

- The bots that decide whether you appear inside an AI answer are the search indexers, so do not block those unless you mean to disappear.

- Consider an llms.txt file, an AI-friendly index of your most important pages, to help answer engines prioritize your authoritative content for retrieval.

- Decide access per bot in robots.txt rather than applying one blanket rule.

A troubleshooting checklist

1. Confirm the page is not noindex (HTML and headers).

2. Confirm robots.txt does not disallow it.

3. Confirm the canonical points to the right URL.

4. Confirm the content renders without requiring JavaScript the crawler skips.

5. Confirm the page is linked internally and in the sitemap.

6. Confirm the content is substantial and distinct.

7. Request indexing and check the URL inspection status.

Run them in order. The moment one check fails, you have likely found your cause.

FAQ