Technical SEO Audit: Crawl, Index, Speed, AI | SEO AEO Specialist SEO & AEO Specialist

Content and links get the attention, but technical SEO is what decides whether any of it is visible. A brilliant page that a crawler cannot reach, cannot render, or is told not to index contributes nothing. A technical SEO audit inspects that infrastructure layer methodically, in the order crawlers process it: reach the page, render it, index it, understand it.

This is a cluster page under the SEO audit pillar. For the on-page side, see the on-page SEO audit guide.

1. Crawling: can bots reach your pages?

Crawling is step one. If a search engine or AI crawler cannot fetch a URL, nothing downstream matters.

• robots.txt. Confirm it does not block important pages or the CSS/JS needed to render them. Blocking render resources is a classic silent killer.

• Crawl traps. Faceted navigation, infinite calendars, session IDs, and sort parameters can generate millions of low-value URLs that waste crawl budget. Control them with parameters, noindex, or canonicalization.

• Internal reachability. Important pages should sit within three clicks of the homepage. Orphaned pages with no internal links may never be discovered.

• Status codes. Live pages return 200. Removed pages return 404 or 410, not a 200 "soft 404." Servers should not return 5xx errors under normal crawling.

• Redirect hygiene. Collapse redirect chains to a single hop. Every extra hop wastes crawl budget and leaks a little authority.

2. Indexing: are the right pages in the index?

Being crawled is not the same as being indexed. This is where most quiet visibility losses hide.

• noindex accidents. A stray noindex from a staging config or a plugin can deindex money pages. Audit for it explicitly.

• Canonicals. Every page should declare a canonical pointing to the version you want ranked. Watch for conflicts where the canonical, the sitemap, and internal links disagree.

• Duplicate and near-duplicate content. Consolidate variants (trailing slashes, www vs. non-www, HTTP vs. HTTPS, tracking parameters) onto one canonical URL.

• Index coverage. Verify pages you want indexed are indexed, and that thin, duplicate, or utility pages are not bloating the index.

3. Sitemaps and structure

• XML sitemap. Current, lists only indexable canonical URLs, no 404s or redirects, submitted in Search Console.

• Site architecture. A shallow, logical hierarchy helps both crawlers and users. Deep, tangled structures bury pages.

• hreflang. On multilingual or multiregional sites, hreflang tags must be reciprocal and reference valid, indexable URLs.

4. Speed and Core Web Vitals

Page experience is a real ranking input, and speed is the part users feel. The three Core Web Vitals and their "good" thresholds:

A page passes when at least 75% of real-user visits hit the "good" threshold for each metric. INP replaced First Input Delay as the responsiveness metric and is the most commonly failed vital in 2026, so audit it first. Common fixes: reduce main-thread work, defer non-critical JavaScript, and break up long tasks.

Also check HTTPS everywhere (no mixed content), mobile rendering (mobile-first indexing is the default), and image weight (size, compress, and use modern formats).

5. Rendering: does your content survive JavaScript?

If your site renders content client-side, verify that crawlers actually see the rendered output. Search engines can render JavaScript, but it is slower and less reliable than server-rendered HTML. AI crawlers are often worse at it — many do not execute JavaScript at all.

• Check that primary content and links exist in the rendered HTML, not only after client-side hydration.

• Prefer server-side rendering or static generation for content you need indexed and cited.

• Avoid hiding critical text inside images, canvas, or scripts.

6. AI-crawlability: the new technical layer

This is where a 2026 technical audit diverges from a 2020 one. Answer engines now drive meaningful traffic, and they depend on their own crawlers reaching your content.

• AI crawler access. Decide deliberately whether to allow GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot in robots.txt. If you want AI-search visibility, blocking them removes you from those answers entirely.

• Extractable HTML. AI crawlers that do not run JavaScript need your facts in the served HTML. Content locked behind client-side rendering is invisible to them.

• llms.txt. Consider an llms.txt file that indexes your most important pages with short descriptions, giving AI systems a curated map of your site.

• Self-contained answers. Structure key facts so a model can lift them cleanly. Answer-first paragraphs and valid schema make extraction reliable.

Note the trade-off: AI crawlers currently crawl far more than they refer back. That is a strategic call, not a technical default. But if AI visibility is a goal, the technical prerequisites above are non-negotiable.

Common technical SEO problems, ranked

SEO AEO Specialist runs this full technical audit — crawl, index, speed signals, rendering, and AI-crawlability — and hands back a prioritized list. The free tier covers 50 pages, no signup; the €9 report covers your whole site.

FAQ