How to audit your site for AI crawler visibility: the complete checklist

Your website might rank perfectly in Google Search. But that does not mean AI chatbots can find, understand, or cite your content. AI crawlers operate on different schedules, prioritize different signals, and choke on the same technical mistakes that do not hurt traditional SEO. Running a technical GEO audit means checking whether your site is actually crawlable by the AI bots that now drive referral traffic.

This chapter walks through the exact checklist your site needs to pass, so you can eliminate the technical barriers to AI visibility before investing in content optimization.

What a technical GEO audit actually checks

A technical GEO audit is different from a traditional technical SEO audit. Both check crawlability and speed. But GEO audits also verify:

• Whether you explicitly allow AI crawlers to access your site
• Whether AI bots can read your content without executing JavaScript
• How fast your pages load from the perspective of a bot that does not wait
• Whether your structured data is present in the initial HTML (not loaded after rendering)
• How your site handles URL canonicalization, redirects, and internal linking
• Whether your content is in a format AI extraction tools can easily parse

Traditional SEO focuses on ranking. GEO focuses on discoverability by machine-learning systems that work differently than search engines.

Why this matters right now

AI referral traffic grew 527% year-over-year in early 2025. If your site cannot be crawled by AI bots, you are invisible to ChatGPT, Perplexity, Claude, Gemini, and every other system sending that traffic. A broken robots.txt file or JavaScript-only content does not just hurt your ranking—it prevents your site from being indexed by AI systems entirely.

The cost is not a few lost clicks. It is missing entire categories of traffic that convert at five times the rate of traditional search visitors.

The technical GEO audit checklist

This checklist covers the essentials. If your site passes all of these, you have eliminated the primary barriers to AI visibility.

Step 1: Verify AI crawlers can access your site

Open your robots.txt file (add /robots.txt to your domain root). Look for any lines that might block AI crawlers.

Many sites accidentally block ChatGPT's bot with rules like Disallow: / or User-agent: * Disallow: /. If you see blanket blocks, remove them. If you want to block specific AI crawlers, that is your choice—but blocking all of them kills AI visibility.

Check for rules targeting specific bots. Common culprits:

• User-agent: GPTBot — blocks ChatGPT's crawler
• User-agent: PerplexityBot — blocks Perplexity
• User-agent: ClaudeBot — blocks Claude
• User-agent: GoogleBot-Extended — blocks Google's AI crawler

If you want to allow AI crawlers but exclude specific sections of your site (like admin pages), use specific Allow/Disallow rules. Never use a blanket block.

Step 2: Set up or verify llms.txt

This is newer than robots.txt and most sites do not have it yet. Llms.txt is a file that tells AI systems your content policy and preferred crawling parameters. It lives at /.well-known/llms.txt.

Your llms.txt should include:

• A clear policy on whether AI systems can access your content
• Your website description
• A link to your privacy policy
• Instructions on how you want your content attributed

If you have llms.txt, verify it exists and contains accurate information. If you do not, this is a quick win that signals to AI systems you understand how they work.

Step 3: Confirm your site is not JavaScript-only

AI crawlers do not execute JavaScript. They only see raw HTML. If your website is a single-page application or relies on client-side rendering to display content, AI crawlers will see a blank page.

To test this, use the Fetch and Render tool in Google Search Console or open your page's HTML source (right-click > View Page Source). If you see actual content in the HTML (paragraph tags, headings, text), you are fine. If you see mostly script tags and no content, your site needs server-side rendering.

If this applies to you, work with your developer to add server-side rendering, static generation, or a hybrid approach for critical pages.

Step 4: Check for redirect chains and 404 errors

Broken redirects and 404 errors stop AI crawlers cold. Use a tool like Screaming Frog or Google Search Console to find pages that return 404 status codes.

For each broken page:

• Redirect it to a relevant existing page (if the content moved), or
• Fix the broken link, or
• Remove the page entirely if it is no longer needed

Do the same for redirect chains—if Page A redirects to Page B which redirects to Page C, simplify it to A straight to C. AI crawlers follow redirects, but extra hops waste crawl budget.

Step 5: Verify your XML sitemap is valid

Your XML sitemap tells crawlers which pages on your site exist. Check that it is correct:

• Submit it to Google Search Console and Bing Webmaster Tools
• Verify it lists your most important pages (your top-traffic pages should be there)
• Check that every URL in the sitemap returns a 200 status code (not 404 or 302)
• If the sitemap is large (over 50,000 URLs), split it into multiple sitemaps using a sitemap index

An invalid or missing sitemap does not break crawlability, but it makes it harder for AI systems to discover new content quickly.

Step 6: Check your site speed on mobile

AI crawlers have strict timeout limits. If your page takes longer than 10-15 seconds to load from a bot's perspective, they give up.

Run Google PageSpeed Insights on your top 10 highest-traffic pages. Check specifically:

• First Contentful Paint (FCP) — time until text or images appear
• Cumulative Layout Shift (CLS) — how much does the page move around while loading
• Interaction to Next Paint (INP) — responsiveness to clicks (less important for crawlers, but still matters)

If your pages are slow, the fastest wins are:

• Compress images (most sites waste huge amounts of bandwidth here)
• Minify CSS and JavaScript
• Enable browser caching
• Use a Content Delivery Network (CDN)

For most sites, getting to a 3-second mobile load time cuts out 80% of speed issues that affect AI visibility.

Step 7: Check your structured data

Structured data (also called schema markup) helps AI systems understand what your content is about. Common schema types for GEO:

• Article schema — for blog posts and news
• FAQPage schema — for FAQ content
• HowTo schema — for instructional content
• Organization schema — for company information
• Product schema — for e-commerce
• LocalBusiness schema — for local brands

Make sure the structured data is in your initial HTML (not loaded after JavaScript runs). Use Google's Rich Results Test or Schema.org's validator to check your markup.

Step 8: Verify your site is mobile-responsive

AI crawlers access your site on all device types, but they prioritize mobile. If your site is not mobile-responsive, AI systems will have trouble reading it.

Test your site using Google's Mobile-Friendly Test. If it fails, work with your developer to add responsive design.

Step 9: Check for URL canonicalization issues

If you have duplicate content on multiple URLs (like example.com, www.example.com, example.com/?utm_source=social, etc.), add rel=canonical tags to tell crawlers which version is the main one.

Also check for:

• HTTP vs. HTTPS (pick one and redirect all traffic to it)
• www vs. non-www (pick one)
• Trailing slashes (be consistent)

Canonicalization does not break crawlability, but it helps crawlers spend their budget on the right pages.

Step 10: Audit your internal linking

AI systems use internal links to understand your site structure and topic relationships. Check:

• Are your most important pages linked to from your homepage?
• Are topic-related pages linking to each other?
• Are anchor texts descriptive? ("Learn more about GEO" is better than "Click here")

Poor internal linking does not prevent crawling, but it makes it harder for AI to understand what your site is about.

Where to start if you only have one hour

If you have limited time, fix these in this order (biggest impact first):

1. Check robots.txt for accidental blocks (5 minutes)
2. Verify your site is not JavaScript-only (10 minutes)
3. Fix 404 errors and redirect chains (1-2 hours)
4. Check mobile speed on your top 10 pages (15 minutes)
5. Verify structured data on key page types (30 minutes)
6. Set up llms.txt (15 minutes)

If you pass steps 1-6, you have eliminated the main barriers to AI crawlability. Steps 7-10 are optimizations for faster discovery and better understanding.

How to audit your site without tools

You do not need paid software to run a basic GEO audit. A browser and 30 minutes can catch most issues:

• Open your robots.txt directly (type your domain plus /robots.txt in the browser address bar)
• View page source on your homepage (right-click > View Page Source) and scan for actual content
• Open your XML sitemap (same way) and spot-check that URLs load
• Run Google PageSpeed Insights on your top pages
• Test mobile responsiveness using browser developer tools (F12, then toggle device toolbar)
• Use Google's Rich Results Test to check schema markup

Tools like Screaming Frog (free tier) and Google Search Console are free and more thorough, but they are not required to catch the big issues.

What breaks AI crawlers most often

These mistakes block AI crawlers:

• Robots.txt blocks all crawlers (Disallow: / with no Allow exceptions). Solution: Remove the blanket block or add specific Allow rules.
• Content only exists after JavaScript runs. Solution: Switch to server-side rendering or static generation for critical pages.
• Pages load in 10+ seconds on mobile. Solution: Compress images, minify code, use a CDN.
• No structured data or incorrect schema markup. Solution: Add schema to your key page types using Google's markup helper.
• Endless redirect chains or broken links. Solution: Audit with Google Search Console or Screaming Frog and fix every 404.

Most of these are not hard to fix. They just require knowing where to look.

How to measure the impact after you audit

After you fix technical issues, the impact shows up in:

• AI crawler traffic — check your analytics for traffic from ChatGPT, Perplexity, and other bots
• Brand citations — search your brand name in ChatGPT, Perplexity, and other AI systems. Do they mention you more after fixing crawlability?
• Content indexing — use the Google Search Console URL Inspection tool to confirm your pages are crawlable
• Discovery latency — how fast do AI systems discover new content you publish? Faster crawlability means faster discovery

You will not see results immediately. Technical fixes typically show impact within 30-60 days. Content changes (more structured data, better formatting) take 8-12 weeks.

What WEMASY handles for you

WEMASY's website builder automatically handles most technical GEO requirements:

• Server-side rendering built-in (your content is in the initial HTML)
• Mobile-responsive design by default
• Fast loading times on WEMASY-hosted sites
• Structured data support for common page types
• Automatic sitemap generation
• Security and HTTPS enabled

If you host your site with WEMASY, you are already passing most of the technical GEO checks. To ensure full visibility to AI crawlers, focus on the content side: clear writing, strong structure, and relevant schema for your page type.

You can check your technical health in WEMASY's Analytics & Insights tool, which tracks crawlability and performance metrics.

Frequently asked questions