How to run an SEO audit with Claude Code

Audit in the right order, or the wrong thing gets fixed first

Most audit checklists are alphabetical. That is how you end up rewriting title tags on a site no crawler has ever visited. The order that matters is causal — each layer is worthless if the one above it is broken:

1. Reachability — does a bot get a 200 from your origin at all?
2. Discovery — does anything on the open web tell a crawler your URLs exist?
3. Crawl — are bots actually arriving, and fetching content rather than just robots.txt?
4. Indexing — has anything been added to a search index?
5. On-page — titles, headings, schema, internal links.
6. Content — does the page answer the query someone typed?

Work top-down and stop at the first layer that fails. A new domain with no inbound links usually fails at step 2, and no amount of step-5 work will move it — that is the single most expensive mistake in this list, because step 5 feels like progress and produces visible diffs.

Layer 1 and 2 — reachability and discovery

These are the two an agent can settle in about a minute, and the two people skip.

# Does a crawler UA actually get served?
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/ | head -1

# Is anything blocking at the robots layer?
curl -s https://example.com/robots.txt

# Does the sitemap exist, parse, and list what you think it lists?
curl -s https://example.com/sitemap.xml | grep -c "<loc>"

Discovery is the harder half and has no single command. The question is blunt: does any page a crawler already visits contain a link to yours? If the honest answer is no, your sitemap is a map to a place nothing knows exists. Worth knowing before you invest weeks: links inside GitHub READMEs are rendered with rel="nofollow", so they carry clicks but are a weak discovery signal — verify with curl -s <repo-url> | grep -o '<a[^>]*yourdomain[^>]*>' rather than assuming.

Layer 3 — read your own access log, not a third-party tool

Crawl evidence is the one signal that is entirely first-party. It cannot be rate-limited, cannot return someone else's false negative, and needs no account. If you control the origin, this is the highest-trust data in the whole audit:

for b in Googlebot bingbot YandexBot; do
  echo "$b total=$(grep -c "$b" access.log) \
 content=$(grep "$b" access.log | grep -vcE 'robots.txt|\\.txt')"
done

Split total hits from content hits. A bot that fetches only robots.txt has not read a page, and a site with thousands of hits that are all health checks and asset requests has no crawl at all. The distinction between "a bot came" and "a bot read something" is where most crawl reports quietly mislead.

On static hosts that give you no access log, this layer is simply unavailable — say so in the audit rather than substituting a guess. An unmeasurable layer reported as "fine" is worse than one reported as unknown.

Layer 4 — indexing is the layer you probably cannot measure yourself

Crawled and indexed are different states, and conflating them is the most common error in agent-run audits. A page can be fetched daily and never listed.

The honest position: the authoritative answers live in Google Search Console and Bing Webmaster Tools, both free but both requiring an account you have verified. Scraping a search results page as a substitute is unreliable in a specific and dangerous way — the major engines disallow /search in robots.txt, and the ones that do not will eventually serve an anomaly page with zero results that looks exactly like "you are not indexed."

If an agent checks indexing by scraping, insist it runs a positive control in the same session — a small site that is certainly listed. If the control returns zero, the run proves nothing and the verdict must be withheld, not reported as absence.

A zero with no control behind it is not a finding. It is a blank.

Layer 5 — the on-page checks worth an agent's time

Once the layers above hold, on-page work starts paying. These are the checks that repeatedly find real defects rather than cosmetic ones:

• One h1 per page, and it matches the query intent. Not the brand name.
• Title under ~60 characters, front-loaded. The distinguishing words go first, because the tail gets truncated.
• Meta description written, not inherited. Platforms that auto-derive it from body copy often emit two thousand characters of raw markdown — check what is actually served, with curl, not what the CMS preview shows.
• Server-rendered content. Fetch the page with curl and grep for a sentence you can see in the browser. If it is absent, your content exists only after JavaScript runs, and you are depending on the renderer queue.
• Internal links that actually point somewhere. Orphan pages — reachable from the sitemap but from no other page — are common and invisible without a crawl.
• Canonical tags that agree with themselves. A canonical pointing at a different URL than the one serving it is a self-inflicted deindexing.

Make the agent prove its findings

The failure mode of an agent-run audit is not missing an issue. It is a confident, well-formatted report of issues it never verified — plausible, specific, and wrong. Two habits remove most of it:

1. Every finding cites the command that produced it and the raw output, not a summary. "Title too long" without the title is unfalsifiable.
2. Findings that could not be checked are listed as unchecked, in their own section, with the reason. An audit whose unknowns are visible is worth more than one that reads as complete.

A prompt that gets this reliably: "Audit in causal order — reachability, discovery, crawl, indexing, on-page, content. Stop at the first layer that fails and say so. Quote the command and its raw output for every finding. List anything you could not verify under UNCHECKED with the reason. Do not infer a layer from the one above it."

Common questions

Can Claude Code replace a paid SEO crawler?

For a site of a few dozen pages, an agent with shell access covers most of what a crawler reports — status codes, titles, headings, canonicals, internal links, sitemap coverage. Where paid tools stay ahead is scale, historical trend data, and index-status APIs that require credentials. For a small owned site the agent is usually enough; for a large one it is a supplement.

Why is my site crawled but not appearing in search?

Crawling and indexing are separate steps. A crawler fetching your pages daily is not a commitment to list them. Common causes include a very new domain with no inbound links, thin or duplicated content, a canonical tag pointing elsewhere, or a noindex header left over from staging. Search Console and Bing Webmaster Tools are the only sources that answer this definitively.

How do I check indexing without an account?

Reliably, you mostly cannot. Major engines disallow scraping their results pages, and the ones that permit it will serve a rate-limit page that mimics a zero-result page. If you try anyway, run a positive control — a small site you know is listed — in the same session, and treat any run where the control fails as inconclusive rather than negative.

What order should an SEO audit follow?

Causal order: reachability, then discovery, then crawl, then indexing, then on-page, then content. Each layer is meaningless if the one above it is broken, so stopping at the first failure saves the work that would have been wasted below it.