Website Migration Indexing Triage: How to Diagnose Quality, Originality & Crawl Signals When Pages Won’t Reindex

• 10–50 representative URLs: key templates (product, category, article/docs, location) plus at least one “problem” directory.
• Old → new URL mapping.
• Google Search Console access (URL Inspection, Sitemaps, Indexing).
• Server/CDN/WAF access (or someone who can check rules/logs and caching).

In practice, I start with GSC URLs showing “Crawled — currently not indexed” or “Duplicate,” then add a few from the XML sitemap and top landing pages. Following the order keeps you from rewriting pages that still can’t be crawled—or can’t be selected as canonical.

Step 1 — Confirm crawl access and consistent fetch behavior

Goal: prove Google can reliably fetch the final page version.

1) Check the final HTTP status for each sample URL

• Target: 200 OK on the canonical URL.
• Common migration drags: redirect chains/loops, inconsistent 200/3xx behavior, intermittent 5xx from origin/CDN.

Worked example: 3-hop redirect chain → single hop

When I ran post-migration checks, I often saw:

http://example.com/page
  → 301 https://example.com/page
    → 301 https://www.example.com/page/
      → 200 https://www.example.com/page/

Fix: pick one preferred host + slash policy and redirect in one step:

http://example.com/page  → 301 https://www.example.com/page/
https://example.com/page → 301 https://www.example.com/page/

2) Robots.txt + meta robots

• robots.txt Disallow blocks crawling.
• meta robots noindex allows crawl but blocks indexing.
• After migrations, check key templates for accidental noindex, especially categories and docs.

3) HTTPS/host consistency + UA parity

• Ensure http/https and www/non‑www resolve cleanly (no loops).
• Confirm Googlebot sees the same core content as users. WAF/CDN rules, device detection, geo-variants, or edge caching can produce mismatched renders.

Decision: if any item fails, fix it before touching canonicals or content.

Step 2 — Validate index-selection signals: canonicals, duplicates, and URL variants

Once Google can crawl, remove ambiguity: one content item → one indexable URL.

1) Canonical correctness

• Usually: self-referential canonical on the chosen URL.
• Common migration failure: canonicals still point to old domains/paths or to a nearby “equivalent” page.

Worked example: canonical still points to old URL

Problem:

URL: https://newsite.com/widgets/blue-widget
<link rel="canonical" href="https://oldsite.com/products/blue-widget" />

Correction:

<link rel="canonical" href="https://newsite.com/widgets/blue-widget" />

2) Kill duplicate clusters

• Old and new URLs both return 200 (old site not fully redirected).
• Public staging copies.
• Indexable variants: parameters, trailing slash/case, printer/mobile.

3) Normalize variants cleanly

• Enforce one version for host, slash, and lowercase.
• Handle parameters consistently (redirect, canonicalize, or noindex—don’t mix signals).
• For pagination, keep intent aligned; don’t canonical page 2 to page 1 unless you truly want only page 1 indexed.

4) Sitemap hygiene

• Include only canonical 200 URLs you want indexed.
• Remove redirected, non-canonical, noindex, and parameterized/faceted variants.

Decision: if canonicals/duplicates/sitemaps are wrong, fix and wait for recrawl. Don’t rewrite content until Google can identify the canonical set.

Step 3 — Diagnose quality, originality, and soft-404 signals suppressing reindexing

If Step 1 and Step 2 are clean and you still see “Crawled — currently not indexed”, “Duplicate,” or soft‑404 behavior, treat it as a selection problem. Google can fetch the page, but it looks low-value, duplicative, or effectively empty.

1) Template-induced thinness (migration regression)

In audits, the most common regressions are template changes that shrink unique main content:

• Copy moved into collapsed tabs that aren’t rendered server-side.
• Specs/reviews/comparisons removed.
• Above-the-fold becomes mostly nav, hero, and boilerplate.

2) Originality regression: old vs new

Compare old HTML to new on 5–10 non-indexing URLs:

• What unique sections disappeared?
• Did many pages become near-identical due to shared blocks?
• Was specific copy replaced with generic text?

“Same intent, less substance” is enough for Google to skip reindexing.

3) Soft 404 patterns (even with 200 status)

Watch for category pages with few/zero items, heavy empty-state messaging, doorway-like location near-duplicates, and indexable internal search results.

4) Sitewide vs page-level

If non-indexing clusters by template/folder, fix the template first. One good template fix can move an entire directory.

Decision: do template/content work only after crawl access and canonical signals are stable.

Supporting — Recovery actions after blockers are fixed (crawl signals that help reindexing)

Once Step 1 and Step 2 are green, push recrawl signals that tend to work in practice.

• Restore internal linking: link priority migrated pages from crawlable hubs (nav, categories, related modules, HTML sitemap). When I ran one migration audit, we found “valid” URLs that were effectively orphaned after a menu change.
• Clean redirect mapping: remove chains/loops, avoid mass “redirect to home,” keep redirects to one hop.
• Use reindex nudges selectively: submit a short priority list (top traffic/revenue), and refresh sitemap lastmod only for pages you actually changed.

These help once fundamentals are correct. They won’t override noindex, robots blocks, or duplicate clustering.

Conclusion — Your next 3 moves and when to escalate

Run the triage in this order: crawl access → index signals → quality/originality/soft‑404. It prevents busywork and surfaces the real failure mode.

Your next 3 moves:

1) Fix crawl blockers: get canonical URLs to 200, remove chains/loops, correct robots/noindex mistakes, and confirm Googlebot gets the same core content as users.

2) Correct index-selection signals: ship self-referential canonicals, eliminate duplicate 200s (old site, staging, variants), and keep sitemaps limited to canonical 200 URLs.

3) Repair template-level value where needed: restore missing main content, reduce near-duplicate sets, and fix soft‑404 intent mismatches (empty categories, doorway-like location pages).

Escalate if Step 1 and Step 2 are clean but non-indexing is widespread across a directory/template. Prioritize template/content remediation and internal linking, and consider a rollback if the new template removed substantial main content at scale. Expect reprocessing to take days to weeks depending on crawl demand, so focus changes by template instead of repeatedly submitting the same URL.

Sources

1. Google Search Central: Canonicalization
2. Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag
3. Google Search Central: Control crawling and indexing with robots.txt