Noindex left over from staging? How to find it in production

A noindex meant for a preview or staging site ended up in production. How to find it in the HTML or headers, remove it safely and get pages back into Google.

· 6 min read

The site has been live for weeks. It loads, the pages look right, people you send the link to can use it. But a search for your own brand name finds nothing, and Google Search Console lists your pages under "Excluded by 'noindex' tag".

That message means a page told Google not to index it, and Google did as it was told. Nobody wrote "please hide my site" on purpose. In almost every case the rule was meant for a preview or staging copy of the site and made its way into production.

This guide covers where that rule hides, how to check for it in two minutes, and how to remove it without also exposing your staging site.

How a staging noindex ends up in production

noindex is a rule that tells search engines to keep a page out of their results. It can live in two places:

  • a meta tag in the HTML: <meta name="robots" content="noindex">
  • an HTTP response header: X-Robots-Tag: noindex

Google treats both the same way. Its guide to blocking indexing with noindex describes both forms and notes that the page must stay crawlable for the rule to be seen at all.

Teams add noindex to staging for a good reason: a second copy of the site in search results competes with the real one. The problem is how it gets switched off. Common ways it leaks:

  • An environment check that defaults the wrong way. Code like "add noindex unless ENV === 'production'" works until the production build runs without that variable set, or with a different value such as prod. Then production gets the staging behaviour.
  • A hardcoded tag in the template. An AI builder or a developer added the meta tag to index.html or the root layout while the site was private, and it stayed when the domain went live.
  • A framework setting. In Next.js, robots: { index: false } in the root metadata renders a noindex tag on every page that inherits it.
  • A CMS checkbox. WordPress has a "Discourage search engines from indexing this site" option under Settings → Reading. It is often ticked during the build and forgotten.
  • A hosting default on the wrong domain. Vercel adds X-Robots-Tag: noindex to preview deployments by default. Its response headers documentation explains when the header is added and when it is not. If your public site is actually served from a preview deployment or a branch domain rather than the production one, it can carry that header.
  • A copied robots.txt. Not noindex strictly speaking, but the same symptom: a staging robots.txt with Disallow: / shipped to production.

AI-built sites hit this more than most, because the switch from "testing" to "live" is often a single click, and nobody reviews which settings came along with it.

Check it yourself

You need a terminal or a browser. Replace yourdomain.com with your domain and check more than just the home page: pick a product page, a blog post, a landing page.

1. Look for the meta tag in the raw HTML

curl -s https://yourdomain.com/ | grep -io '<meta[^>]*name="\(robots\|googlebot\)"[^>]*>'

No output is good. If you see content="noindex" (or noindex, nofollow), you found it.

In a browser: open the page, press Ctrl+U (Cmd+Option+U on a Mac) to view the source, and search for robots.

2. Look for the tag after JavaScript runs

On single-page apps, a script can add the tag after the page loads. The raw HTML is clean, but Google renders the page and sees the tag anyway. Open the page, open DevTools (F12), go to the Console and run:

[...document.querySelectorAll('meta[name="robots"], meta[name="googlebot"]')].map(m => m.content)

An empty list is good. Anything with noindex is a problem, even if step 1 was clean. If you want to know more about the gap between raw and rendered HTML, see React SPA serves empty HTML? What Google sees.

3. Look for the header

curl -sI https://yourdomain.com/ | grep -i x-robots-tag

No output is good. x-robots-tag: noindex means the server or the hosting platform is adding it. Check www and the bare domain separately: they can be served by different configurations.

4. Check robots.txt

curl -s https://yourdomain.com/robots.txt

A line Disallow: / under User-agent: * blocks the whole site from crawling.

5. Ask Google what it saw

In Google Search Console, paste a page address into URL Inspection. The result shows whether indexing is allowed and whether the page was blocked by robots.txt. The Page indexing report lists every page excluded by a noindex tag, which tells you whether the problem is one template or the whole site.

How to fix it

Find the source first. Deleting the tag in the built output only lasts until the next deploy.

  1. Search the code for noindex, robots, X-Robots-Tag and index: false. Check templates, layout files, next.config, vercel.json, _headers, nginx configs and middleware.
  2. Make production the safe default. The rule should be "add noindex only when this is clearly a preview or staging build", not "add noindex unless this is production". A missing variable should then lead to an indexable site, not a hidden one.
  3. Keep staging hidden another way. Password protection or your host's deployment protection keeps staging private for people and crawlers alike. Google itself says robots.txt is not a way to keep pages out of search; noindex or a password is.
  4. Check which domain serves production. On Vercel, the public domain should point at the production deployment, not a preview or branch.
  5. Fix robots.txt if it blocks everything, and list your sitemap in it.
  6. Ask Google to recrawl. In Search Console, use URL Inspection → Request indexing for your main pages, and resubmit the sitemap. Pages return as Google recrawls them; it is not instant.

One detail that trips people up: if robots.txt blocks a page, Google cannot read the page at all, so it also cannot see that you removed the noindex. Fix robots.txt first, then the tag.

A prompt you can give Lovable, Cursor or Claude Code:

My production site is excluded from Google because of a noindex rule.
Find every place in this project that can add noindex: meta robots or
googlebot tags in HTML templates and layouts, framework metadata such as
Next.js robots: { index: false }, X-Robots-Tag headers in vercel.json,
_headers, next.config, middleware or server config, and robots.txt rules.
For each one, tell me the file, the line and under which conditions it applies.
Then change the logic so noindex is added only when the build is explicitly
a preview or staging environment, and production is indexable when the
environment variable is missing. Do not remove protection from staging.
Show me the diff before applying it.

After the fix is deployed, run the curl commands from the previous section again on the production domain, not on a preview URL.

Keep it from coming back

A leftover noindex is quiet. The site works, nothing errors, and the first sign is traffic that never arrives. It also tends to come back: a new layout, a framework upgrade or a change in how environments are set up can re-enable the staging branch of the logic.

Two habits help:

  • Put the curl checks from this guide in your release checklist, or in a CI step that fails if the production HTML or headers contain noindex.
  • After switching domains or hosting, check the Page indexing report in Search Console a few days later.

If the site is new and never showed up at all, Why your Lovable site isn't showing up on Google walks through the other common causes.

ScanYourSite reads your home page as raw HTML and as rendered by a real browser, and flags a meta robots noindex in either, including one added by JavaScript. It also flags a robots.txt that blocks the whole site. The free check covers one site with a weekly check and a weekly summary email. Pro checks daily and after every deploy and alerts you right away, so a noindex that slips back in shows up the day it ships. The header (X-Robots-Tag) and inner pages are still worth checking with the commands above. You can see what it checks for AI-built sites, or run it now: https://scanyoursite.app/?check=yourdomain.com

Check your own site in a minute

Free, no sign-up. You get a prioritised list of what’s wrong and a prompt you can paste back into Lovable, Bolt, Cursor or Claude Code.

Free, no sign-up. The basic check makes only the requests an ordinary browser would.

SEO for AI-built sites