Toolsseosearch-console

Ask Google which of your pages it has indexed, one URL at a time

Christian Drexler

Two days after I moved this site to a new stack I wanted to know one thing: does Google know the new pages exist. Search Console could not tell me. The Pages report is a chart that lags by several days, it groups addresses into reasons, and the list behind each reason is capped and sampled. For a site that changed every URL last week, the report was still describing the old one.

There is an API that answers the question directly. You hand it one address, it tells you what Google's index knows about that address right now. Loop it over the sitemap and you have the real picture.

The call

The endpoint is the URL Inspection API, part of the Search Console API. It takes the address you want to know about and the property it belongs to.

from googleapiclient.discovery import build

svc = build("searchconsole", "v1", credentials=creds, cache_discovery=False)
res = svc.urlInspection().index().inspect(body={
    "inspectionUrl": "https://example.com/blog/some-post",
    "siteUrl": "sc-domain:example.com",
}).execute()

status = res["inspectionResult"]["indexStatusResult"]
print(status.get("coverageState"), status.get("lastCrawlTime"))

siteUrl is the property exactly as Search Console names it. A domain property is sc-domain:example.com, a URL-prefix property is the full address with the trailing slash. Get this string wrong and every call fails with a permission error that sounds like an authentication problem and is not.

The fields I actually read from the answer:

  • coverageState: the sentence you know from the interface, such as "Submitted and indexed", "Crawled - currently not indexed" or "URL is unknown to Google".
  • verdict: PASS, NEUTRAL or FAIL, handy for counting.
  • lastCrawlTime: when Googlebot last fetched the page. Empty means never.
  • userCanonical and googleCanonical: the canonical you declared and the one Google chose. When they differ, Google has decided your page is a copy of another one.

Credentials without a browser

For a script that runs unattended, a service account is the comfortable way. Create one in a Google Cloud project, enable the Search Console API there, download the key file, and then add the service account's email address as a user to the property in Search Console, under Settings, Users and permissions. Read access is enough for inspection.

That last step is the one people skip. The service account is a separate identity. It sees nothing until the property owner invites it, exactly like a colleague would not.

from google.oauth2 import service_account

creds = service_account.Credentials.from_service_account_file(
    "gsc-key.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)

Over the whole sitemap

The sitemap is already the list of addresses you want indexed, so it is the natural input. Pull the <loc> entries with a regular expression and run the inspections in a small thread pool.

import re, urllib.request
from collections import Counter
from concurrent.futures import ThreadPoolExecutor

SITE, SITEMAP = "sc-domain:example.com", "https://example.com/sitemap.xml"
urls = re.findall(r"<loc>([^<]+)", urllib.request.urlopen(SITEMAP).read().decode())

def inspect(u):
    svc = build("searchconsole", "v1", credentials=creds, cache_discovery=False)
    try:
        r = svc.urlInspection().index().inspect(
            body={"inspectionUrl": u, "siteUrl": SITE}).execute()
        s = r["inspectionResult"]["indexStatusResult"]
        return u, s.get("coverageState"), s.get("userCanonical"), s.get("googleCanonical")
    except Exception as e:
        return u, f"ERROR {e}"[:120], None, None

with ThreadPoolExecutor(10) as ex:
    rows = list(ex.map(inspect, urls))

print(Counter(r[1] for r in rows))
for u, state, mine, theirs in rows:
    if state != "Submitted and indexed":
        print(state, "|", u)
    elif mine and theirs and mine.rstrip("/") != theirs.rstrip("/"):
        print("canonical differs |", u, "->", theirs)

Each worker builds its own client, because the client object is not safe to share between threads. Ten workers get through about 160 addresses in two minutes.

The quota is 2,000 inspections per property per day and 600 per minute. For a site with a few hundred pages that is plenty for a daily run. For a site with fifty thousand pages it is a sample, and you would pick the pages that matter.

What it told me

The first run, two days after the move, printed this for 154 addresses: 153 times "URL is unknown to Google", once "Submitted and indexed". The one indexed page was the home page, and its last crawl date was from before the move. Google was still holding the old site and had not looked at the new one at all.

The reason was in the sitemap entry in Search Console. It had been submitted years ago, and the last successful fetch was months old. Google had no reason to come back. I resubmitted the sitemap, which is one more API call, and pushed the list to Bing through IndexNow on the same day.

Three and a half weeks later the same script printed 152 indexed out of 156. Two tool pages sat at "Crawled - currently not indexed", which is Google's way of saying it read the page and was not convinced. Two addresses of a brand-new post were still unknown. That is a list I can act on: two pages to improve, two to wait for. The chart in the interface would have shown me a green line going up.

Reading the three states that matter

URL is unknown to Google. Google has never fetched this address. Rewriting the page changes nothing, because nobody has read it. This is a discovery problem: check that the address is in the sitemap, that the sitemap was fetched recently, and that at least one other page links to it.

Discovered - currently not indexed. Google knows the address and has not fetched it yet. On a small site this usually clears within days. If it stays, the page has too few internal links pointing at it.

Crawled - currently not indexed. Google fetched the page and decided not to keep it. This is the one about substance. Compare the page with its siblings that made it into the index. It is usually shorter, or close to another page, or has little on it that is its own.

Where it earns its place

The sweep is most useful in three situations.

After a move or a redesign. You learn within days whether the new addresses are being picked up, well before the report catches up.

When Search Console sends the canonical mail. "Duplicate, Google chose different canonical than user" arrives without the list of pages. The sweep prints every address where the two canonicals differ. On another site of mine it pointed at two translated pages whose text was still in the original language. Google was right.

As a weekly habit. Counting the coverage states once a week and storing the numbers gives you a history the interface does not keep in this form.

It is read-only, and it changes nothing about how Google sees the site. It only replaces a guess with a list. Most of the time, that is the part that was missing.

seosearch-consolepythonsitemap