Big runs without getting throttled: -delay, -q, and -dry-run
Posted on July 19, 2026 by Abhay khant
Key Takeaways
Run gsc-indexer over thousands of URLs without Google throttling you: raise -delay, use -q for clean progress, and survive 429/5xx backoff.
A sitemap can resolve to 100+ URLs. At a polite delay that's a 15–20 minute run. These three flags keep it calm and in-control.
GSC rate limits are real
Google throttles sustained inspect volume. If you blast hundreds of requests with no delay, the run will stall or error partway through. This is Google's limit, not a bug in the tool. slow down and it works.
-delay: space out the requests
Default is 1s. For large batches, raise it:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -delay 10s -report ./report
-delay applies between every URL, including ones expanded from a
sitemap. Start at 8s–10s for big sitemaps and adjust if you still see
throttling.
-dry-run: preview before you commit
A dry run expands sitemaps, dedupes, and lists every URL that would be inspected. with no API calls and no credentials required:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -dry-run
Use it to confirm the count and contents before a long run. Great for catching a wrong sitemap URL early.
-q: quiet mode for long runs
By default the tool prints a block per URL. For a 100-URL run that's a lot of
noise. -q collapses it into a single live progress line with a remaining-time
estimate, then prints the summary:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -delay 10s -q -report ./report
Errors are still shown even in quiet mode, so you won't miss a failure.
⚠️ Flags must come BEFORE the URLs
This is the most common mistake. Go's flag parser stops at the first non-flag argument, so this fails:
./gsc-indexer -creds sa.json "https://www.toolsura.com/x/" -color always
## "-color" gets treated as a URL → 403 from Google
Put all flags first:
./gsc-indexer -creds sa.json -color always "https://www.toolsura.com/x/"
(The tool now catches this and prints a clear error instead of a confusing 403, but it's still best to order flags first.)
Next
- Automating nightly? CI and -json.
- Hit an error anyway? Troubleshooting.
Sources
- Google Search Console
- URL Inspection API documentation
- Google Search APIs overview
- Service account OAuth (Google Identity)
- About service accounts (Google Cloud IAM)
- Verify site ownership
- Property types: URL-prefix vs Domain
- URL Inspection tool help
- Indexing API (JobPosting/BroadcastEvent only)
- Request indexing / inspect a URL
Frequently Asked Questions
Why did my large run stall or error partway?
Google rate-limits sustained inspect volume. If you blast thousands of requests with the default 1s delay, you hit HTTP 429 or 5xx and the run slows or fails. The fix is to slow down, not to retry harder, so raise -delay before a big batch.
What -delay should I use for a big site?
Start around 10s for thousands of URLs. That stretches the run to many minutes but keeps you under Google's throttle and avoids the backoff spiral. The tool already retries with backoff, but a polite delay prevents the throttling from starting in the first place.
How do I keep progress readable on a long run?
Add -q for a single live progress line with a remaining-time estimate instead of per-URL detail, and pair it with -report to capture the final split. For a sitemap, run -dry-run first so you know the URL count before committing to the wait.
What happens when Google returns 429 or 5xx?
The tool retries with backoff automatically, so a few transient errors are normal and recover on their own. Persistent 429s mean you are still too fast; raise -delay further. Only hard network or parse failures surface as actual run failures.
Is there a hard cap on URLs per run?
Sitemap expansion stops at 50,000 URLs as a safety limit. There is no cap on batch or stdin input, but a polite -delay is what keeps very large runs from being throttled. Size the delay to your volume rather than relying on retries to save you.