Avoid throttling when gsc-indexer runs over thousands of URLs
· 3 min read
Run gsc-indexer over thousands of URLs without Google throttling you: raise -delay, use -q for clean progress, and survive 429/5xx backoff.
A sitemap can resolve to 100+ URLs. At a polite delay that's a 15–20 minute run. These three flags keep it calm and in-control.
GSC rate limits are real
Google throttles sustained inspect volume. If you blast hundreds of requests with no delay, the run will stall or error partway through. This is Google's limit, not a bug in the tool: slow down and it works.
-delay: space out the requests
Default is 1s. For large batches, raise it:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -delay 10s -report ./report
-delay applies between every URL, including ones expanded from a
sitemap. Start at 8s–10s for big sitemaps and adjust if you still see
throttling.
-dry-run: preview before you commit
A dry run expands sitemaps, dedupes, and lists every URL that would be inspected. with no API calls and no credentials required:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -dry-run
Use it to confirm the count and contents before a long run. Great for catching a wrong sitemap URL early.
-q: quiet mode for long runs
By default the tool prints a block per URL. For a 100-URL run that's a lot of
noise. -q collapses it into a single live progress line with a remaining-time
estimate, then prints the summary:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -delay 10s -q -report ./report
A real 3-URL quiet run collapses to one line (imagine 300 URLs printing just one, not a block each):
Errors are still shown even in quiet mode, so you won't miss a failure.
⚠️ Flags must come BEFORE the URLs
This is the most common mistake. Go's flag parser stops at the first non-flag argument, so this fails:
./gsc-indexer -creds sa.json "https://www.toolsura.com/x/" -color always
## "-color" gets treated as a URL → 403 from Google
Put all flags first:
./gsc-indexer -creds sa.json -color always "https://www.toolsura.com/x/"
(The tool now catches this and prints a clear error instead of a confusing 403, but it's still best to order flags first.)
--wait on big runs: skip it
--wait polls Google after each URL until it indexes or the timeout hits,
which is exactly what you don't want on a 100-URL sitemap run. It's for one
important fresh URL, not a batch. Details in
understanding results.
Next
- Automating nightly? CI and -json.
- Hit an error anyway? Troubleshooting.
Sources
- Google Search Console
- URL Inspection API documentation
- Google Search APIs overview
- Service account OAuth (Google Identity)
- About service accounts (Google Cloud IAM)
- Verify site ownership
- Property types: URL-prefix vs Domain
- URL Inspection tool help (request indexing)
- Indexing API (JobPosting/BroadcastEvent only)
Written by
Abhay Khant
Hi, I'm Abhay Thakor
I'm a developer, builder, and technology enthusiast interested in turning ideas into useful products.
I enjoy working across web development, cybersecurity, automation, SEO, and content creation. Rather than only learning technologies individually, I like putting them together to build things that people can use.
One of my main projects is Toolsura, a free online utility platform that brings practical tools into one place. I'm building Toolsura as a product and a learning project where I can explore development, infrastructure, SEO, content distribution, automation, and product growth.
What I Build
My work currently revolves around several areas:
Web development with Next.js, React, Tailwind CSS, and modern web technologies
Backend development with Go, APIs, databases, and server-side systems
Automation using n8n, APIs, Docker, and connected services
SEO and content systems for building organic search traffic
Cybersecurity including web security, bug bounty, SOC concepts, and malware analysis
Content creation including video editing, YouTube content, graphics, and promotional videos
I enjoy going beyond tutorials and experimenting with complete systems, from the frontend interface all the way to the backend, deployment, automation, and distribution.
Toolsura
Toolsura is one of the projects I'm most actively building.
The idea is simple: create useful, accessible online tools that solve small problems quickly.
Toolsura includes utilities for things such as text, JSON, URLs, PDFs, images, security checks, and other everyday technical tasks.
Building it has also become a practical way for me to learn how a modern web product works.
That means working on:
Product → Development → Infrastructure → SEO → Content → Distribution → Automation → Growth
Technology & Learning
I'm particularly interested in understanding how systems work rather than simply learning how to use them.
My development work has included technologies such as:
Next.js · React · TypeScript · Tailwind CSS · Go · Gin · GORM · PostgreSQL · MongoDB · Docker · RabbitMQ · MinIO · Strapi
I've also worked with Linux environments and tools such as Kali Linux, Termux, and related security tooling.
My cybersecurity interests include web application security, bug bounty research, SOC analysis, and malware analysis.
Building in Public
I also create content around the things I build and learn.
This includes:
Tutorials and technical articles
YouTube videos
Product demonstrations
Visual explainers
Promotional videos
SEO-focused educational content
My goal is to turn projects and experiments into useful resources that other people can learn from.
Why I Build
For me, building is one of the best ways to learn.
Instead of learning a technology in isolation, I prefer starting with a real problem and figuring out what is required to solve it.
Sometimes that means writing code. Other times it's debugging a deployment at midnight, or working out why a page isn't being indexed, why an API isn't behaving correctly, or how to automate something that would otherwise take hours.
Those problems are part of the process.
Every project gives me another opportunity to understand how technology works in the real world.
What I'm Working Toward
I'm continuing to develop my skills across software engineering, cybersecurity, automation, and digital products.
At the same time, I'm building Toolsura into a useful platform and experimenting with different ways to create, distribute, and grow technical products.
There is still a lot to learn, but that's exactly what makes building interesting.
Let's Connect
If you're interested in web development, cybersecurity, automation, SEO, online tools, or building products, feel free to connect.
I'm always interested in learning, experimenting, and building something useful.
Build. Learn. Improve. Repeat.