Indexing many URLs: batch files, stdin, and sitemaps
Posted on July 19, 2026 by Abhay khant
Key Takeaways
Submit URLs to Google in bulk with gsc-indexer: batch files, stdin pipes, and sitemap expansion into every <loc> (capped at 50,000 URLs).
One URL at a time is fine for a quick check, but the tool is built for lists. Here are the three ways to feed it many URLs. and they all compose.
1. Batch file (-batch)
Put one URL per line. Lines starting with # are ignored, as are blank lines.
urls.txt:
## pages to refresh
https://www.toolsura.com/post-a/
https://www.toolsura.com/post-b/
https://www.toolsura.com/post-c/
./gsc-indexer -creds sa.json -batch urls.txt
2. Stdin pipe
Pipe a list straight in. handy in scripts:
cat urls.txt | ./gsc-indexer -creds sa.json
The tool auto-detects a piped stdin and reads from it (it only reads stdin when input isn't coming from an interactive terminal, so your keyboard won't block it).
3. Positional arguments
Multiple URLs as plain arguments also work:
./gsc-indexer -creds sa.json "https://www.toolsura.com/a/" "https://www.toolsura.com/b/"
4. Sitemaps (.xml)
Hand it a sitemap URL and it fetches the file and expands it into every
<loc> entry:
./gsc-indexer -creds sa.json "https://www.toolsura.com/sitemap.xml" -report ./report
- Sitemap indexes are followed recursively. a sitemap that lists child
sitemaps (
<sitemap><loc>) is expanded all the way down. - This is the easiest way to (re-)request indexing for an entire site.
All sources compose and dedupe
Mix them freely. the tool merges batch + args + sitemap + stdin, removes duplicates, and runs the unique set together:
./gsc-indexer -creds sa.json -batch urls.txt "https://www.toolsura.com/extra/" "https://www.toolsura.com/sitemap.xml"
The URL cap
Expansion is capped at 50,000 URLs to protect against a runaway sitemap hanging the run. If you somehow exceed it you'll get a clear error telling you so. For context, a normal blog sitemap is a few hundred URLs.
Next
- Track progress across runs: -report and -diff.
- Big sitemap run stalling? Throttling and quiet mode.
Sources
- Google Search Console
- URL Inspection API documentation
- Google Search APIs overview
- Service account OAuth (Google Identity)
- About service accounts (Google Cloud IAM)
- Verify site ownership
- Property types: URL-prefix vs Domain
- URL Inspection tool help
- Indexing API (JobPosting/BroadcastEvent only)
- Request indexing / inspect a URL
Frequently Asked Questions
How do I submit a list of URLs?
Put one URL per line in a file, with # comments and blank lines ignored, then run './gsc-indexer -creds sa.json -batch urls.txt'. You can also pipe the same list through stdin. Both methods are deduplicated with the other sources before the run starts.
Can I inspect a whole sitemap at once?
Yes. Pass any argument ending in .xml and the tool fetches it and expands every into a URL to inspect. Nested sitemap indexes are followed recursively, so one sitemap index can resolve to your entire URL set without you listing anything by hand.
Is there a limit on how many URLs a sitemap can produce?
Expansion is capped at 50,000 URLs as a safety limit, which is far more than most sites need. For very large properties, use a polite -delay so Google does not throttle the burst of inspection requests the tool sends on your behalf.
How do positional URLs, batch, and sitemaps combine?
They are all composable and merged, then deduplicated, then run together. You can pass a couple of URLs, a -batch file, and a sitemap in the same command, and the tool will inspect the union of all three with no duplicates.
Should I preview a big sitemap run first?
Always. Run with -dry-run to see the full expanded URL count and contents without calling the API. At a polite -delay of 10s, a 100-URL sitemap is a 15 to 20 minute run, so confirming the set first saves you from a long run with a wrong URL list.