CaptureGuide
7 min read
How to find all the pages of a website, and save every one of them
A website has no built-in list of its pages, so you assemble one from its sitemap, from a search engine, from Search Console if the site is yours, or from a crawler that follows its links. Here is what each source finds and misses, and how to turn the list into one file per page.
No website comes with a complete list of its pages. What you can get is a list from one of four
places: the sitemap the site publishes for search engines, a site: search, the Page indexing
report in Google Search Console if the site is yours, or a crawler that starts at the home page
and follows every link. Each one misses something different, so for a site that matters, use two
and compare. Once you have the list, the last step is turning it into files, one per page, which
Page Scanner's command line does from a text file of addresses.
The four sources at a glance
| The sitemap | A site: search | Search Console | A crawler | |
|---|---|---|---|---|
| Where the list comes from | a file the site publishes | what Google has indexed | what Google crawled, for your site | the site's own links, followed from one page |
| What it misses | pages the owner left out of it | "not always exhaustive" | over 1,000 URLs in one list | pages no link reaches |
| What you need | a browser, or curl | a browser | the site added to your account | a tool installed or running in your browser |
| Pages behind a sign-in | ✗ only what the owner lists | ✗ | ✗ | ✓ if it runs signed in, as Page Scanner does |
| Good for | a first list in a minute | a quick look, not a count | your own site, indexed or not | finding what the sitemap forgot |
✓ yes, ✗ no.
1. Read the site's sitemap
Most sites publish a sitemap for search engines: an XML file listing the addresses the owner wants
crawled. Try /sitemap.xml at the root of the site first. If it is not there, the site's
robots.txt may say where it is, since the sitemaps protocol lets a site name its sitemap there
with a Sitemap: line:
curl -s https://example.com/robots.txt | grep -i '^sitemap:'To turn the sitemap into a plain list of addresses, one per line, in a terminal on a Mac or Linux:
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' > pages.txtA large site splits its sitemap: the protocol allows at most 50,000 addresses in one file, and a
site with more lists its sitemap files in a sitemap index instead. If the addresses in pages.txt
end in .xml, that is what you have; run the same command on each of them.
What it misses: the sitemap is the owner's list, not the site. A page they left out, or one a CMS never added, is not in it, and an old one may still be there after the page is gone. It is the fastest first list, and the one to check against the others.
2. Search with site:
Type site:example.com into Google and you see pages it has indexed under that address, and
site:example.com/blog/ narrows it to a folder. It needs nothing but a browser.
It is not a count. Google's own documentation says the operator "doesn't necessarily return all the URLs that are indexed", that "the list of URLs returned is not always exhaustive", and that "bigger sites shouldn't expect to see all their URLs in the results." Use it to see whether a page is indexed at all, or to spot a section you did not know about, not as the list.
3. If the site is yours: Search Console
For a site you have added to Google Search Console, the Page indexing report "shows how many URLs on your site have been crawled and indexed by Google", and the ones it found but did not index, with the reason for each. View data about indexed pages lists them, and each reason opens a table of example pages.
Each of those lists "is limited to 1,000 items, and isn't guaranteed to show all URLs in a given
status." So it is the best list for a small site you own, and for a large one it tells you where
to look rather than everything there is. Its other use is the pages Google found but did not
index, which neither the sitemap nor a site: search shows you.
4. Crawl it by following links
A crawler opens a page, collects its links, opens those, and keeps going until it runs out of new addresses on the site. It finds what the site links to, whether or not the sitemap lists it, and reports what a visitor would hit along the way, such as a link that answers 404.
Screaming Frog SEO Spider is a desktop crawler for Windows, macOS and Linux. In its words, "The free version is restricted to crawling up to 500 URLs in a single crawl," and a paid licence removes the limit. Its page gives the price as "$279 Per Year" in one place and "£199 per year" in another. It crawls the HTML a page is served by default and can render pages with Chromium, and the same page says both that the free version has JavaScript rendering and that it does not, so check before you rely on it. It exports what it found as a list.
Page Scanner's Map this site is a crawler inside your browser, for a site you are signed in to
or a small one. Open the start page, click the Page Scanner icon and press Map this site; it
reads the site's pages with your browser's cookies, two or three links deep, keeps to that site,
follows each host's robots.txt except for the page you started on, and lists what it found
before it captures anything. It then captures up to 30 pages, one of each kind (a blog's posts are
one row with a count), and draws them as a map joined by their links. It is a preview, and how it
works is in how to capture every page of a website.
What every crawler misses: a page that no link reaches, such as a landing page used only in ads or an old page that was unlinked but not deleted. That is what the sitemap and Search Console are for.
Then save every page
With the addresses in pages.txt, Page Scanner's command line captures each one in your own
Chrome, so pages behind your sign-in come out as you see them, and writes one file per page: a PDF
whose text you can select and search, or a PNG or JPEG. It needs Node.js 24 or newer and the Page
Scanner extension, connected once under Local Agents in its settings
(installing the CLI).
npx @page-scanner/cli installnpx @page-scanner/cli scan --urls pages.txt --out ./site/ --name '{n}-{host}'The pages are scanned one at a time, in order, into the ./site/ folder, and {n} numbers the
files in the order of the list. A page that fails is reported with its address while the rest
carry on. One list takes up to 1,000 addresses, so split a longer one first, for example with
split -l 1000 pages.txt part-, and run each part. Add --format png for images instead of
PDFs, or --markdown beside to keep each page's text as Markdown next to its file.
If the site is small and you would rather not use a terminal, Map this site above captures up to 30 of its pages and opens any of them in the editor to crop, mark up and export. For a handful of pages you already have open, select the tabs and scan them into one document, as in saving several tabs as one PDF.
Frequently asked questions
Is there a way to see every page, guaranteed? Only from the inside: the site's CMS, its server or its database knows what it serves. From the outside, the sitemap, Search Console and a crawl together come closest, and comparing them is how you find the pages one of them missed.
Why does the crawler find pages the sitemap does not have? The sitemap is written by the owner or the CMS and kept up to date only as well as they keep it. A crawler sees the links that are on the pages now.
Can I list the pages of a site I have to sign in to? A sitemap, a site: search and Search
Console only know what the public can reach. A crawler that runs in your signed-in browser can
list the rest: Page Scanner's map reads pages with your own cookies, and never follows a link
whose address says it signs out, deletes or unsubscribes.
Does it matter that some links are added by JavaScript? It does for a crawler that reads only the HTML a page is served, which includes Page Scanner's map: a link drawn afterwards by a script is not followed. A sitemap or Search Console still lists those pages, and the CLI captures each one as your browser renders it.
What about comparing the pages later? Keep the files, scan the same list again another day, and compare two captures of a page, as in how to compare two versions of a web page.
Where this comes from
- Sitemaps: the sitemaps.org protocol, on the
Sitemap:line inrobots.txt, the 50,000-address limit and sitemap index files. site:search: Google Search Central, the site: search operator.- Search Console: Search Console Help, Page indexing report.
- Screaming Frog: the SEO Spider's page.
- Page Scanner: Map a site and the CLI's commands.