go-sitemap-parser
Go language library for parsing Sitemaps
About go-sitemap-parser
This large README is shown as a plain-text preview. Read the full README at the source
# go-sitemap-parser
[](https://codecov.io/gh/aafeher/go-sitemap-parser)
[](https://github.com/aafeher/go-sitemap-parser/actions/workflows/go.yml)
[](https://pkg.go.dev/github.com/aafeher/go-sitemap-parser)
[](https://scorecard.dev/viewer/?uri=github.com/aafeher/go-sitemap-parser)
[](https://github.com/avelino/awesome-go)
A Go package to parse XML Sitemaps compliant with the [Sitemaps.org protocol](http://www.sitemaps.org/protocol.html).
> For information on reporting security vulnerabilities, see [SECURITY.md](SECURITY.md).
## Features
- Recursive parsing (sitemap index → sitemaps → URLs)
- The sitemaps a call finds can be queried, whether or not they were fetched, with the date a sitemap index gives for each
- The date a page was last modified is returned for the pages of RSS and Atom feeds as well (`<pubDate>` or `<dc:date>`, `<updated>`, `<modified>` or `<published>`)
- Iterators over the URLs and the errors of a call (`URLs()`, `Errors()`), the URLs as copies that are the caller's own
- Concurrent (multi-threaded) fetching and parsing
- Configurable follow rules to filter which sitemaps to parse
- Configurable URL rules to filter which URLs to include
- Configurable HTTP response size limit
- Limits on the sitemaps a call fetches and on the URLs it collects, so that a call comes to an end whatever a server lists
- A limit on the validation errors a call records, so that its errors are bounded like its results are
- Tolerant mode (default): resolves relative URLs in `<loc>` elements; rejects URLs exceeding 2,048 characters after resolution
- Strict mode: validates URLs per the sitemaps.org specification
- A value that cannot be parsed (e.g. a malformed `<lastmod>`) costs only itself, never the whole sitemap
- Google Image Sitemap extension (`<image:image>`)
- Google News Sitemap extension (`<news:news>`)
- Google Video Sitemap extension (`<video:video>`)
- XHTML hreflang extension (`<xhtml:link>`)
- XML documents in encodings other than UTF-8 (e.g. `ISO-8859-1`, `windows-1252`) are transcoded automatically
- Typed errors: `*ConfigError`, `*NetworkError`, `*ParseError`, `*ValidationError` — inspectable via `errors.As`
- A limit that was reached can be told with `errors.Is`: `ErrMaxResponseSize`, `ErrMaxDepth`, `ErrMaxSitemaps`, `ErrMaxURLs`, `ErrMaxValidationErrors`
- Thread-safe
## Formats supported
- `robots.txt`
- XML `.xml`
- RSS 2.0
- Atom 1.0
- Plain text `.txt`
- Gzip compressed files (e.g., `.xml.gz`, `.txt.gz`)
XML documents do not have to be UTF-8 encoded, and a document may begin with a UTF-8 byte order mark, see [Character encoding](#character-encoding).
A document of any of these formats may be gzip compressed. A gzip file is read to the end of its last member: a file that was appended to, or put together from several (`cat a.gz b.gz`), holds a member for each part, and its content is that of all of them. What follows the last member and is no member, such as a newline, is ignored. Content that cannot be decompressed to its end is not parsed, and is reported once, as a `*ParseError` (`gzip decompression failed: unexpected EOF`). See [`examples/gzip`](examples/gzip/main.go) for a runnable example.
A `robots.txt` is recognised by its URL: the document passed to `Parse()` is read as one when the last segment of the path of its URL is `robots.txt`. The letter case makes no difference, and neither does a query or a fragment that follows the path, so `https://example.com/robots.txt?v=2` and `https://example.com/ROBOTS.TXT` are read as a `robots.txt` too. What follows the path is no part of it: `https://example.com/sitemap.xml?from=/robots.txt` is the URL of a sitemap, and is parsed as one. The URL the document was served from counts as well as the one passed in: a URL of another name that is redirected to a `robots.txt` is read as one, and so is a `robots.txt` that is redirected to a URL of another name, see [Redirects](#redirects). The URLs a request passes between the two do not count. Only the document passed to `Parse()` can be a `robots.txt`: one that a sitemap index lists is parsed as a sitemap.
Of a `robots.txt` only the `Sitemap:` lines are read. The name of the directive is not case-sensitive, the line may be indented, and what follows a `#` is a comment. The sitemaps the lines name are fetched and parsed; how their URLs are checked is described under [Strict mode](#strict-mode). See [`examples/robots`](examples/robots/main.go) for a runnable example.
Of a feed, the location of a page and the date it was last modified are taken, of a text sitemap the location only. `ChangeFreq` and `Priority` are `nil` for their URLs, and they have nothing of the extensions:
- RSS: the `<link>` of every `<item>`, and the `<pubDate>` of the item as `LastMod`, or its `<dc:date>` if it has no `<pubDate>`, see [`examples/rss`](examples/rss/main.go)
- Atom: for every `<entry>`, the first `<link>` with `rel="alternate"` or without a `rel` attribute, and the `<updated>` of the entry as `LastMod`, or its `<modified>` or `<published>` if it has no `<updated>`, see [`examples/atom`](examples/atom/main.go)
- Plain text: every line that begins with `http://` or `https://`; empty lines, lines that begin with `#` and every other line are skipped, see [`examples/text`](examples/text/main.go). `LastMod` is `nil`
The date of a feed entry is read in the format its feed prescribes:
- `<pubDate>` holds a date and time of [RFC 822](https://www.rfc-editor.org/rfc/rfc822#section-5), such as `Mon, 15 Jan 2024 10:30:00 GMT`. The day of the week and the seconds may be left out, the year has four digits or two (`24` is 2024, `69` to `99` are 1969 to 1999), and the time zone is an offset (`+0100`) or one of the names `UT`, `GMT`, `UTC`, `Z`, `EST`, `EDT`, `CST`, `CDT`, `MST`, `MDT`, `PST` and `PDT`. A name stands for the zone the RFC says it does, whatever the time zone of the machine is. Another name (`CET`) is not known, and the date is not read
- `<updated>` holds a date in one of the formats of a `<lastmod>`, among them the one Atom prescribes ([RFC 3339](https://www.rfc-editor.org/rfc/rfc3339), `2024-01-15T10:30:00Z`)
- `<dc:date>`, `<modified>` and `<published>` hold a date in one of the formats of a `<lastmod>` as well
An entry may give its date in another element than the one its format has for it, and that element is read where the other is absent or empty:
- An RSS item without a `<pubDate>` takes its date from its `<dc:date>`, the date of [Dublin Core](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/elements11/date/), which feeds use for a date in the format the rest of the web writes its dates in. It is the one element of another namespace that is read, and it is told by that namespace (`http://purl.org/dc/elements/1.1/`), whatever its prefix.
- An Atom entry without an `<updated>` takes its date from its `<modified>`, which is what Atom 0.3 calls the element, and if it has neither, from its `<published>`, the date the page was first available.
The element of the format comes first wherever it stands in the entry. One that holds something that is no date is the date of its entry all the same: it is reported, and the other element is not read in its place.
The elements of a feed are told from those of other formats by their namespace. A feed may hold both: an RSS item often has an `<atom:link>` next to its `<link>`. An element is read only if it is in the namespace of the `<item>` or of the `<entry>` it stands in, which is none for an RSS item and that of Atom for an Atom entry. The `<dc:date>` of an RSS item is the one exception. An `<atom:link>` is therefore neither the `<link>` of an RSS item nor read in its place, wherever it stands in the item, and an item that has no other link is skipped like one without a link.
The same holds for the entries of a sitemap and of a sitemap index, which may hold elements of other formats as well. The `<loc>`, `<lastmod>`, `<changefreq>` and `<priority>` of a `<url>`, and the `<loc>` and `<lastmod>` of a `<sitemap>`, are read only if they are in the namespace of the entry they stand in: that of the protocol, or none for a document that declares no namespace. An element of that name of another namespace, an `<x:loc>` for one, is neither the element of the entry nor read in its place, wherever it stands and whatever it holds. The elements of the extensions are told by their own namespace: an `<image:image>` is one of `http://www.google.com/schemas/sitemap-image/1.1`, whatever its prefix.
An entry without a date has none, and so has one whose date is empty. A date in another format is left unset (`nil`) and reported as a `*ValidationError`, and the page is returned all the same, in both modes, see [Strict mode](#strict-mode). Only the date of the entry itself is read: the dates of the feed are not (`<lastBuildDate>`, `<pubDate>` and `<dc:date>` of an RSS channel, the `<updated>` and `<modified>` of an Atom feed), and neither are the `<issued>` and `<created>` of an entry of Atom 0.3.
## Requirements
Go 1.26 or later.
## Installation
```bash
go get github.com/aafeher/go-sitemap-parser
```
```go
import "github.com/aafeher/go-sitemap-parser"
```
## Usage
### Create instance
To create a new instance with default settings, you can simply call the `New()` function.
```go
s := sitemap.New()
```
### Configuration defaults
- userAgent: `"go-sitemap-parser (+https://github.com/aafeher/go-sitemap-parser/blob/main/README.md)"`
- fetchTimeout: `3` seconds
- maxResponseSize: `52428800` (50 MB)
- maxDepth: `10`
- maxConcurrency: `16`
- maxSitemaps: `50000`
- maxURLs: `10000000`
- maxValidationErrors: `100000`
- multiThread: `true`
- strict: `false`
- httpClient: `nil` (a default `*http.Client` is created per call with the configured `fetchTimeout`)
### Overwrite defaults
#### User Agent
To set the user agent, use the `SetUserAgent()` function.
```go
s := sitemap.New()
s = s.SetUserAgent("YourUserAgent")
```
... or ...
```go
s := sitemap.New().SetUserAgent("YourUserAgent")
```
#### Fetch timeout
To set the fetch timeout, use the `SetFetchTimeout()` function. It should be specified in seconds as a **uint16** value (1–65535 seconds). A value of `0` is rejected and a `*ConfigError` is recorded. Note: when a custom HTTP client is set via `SetHTTPClient()`, this value has no effect — the client's own `Timeout` field controls the request deadline.
```go
s := sitemap.New()
s = s.SetFetchTimeout(10)
```
... or ...
```go
s := sitemap.New().SetFetchTimeout(10)
```
#### Max response size
To set the maximum allowed HTTP response size, use the `SetMaxResponseSize()` function. It should be specified in bytes as an **int64** value. The default is 50 MB, matching the [sitemaps.org protocol](http://www.sitemaps.org/protocol.html) limit. Responses exceeding this limit will result in an error.
The same limit caps the **decompressed** size of gzip-compressed content, of all the members of a gzip file together, so a small `.gz` response cannot expand without bound in memory. This applies both to fetched content and to gzip content passed in through the `urlContent` argument of `Parse()`. Content that expands beyond the limit is rejected and reported via `GetErrors()` as a `*ParseError`. If it is the content of the URL passed to `Parse()`, the call fails with that error, see [Parse](#parse).
Both errors wrap `ErrMaxResponseSize`, the `*NetworkError` of a response over the limit and the `*ParseError` of gzip content over it, see [Limit errors](#limit-errors).
The value must be greater than 0. Unlike with `SetMaxSitemaps()`, `SetMaxURLs()` and `SetMaxValidationErrors()`, `0` does not lift the limit: it is rejected with a `*ConfigError` and the limit stays as it was. Every value above it is a limit, up to `math.MaxInt64`, which nothing can exceed.
```go
s := sitemap.New()
s = s.SetMaxResponseSize(10 * 1024 * 1024) // 10 MB
```
... or ...
```go
s := sitemap.New().SetMaxResponseSize(10 * 1024 * 1024) // 10 MB
```
See [`examples/maxresponsesize`](examples/maxresponsesize/main.go) for a runnable example.
#### Max depth
To set the maximum recursion depth for following sitemap indexes, use the `SetMaxDepth()` function. A sitemap index may reference other sitemap indexes; this limits how many levels deep the parser will follow. The default is 10.
```go
s := sitemap.New()
s = s.SetMaxDepth(5)
```
... or ...
```go
s := sitemap.New().SetMaxDepth(5)
```
With `SetMaxDepth(1)` the sitemaps listed by the document passed to `Parse()` are fetched, but if one of them is a sitemap index itself, the sitemaps it lists are not. A `robots.txt` does not count as a level: the sitemaps it names are treated like the document passed to `Parse()`.
When the limit is reached, a `*ParseError` is recorded in `GetErrors()` for every sitemap index whose sitemaps are not followed. It names that sitemap index, by the URL it was served from:
```
parse "https://example.com/sitemap-index-2.xml": max recursion depth of 1 reached
```
The error wraps `ErrMaxDepth`, see [Limit errors](#limit-errors). Reaching the limit does not fail `Parse()`: the URLs collected up to that depth are returned.
See [`examples/maxdepth`](examples/maxdepth/main.go) for a runnable example.
#### Max sitemaps
A sitemap index may list sitemap indexes, which may list further ones, so how many sitemaps a call ends up fetching is up to the server. To limit the number of sitemaps a `Parse()` / `ParseContext()` call fetches, use the `SetMaxSitemaps()` function.
The value is an `int`:
- `0`: no limit.
- a positive value: the call fetches at most that many sitemaps. The default is `50000`, which is as many sitemaps as a sitemap index may list according to the sitemaps.org protocol.
```go
s := sitemap.New()
s = s.SetMaxSitemaps(1000)
```
... or ...
```go
s := sitemap.New().SetMaxSitemaps(1000)
```
The limit is one of the call, not of a sitemap index: the sitemaps a `robots.txt` or a sitemap index lists count on all levels together.
- Every sitemap the call requests counts, whether or not the request succeeds.
- A sitemap that is listed more than once is requested, and counted, once. That goes for a URL that is spelt another way too, where the difference is none to the request: `HTTPS://Example.com:443/sitemap.xml#top` is the sitemap at `https://example.com/sitemap.xml`. The letter case of the protocol and of the host, the default port of the protocol and a fragment make no difference; the path and the query do, so `/Sitemap.xml` and `/sitemap.xml` are two sitemaps.
- The document passed to `Parse()` does not count.
Once the limit is reached, the sitemaps that are left are not fetched, and a `*ParseError` is recorded in `GetErrors()`, once for the call. It names the URL passed to `Parse()`:
```
parse "https://example.com/sitemap-index.xml": limit of 1000 sitemaps reached
```
The error wraps `ErrMaxSitemaps`, see [Limit errors](#limit-errors). Reaching the limit does not fail `Parse()`: the URLs of the sitemaps fetched until then are returned. With multi-threading off, the sitemaps are fetched in the order they are listed, so it is the first ones that are fetched. With multi-threading on, which ones are fetched depends on how fast the server answers. Nothing is recorded if no sitemap had to be left out.
The same limit is on the sitemaps [`GetSitemaps()`](#getsitemaps) returns, which are the ones the documents list, whether or not they are fetched. No more of them are kept than the limit allows, and the same error is recorded if one had to be left out. The two are counted apart, so a call that fetches fewer sitemaps than it finds, as one with [follow rules](#follow-rules) does, can reach the limit by the sitemaps it finds: with a limit of 1,000, a call for a sitemap index of 1,200 entries records the error whichever of the sitemaps are followed.
Negative values are rejected and an error is recorded in `GetErrors()`. The limit of a call is the one set when the call starts.
See [`examples/maxsitemaps`](examples/maxsitemaps/main.go) for a runnable example.
#### Max URLs
The URLs of a call are kept in memory until the next call, and how many there are is up to the server as well. To limit the number of URLs a `Parse()` / `ParseContext()` call collects, use the `SetMaxURLs()` function.
The value is an `int`:
- `0`: no limit.
- a positive value: the call collects at most that many URLs. The default is `10000000`.
```go
s := sitemap.New()
s = s.SetMaxURLs(100000)
```
... or ...
```go
s := sitemap.New().SetMaxURLs(100000)
```
The limit is one of the call as well: the URLs of all the sitemaps count together. What counts are the URLs that `GetURLs()` returns: an entry that is not valid does not count, and neither does a URL that the patterns set with `SetRules()` leave out.
Once the limit is reached, the call stops collecting:
- what is left of the document that reached the limit is left out unread, so it adds no errors either;
- the sitemaps that are left are not fetched, and a sitemap that was under way is not parsed;
- a `*ParseError` is recorded in `GetErrors()`, once for the call. It names the URL passed to `Parse()`:
```
parse "https://example.com/sitemap-index.xml": limit of 100000 URLs reached
```
The error wraps `ErrMaxURLs`, see [Limit errors](#limit-errors). Reaching the limit does not fail `Parse()`: the URLs collected until then are returned. With multi-threading off, they are the first ones in the order the sitemaps list them. With multi-threading on, which ones they are depends on how fast the server answers. Nothing is recorded if nothing had to be left out.
The default is a last resort rather than a tight bound: ten million URLs take up gigabytes of memory. Set a lower limit when the documents are not trusted, see [SECURITY.md](SECURITY.md).
Negative values are rejected and an error is recorded in `GetErrors()`. The limit of a call is the one set when the call starts.
See [`examples/maxurls`](examples/maxurls/main.go) for a runnable example.
#### Max validation errors
What is wrong with an entry is recorded in `GetErrors()` as a `*ValidationError`: a location that is not valid, a value that cannot be parsed, in strict mode a value the protocol does not allow. An entry that is not valid does not count towards `SetMaxURLs()`, so how many of these errors a call records is up to the server as well: a single document of 50 MB can hold some three million entries without a location. To limit the number of validation errors a `Parse()` / `ParseContext()` call records, use the `SetMaxValidationErrors()` function.
The value is an `int`:
- `0`: no limit.
- a positive value: the call records at most that many validation errors. The default is `100000`.
```go
s := sitemap.New()
s = s.SetMaxValidationErrors(1000)
```
... or ...
```go
s := sitemap.New().SetMaxValidationErrors(1000)
```
The limit is one of the call as well: the validation errors of all the documents count together. What counts are the `*ValidationError` values `GetErrors()` returns, so an entry with two values that are not valid counts twice.
Once the limit is reached, the validation errors that follow are not recorded, and a `*ParseError` is recorded in `GetErrors()`, once for the call. It names the URL passed to `Parse()`:
```
parse "https://example.com/sitemap-index.xml": limit of 1000 validation errors reached
```
The error wraps `ErrMaxValidationErrors`, see [Limit errors](#limit-errors). Nothing is recorded if no error had to be left out.
Frequently Asked Questions
What is go-sitemap-parser?
go-sitemap-parser is a Text Processing library for the Go programming language. Go language library for parsing Sitemaps
How do I install go-sitemap-parser?
Install go-sitemap-parser with the Go module system using `go get aafeher/go-sitemap-parser`. Check the repository for the current installation instructions.
What category does go-sitemap-parser belong to?
go-sitemap-parser is listed under Text Processing, specifically Scrapers.