How the counting changed

Between 11 and 13 October 2026 Palo changed how it counts. Figures from before 11 October and after it are not directly comparable, and the timeline shows a step on those days. This update says what changed and by how much.

Three things happened at once

The census had paused. From 12 September to 11 October 2026 nothing was crawled: the machine that runs most of Palo's crawling could no longer reach the census database. The figures published in that month did not change.

The rules changed. On 11 October new rules for deciding whether a site is alive went live. They are described below.

The census grew. On 12 October Palo read its first full crawl from Common Crawl, a public archive of the web, and queued 57,935 .za domains it had not seen before. Within a day almost all of them had been visited; 50,783 were alive.

The same sites, counted again

Of the 176,791 sites already in the census, 176,729 were checked again under the new rules by 13 October:

Some of this change is the web itself: a month passed without crawling, and some sites came back or went away in that time. The two cannot be fully separated now. The rule changes likely account for most of it, since they move sites in exactly these directions.

How a site's status is decided now

Redirects are judged by where they lead. Before, almost any redirect counted as dead, including a site sending visitors from http to https or to a page on the same site. Now the page the redirects end on decides: a working page means the site is active.

Sites that ask not to be crawled are counted as blocked. A site whose robots.txt file asks crawlers to stay away used to be skipped and left no trace in the census. It is now recorded as blocked, so opt-outs are counted.

Sites with certificate problems still count. A site with an expired or self-signed security certificate used to fail the visit. The census asks whether a site exists, and such a site does.

Robot checks count as blocked (from 13 October). Some sites show visitors a check, such as a Cloudflare challenge, before letting them in. Palo used to record such a site as dead when the check came with an error code, and as active when it did not, with the check page's text taken as the site's own. Now it is recorded as blocked: the site exists, but keeps crawlers out.

What else changed in the counting

• Domains imported from other sources were once added to the census as dead before anyone had looked at them. They are now queued and visited first.
• Keyword-guessed categories (ecommerce, government and so on) were removed. They were guesses; Palo now records only what sites declare about themselves.
• The "SA average" of links to a site now compares like with like. It was every link divided by the number of active sites (about 2.3); it is now the average a site in the link graph actually has (about 1.4).
• Sites that died kept their last rankings until 13 October, so some dead sites appeared in the published lists of most-linked sites. Rankings now include only sites in the current graph.

What is recorded now that was not before

None of these change a site's status; they add facts about it.

• What kind of page a site shows, when it matches a known placeholder: for sale, a registrar's or host's holding page, a server's default page, a suspended account, a maintenance notice. On 13 October (at 17:57 SAST), 6,098 active sites showed such a page. These counts are minimums: placeholders Palo does not recognise are not counted.
• Where on a page each link sits: in the body, a header, the navigation or the footer. About a quarter of links between .za sites appear only in a footer.
• Links to sites outside .za, and which services they go to.
• How and when a site was first found, for sites found from 12 October, and which sources list every site.

Reading the timeline

Rows from before 11 October were counted under the old rules: more dead sites, fewer active ones, almost none blocked. The jump between 11 and 13 October comes mostly from the new rules and the new domains, not from a change of that size in the .za web itself.

The figures in the essay The .za web, September 2026 come from the old rules.