SA WEB CENSUS — Palo, by Dirišwago
source: https://palo.dirishwago.co.za
computed: 2026-10-17 14:22 UTC
licence: the data is CC BY 4.0; credit "Palo, Dirišwago (palo.dirishwago.co.za)"

ABOUT
A census of websites under .za and the links between them, as recorded by
Palo's crawlers. The figures below are counts and graph measures computed
from that record; the sections that follow describe how it was made and
what it does and does not observe.

METHODOLOGY
discovery: the Common Crawl index, the Wayback Machine, DNS lookups of generated
  names, and links found while crawling; until October 2026 also certificate
  transparency logs (crt.sh), through the earlier worker's list (see SOURCES)
crawling: each site's homepage, following up to ten redirects; robots.txt is
  respected (RFC 9309); active sites are also crawled up to 20 pages deep, one
  page per second; sites whose links appear only once JavaScript runs, and dead
  sites, are re-checked in a headless browser
language: as the site declares it (HTML lang attribute, then Open Graph locale,
  then the Content-Language header); never detected from the text
analysis: a directed graph of active and blocked sites, with igraph: PageRank
  (alpha 0.85), exact betweenness centrality (from every site; until 2026-10-16
  estimated from a sample of 5,000), Louvain communities (resolution 1.5),
  strongly connected components
reachability: the sites a site can reach by following links, itself excluded,
  estimated with HyperLogLog sketches (p = 8). A typical count is within about
  5%, and roughly one in ten is off by 10% or more. Estimates above 90% of the
  graph are recomputed exactly. Until 2026-10-16 the sites in a site's own
  strongly connected group were left out, understating reach for those sites.

LIMITATIONS
- The census holds the .za domains Palo has found so far and is unlikely to be complete.
- Since June 2026, server errors (5xx) have been recorded as unavailable instead of dead.
- Since 11 October 2026 a site is judged by where its redirects lead, and robots.txt
  follows RFC 9309. Dead counts before then likely include sites that only redirected,
  and blocked counts likely miss sites that opted out. Earlier timeline figures are
  not directly comparable with later ones.
- Most sites declare no language; LANGUAGE_BREAKDOWN covers only those that do.
- Only links between .za sites in the census are in the graph. Links from outside
  .za, from social media and from search engines are not observed.
- PageRank and betweenness are computed from link structure only, relative to the
  observed graph. Every link has the same weight, whatever its purpose.

DEFINITIONS
sites_found:            every .za domain in the census, whatever its status
active:                 the homepage answered with a page (2xx), directly or after redirects
unavailable:            the server answered with an error (5xx); often temporary, and re-checked
dead:                   a 4xx, no answer, a redirect loop, or a redirect that leads nowhere; likely gone
blocked:                a 429 (rate limited), a robot check instead of the site, or robots.txt
                        disallows Palo or answered with a server error; their pages are not crawled
links_found:            directed links between .za domains, from every crawled page;
                        includes links from sites since found dead or unavailable
links_per_active_site:  links_found divided by active (plotted on the timeline)
graph:                  active and blocked sites and the links between them
graph_mean_in_degree:   graph edges divided by graph nodes: the average number of
                        inbound links a site in the graph has
islands:                active sites with no links to or from any other site in the graph
unrecognised:           active sites that link to others in the graph but have no
                        inbound links from it
largest_scc:            the largest group of sites in which each can reach every other
                        by following links
communities:            Louvain communities with at least two sites
reachability_outliers:  sites whose estimated reach was recomputed exactly; a
                        property of the estimation method
pagerank_without_footer_only: PageRank on the graph with the links only ever seen in a
                        page footer taken out; links whose place is not yet recorded
                        are kept (see graph_edges_place_known)
registered_name:        the name registered in a site's zone: x.co.za for both x.co.za
                        and shop.x.co.za; a name directly under .za is its own
pagerank_between_registered_names: PageRank on the graph with the links between
                        sites under one registered name taken out (x.co.za and
                        shop.x.co.za, or two subdomains of x.co.za); each site is
                        still a site of its own
og_coverage:            active sites with at least one Open Graph tag
schema_coverage:        active sites that declare schema.org types (JSON-LD)
client_rendered:        active sites whose homepage, as served, is an empty shell that
                        JavaScript fills in, by fingerprint rules (a minimum; recorded
                        from 2026-10-15 as sites are crawled). Their links are seen only
                        once Palo's browser crawler has visited them

COUNTS
sites_found: 250119
active: 192065
unavailable: 1881
dead: 45353
blocked: 10820
links_found: 433357
links_per_active_site: 2.26
islands: 62503
unrecognised: 40224
og_coverage: 98159
client_rendered: 5733
schema_coverage: 81930

LANGUAGE_BREAKDOWN (active sites by declared language)
en: 153330
af: 600
zu: 14
xh: 3
nr: 1
ss: 1
st: 1
ts: 1
ve: 1

SOURCES (which sources list each site: listed by, not found by)
common_crawl: the Common Crawl crawls Palo has read (newest CC-MAIN-2026-39)
old_worker: the list kept by Palo's earlier worker until October 2026, from Common
  Crawl CC-MAIN-2025-18 or certificate transparency logs (crt.sh), not told apart
dns: names Palo generated that resolved in DNS
wayback: the Wayback Machine (recorded only from October 2026)
link: a crawled .za site links to it
active_sites: 192065
active_listed_by: common_crawl 135981, old_worker 107472, dns 4240, wayback 0, link 91335
active_listed_only_by: common_crawl 37146, old_worker 21096, dns 3194, wayback 0, link 19347
active_no_source_on_record: 1411
active_in_newest_common_crawl: 135981
active_islands_known_only_from_lists: 60920 of 62563
found_by (sites first seen since 2026-10-12): common_crawl 57921, link 10351, llm 974, lookup 116, old_worker 1899, redirect 471

LINKS LEAVING .za (measured 2026-10-16 02:45 UTC, over 191879 active sites crawled
  since 2026-10-12; each site-to-domain pair counts once, whatever the number of links)
outside_za_share: 70.8% of the links from active sites to other domains
  (667874 of 943843: 667874 to domains outside .za, 275969 to .za domains)
outside_za_share_body_only: 68.5% of the links seen in the page body
  (425507 of 621529: 425507 to domains outside .za, 196022 to .za domains)
foreign_hosts: 173852 in 152795 services (registered domains; renamed and
  shortened domains of one service joined, e.g. twitter.com and t.co with x.com)
top_services (service, sites linking, of which from the page body):
  facebook.com 82991 34107
  instagram.com 55564 20503
  linkedin.com 31068 13471
  x.com 26685 12529
  whatsapp.com 24860 17856
  youtube.com 23154 11464
  google.com 21643 15609
  goo.gl 8854 5125
  tiktok.com 7650 2524
  pinterest.com 7171 4366
  wordpress.org 6748 2592
  shopify.com 4017 829
  cookiedatabase.org 2736 2656
  apple.com 2713 1997
  nightsbridge.com 2364 1878
  wikipedia.org 1705 1654
  g.page 1627 1064
  microsoft.com 1043 998
  wa.link 993 626
  trustindex.io 989 849
by_top_level_domain (suffix, links, of which from the page body):
  com 475432 277180
  org 37215 28544
  me 21346 15229
  net 14427 11248
  gl 9153 5324
  ru 5505 5256
  co.uk 4990 4021
  io 4356 3332
  de 3527 3049
  info 3484 3205
  co 3155 2182
  be 2644 2331
  it 1870 1563
  vip 1825 1756
  page 1695 1108
  other 77250 60179

GRAPH
graph_nodes: 202885
graph_edges: 330518
graph_mean_in_degree: 1.63
strongly_connected_components: 169566
largest_scc: 21083
communities: 6428
density: 0.00000803
weakly_connected: no
reachability_outliers: 0
graph_edges_place_known: 255290
graph_edges_footer_only: 46840
graph_edges_within_registered_name: 46298

TOP_PAGERANK (domain, PageRank; sites with at least two inbound links)
domains.co.za 0.010606
sacoronavirus.co.za 0.005623
google.co.za 0.003940
inforegulator.org.za 0.003925
justice.gov.za 0.003130
webmail.konsoleh.co.za 0.002181
sars.gov.za 0.002012
nightsbridge.co.za 0.001952
popia.co.za 0.001787
bedworthparkguesthouse.co.za 0.001418
discovery.co.za 0.001378
ispa.org.za 0.001295
webbox.co.za 0.001262
payfast.co.za 0.001176
eservices.inforegulator.org.za 0.001160
quicket.co.za 0.000998
webpartner.co.za 0.000950
engineeredmedia.co.za 0.000911
xneelo.co.za 0.000865
infoisinfo.co.za 0.000838

TOP_PAGERANK_WITHOUT_FOOTER_ONLY_LINKS (domain, PageRank without footer-only links,
  rank with all links)
domains.co.za 0.011697 1
sacoronavirus.co.za 0.005014 3
inforegulator.org.za 0.004586 5
google.co.za 0.004165 4
popia.co.za 0.002489 10
nightsbridge.co.za 0.002432 9
justice.gov.za 0.002429 6
webmail.konsoleh.co.za 0.002211 7
eservices.inforegulator.org.za 0.001995 16
sars.gov.za 0.001903 8
bedworthparkguesthouse.co.za 0.001661 11
discovery.co.za 0.001501 12
quicket.co.za 0.001165 17
payfast.co.za 0.001101 15
maps.google.co.za 0.000983 25
infoisinfo.co.za 0.000981 21
parliament.gov.za 0.000808 28
iol.co.za 0.000807 27
mweb.co.za 0.000764 59
thepresidency.gov.za 0.000752 30

TOP_PAGERANK_BETWEEN_REGISTERED_NAMES (domain, PageRank without links within one
  registered name, rank with all links)
sacoronavirus.co.za 0.006679 3
justice.gov.za 0.003907 6
google.co.za 0.003822 4
inforegulator.org.za 0.003415 5
domains.co.za 0.003254 1
webmail.konsoleh.co.za 0.002416 7
popia.co.za 0.002228 10
nightsbridge.co.za 0.002194 9
sars.gov.za 0.001969 8
ispa.org.za 0.001757 13
discovery.co.za 0.001721 12
discoverygreen.co.za 0.001487 46
webbox.co.za 0.001409 14
payfast.co.za 0.001255 15
zarc.web.za 0.001241 22
quicket.co.za 0.001116 17
webpartner.co.za 0.001063 18
engineeredmedia.co.za 0.001014 19
xneelo.co.za 0.000997 20
domaindisputes.co.za 0.000965 42

TOP_BETWEENNESS (domain, betweenness)
justice.gov.za 0.011062
iol.co.za 0.007178
dirco.gov.za 0.004280
thepresidency.gov.za 0.004250
web-design.co.za 0.004150
inforegulator.org.za 0.004138
asata.co.za 0.003212
mandela.ac.za 0.002784
kragdag.co.za 0.002775
usaf.ac.za 0.002746