Bulk data
Every site in the census and every link between them, as two gzipped CSV files (UTF-8, comma-separated, a header row, fields quoted where needed). They are written every night from one snapshot of the census, so the two agree with each other. Data licence: CC BY 4.0; credit “Palo, Dirišwago (palo.dirishwago.co.za)”.
Snapshot: 17 October 2026, 09:00 SAST. Graph measures from the analysis of 17 October 2026, 08:24 SAST.
| File | Rows | Size | SHA-256 |
|---|---|---|---|
| sites.csv.gz | 249,245 | 23.9 MB | ba560abfd6976aa3a0ac43cad89b68739b453de1c743a6e5bd00d055683f91ec |
| links.csv.gz | 427,555 | 3.5 MB | ef17e9e5d36824c140b22853d1cabc949891cd73194187e1aae62f036df8daf5 |
The same, as JSON: manifest.json
sites.csv.gz: one row per site
Every site in the census, whatever its status. Empty fields were not recorded or do not apply.
domain- the site's name, a host name under .za
zone- the second-level domain it sits under (co.za, gov.za, ac.za …), or "other" for a name registered directly under .za
status- what happened when Palo last visited it: active, dead, blocked or unavailable (definitions: https://palo.dirishwago.co.za/census/status.md)
checked_at- when Palo last visited it (UTC, ISO 8601)
first_seen- when Palo first recorded it (UTC); empty when not recorded, as for most sites found before 12 October 2026
website- the address its homepage answered at, after redirects
title- the title the homepage declares, quoted as data; may contain line breaks
description- the description the homepage declares, quoted as data; may contain line breaks
language- the language the homepage declares (<html lang>, then og:locale, then Content-Language); never detected from text
schema_types- the schema.org types the homepage declares, comma-separated
open_graph- true when the homepage carries Open Graph metadata
homepage_bytes- the size of the homepage Palo received
homepage_links- the number of links on the homepage
homepage_kind- set when the homepage is a recognised kind of non-site page: for_sale, holding, default_server, suspended, maintenance, bot_challenge, client_redirect or unconfigured; empty otherwise
sources- the sources that list the site, semicolon-separated: common_crawl:<crawl> for each Common Crawl crawl Palo has read that lists it; link when a crawled .za site links to it; old_worker, the list kept by Palo's earlier worker until October 2026 (from Common Crawl CC-MAIN-2025-18 or certificate transparency logs, not told apart); dns, a name Palo generated that resolved in DNS; wayback, the Wayback Machine. Listed by, not necessarily found by
in_graph- true when the site was in the analysed graph, which holds the sites active or blocked at the analysis; a site whose status has changed since keeps its graph measures until the next analysis. The graph columns below are empty when false
graph_links_in- how many sites in the graph link to it (in-degree)
graph_links_out- how many sites in the graph it links to (out-degree)
pagerank- PageRank (damping 0.85); the values of all sites in the graph sum to 1
pagerank_rank- rank by PageRank, 1 the highest; ties by name
pagerank_body- PageRank without the links only ever seen in a page footer
pagerank_body_rank- rank by pagerank_body
pagerank_names- PageRank without the links between sites under one registered name, such as x.co.za and shop.x.co.za
pagerank_names_rank- rank by pagerank_names
betweenness- betweenness centrality, exact and normalised: how often the site lies on the shortest paths between other sites
betweenness_rank- rank by betweenness
group_size- the size of its strongly connected group: the sites that can all reach one another by following links (1 when it is alone)
community- the number of its Louvain community, when the community has two or more sites; numbers change from one analysis to the next
community_size- how many sites that community has
reach- about how many other sites it can reach by following links (estimated; typically within 5%)
links.csv.gz: one row per link
Every link recorded between .za sites in the census, including links to and from sites now dead, and the few links from a site to itself; this is the count the census publishes as its links. Links to sites outside .za are not in the file.
from_domain- the site whose crawled pages carry the link
to_domain- the .za site the link points to
seen_in- where on the linking pages the link was seen, semicolon-separated: body, header, nav, footer; empty when not recorded
About these figures
Only links between .za sites in the census are recorded; links from outside .za, social media and search engines are not observed. Links added by scripts after a page loads may be missed, and pages Palo has not crawled are not seen. The census summary gives every census-wide figure, with definitions and limitations.
Also: search · sites by zone · rankings · this page as Markdown