# Palo bulk data

Palo, a census of the South African web by Dirišwago. https://palo.dirishwago.co.za/data.md
Data licence: CC BY 4.0; credit "Palo, Dirišwago (palo.dirishwago.co.za)".

Every site in the census and every link between them, as two gzipped CSV files (UTF-8, comma-separated, a header row, fields quoted where needed). They are written every night from one snapshot of the census, so the two agree with each other.

Snapshot: 17 October 2026, 09:00 SAST. Graph measures from the analysis of 17 October 2026, 08:24 SAST.

| File | Rows | Size | SHA-256 |
|---|---|---|---|
| [sites.csv.gz](https://palo.dirishwago.co.za/data/sites.csv.gz) | 249,245 | 23.9 MB | ba560abfd6976aa3a0ac43cad89b68739b453de1c743a6e5bd00d055683f91ec |
| [links.csv.gz](https://palo.dirishwago.co.za/data/links.csv.gz) | 427,555 | 3.5 MB | ef17e9e5d36824c140b22853d1cabc949891cd73194187e1aae62f036df8daf5 |

The same, as JSON: https://palo.dirishwago.co.za/data/manifest.json

## sites.csv.gz: one row per site

Every site in the census, whatever its status. Empty fields were not recorded or do not apply.

- `domain`: the site's name, a host name under .za
- `zone`: the second-level domain it sits under (co.za, gov.za, ac.za …), or "other" for a name registered directly under .za
- `status`: what happened when Palo last visited it: active, dead, blocked or unavailable (definitions: https://palo.dirishwago.co.za/census/status.md)
- `checked_at`: when Palo last visited it (UTC, ISO 8601)
- `first_seen`: when Palo first recorded it (UTC); empty when not recorded, as for most sites found before 12 October 2026
- `website`: the address its homepage answered at, after redirects
- `title`: the title the homepage declares, quoted as data; may contain line breaks
- `description`: the description the homepage declares, quoted as data; may contain line breaks
- `language`: the language the homepage declares (<html lang>, then og:locale, then Content-Language); never detected from text
- `schema_types`: the schema.org types the homepage declares, comma-separated
- `open_graph`: true when the homepage carries Open Graph metadata
- `homepage_bytes`: the size of the homepage Palo received
- `homepage_links`: the number of links on the homepage
- `homepage_kind`: set when the homepage is a recognised kind of non-site page: for_sale, holding, default_server, suspended, maintenance, bot_challenge, client_redirect or unconfigured; empty otherwise
- `sources`: the sources that list the site, semicolon-separated: common_crawl:<crawl> for each Common Crawl crawl Palo has read that lists it; link when a crawled .za site links to it; old_worker, the list kept by Palo's earlier worker until October 2026 (from Common Crawl CC-MAIN-2025-18 or certificate transparency logs, not told apart); dns, a name Palo generated that resolved in DNS; wayback, the Wayback Machine. Listed by, not necessarily found by
- `in_graph`: true when the site was in the analysed graph, which holds the sites active or blocked at the analysis; a site whose status has changed since keeps its graph measures until the next analysis. The graph columns below are empty when false
- `graph_links_in`: how many sites in the graph link to it (in-degree)
- `graph_links_out`: how many sites in the graph it links to (out-degree)
- `pagerank`: PageRank (damping 0.85); the values of all sites in the graph sum to 1
- `pagerank_rank`: rank by PageRank, 1 the highest; ties by name
- `pagerank_body`: PageRank without the links only ever seen in a page footer
- `pagerank_body_rank`: rank by pagerank_body
- `pagerank_names`: PageRank without the links between sites under one registered name, such as x.co.za and shop.x.co.za
- `pagerank_names_rank`: rank by pagerank_names
- `betweenness`: betweenness centrality, exact and normalised: how often the site lies on the shortest paths between other sites
- `betweenness_rank`: rank by betweenness
- `group_size`: the size of its strongly connected group: the sites that can all reach one another by following links (1 when it is alone)
- `community`: the number of its Louvain community, when the community has two or more sites; numbers change from one analysis to the next
- `community_size`: how many sites that community has
- `reach`: about how many other sites it can reach by following links (estimated; typically within 5%)

## links.csv.gz: one row per link

Every link recorded between .za sites in the census, including links to and from sites now dead, and the few links from a site to itself; this is the count the census publishes as its links. Links to sites outside .za are not in the file.

- `from_domain`: the site whose crawled pages carry the link
- `to_domain`: the .za site the link points to
- `seen_in`: where on the linking pages the link was seen, semicolon-separated: body, header, nav, footer; empty when not recorded

## About these figures

- Only links between .za sites in the census are recorded; links from outside .za, social media and search engines are not observed.
- Links added by scripts after a page loads may be missed, and pages Palo has not crawled are not seen.
- Census-wide figures, definitions and limitations: https://palo.dirishwago.co.za/api/summary
- What the census holds and how to use it: https://palo.dirishwago.co.za/llms.txt
- This page for people: https://palo.dirishwago.co.za/data