21  Interpreting Clusters

Keywords

Quarto, reproducible publishing, coding agents, Markdown, Python, EPUB, data analysis, k-means clustering

Cluster 4 is a number, and a number is hard to act on. “Rural, low-coverage, high-risk water” is a phrase an official can repeat and a council can argue over, and anyone can check it, provided a rule produced it rather than someone’s eye. The names carry most weight in the comparison the analysis was built for: infant mortality, exam scores and school dropout, which the clustering never saw, kind by kind. That comparison is where readers draw conclusions, so it is where honesty matters most, and it gets three views, because each hides something the others show.

In this chapter
  • Read the profile heatmap and describe a cluster by the features that set it apart.
  • Explain the rule that names each cluster, and check a name against the profile and against one member, Tunja.
  • Compare the outcomes between the kinds by year, in standard-deviation units and as distributions, and say why each gap is a description, not an effect.
  • Add the heatmap and the outcome lines to the clusters page.

Start fromyour water-quality folder at checkpoint 07, with k-means and the clusters page.

You buildoutcome rules, profiles and names in scripts/analysis.py, and the heatmap and outcome lines in clusters.html.

Checkpointcheckpoints/08-water-interpret/

21.1 What sets each cluster apart

A cluster’s profile is the mean of each feature over its rows, in z units (Section 19.1): +1 means one standard deviation above that year’s average across all kept rows (Figure 21.1). Read along a cluster’s line to describe it, and along a feature’s line to see which feature separates the clusters.

Heatmap of four clusters by nine features, each cell printed with its value. Cluster 4 is red for IRCA and poverty and dark blue for water and sewer coverage. Cluster 1 is blue for rural share and red for municipal performance. Cluster 3 is blue for altitude. Cluster 2 is red for altitude.
Figure 21.1: Cluster profiles in z units: standard deviations above (red) or below (blue) the year’s average. IRCA: utility-supplied. Altitude: derived SRTM point.

The highest-IRCA cluster, cluster 4, stands out on three things at once: piped-water coverage (−1.3 SD), poverty (+1.0 SD) and IRCA (+0.8 SD). Each cluster is such a combination, which is exactly what a single variable could not have found.

21.2 Name them by a rule

Names make clusters usable, and they also make them persuasive, so they need a rule that anyone can check. The analysis script’s name_rule() builds each name from three parts:

  • Setting, tested in this order: “Rural” if fewer than 40% of the cluster’s rows are urban zones; otherwise “Cities and large towns” if its rural share is 0.5 SD or more below average, “Lowland towns” if its altitude is 0.5 SD or more below, “Highland towns” if 0.5 SD or more above, and plain “Towns” if none of these holds.
  • Coverage: “low-coverage” if the cluster’s mean piped-water coverage is 0.5 SD or more below average.
  • Risk: the Resolución 2115 band of the cluster’s median utility-supplied IRCA, the middle value when its zone-years are sorted by IRCA.
Table 21.1: The four named clusters: zone-years and municipalities in each, and the median utility-supplied IRCA. A municipality counts in every cluster that holds one of its zone-years.
Kind Zone-years Municipalities Median IRCA (0–100)
1 Cities and large towns, no-risk water 2,271 284 1.6
2 Highland towns, no-risk water 3,750 558 4.0
3 Lowland towns, low-risk water 2,451 479 9.2
4 Rural, low-coverage, high-risk water 2,763 729 35.2

Because the rule reads the profile, the names never depend on which number k-means happened to give a cluster. Two words in them need care. “Low-coverage” is a relative position, not an absolute one: the rural cluster’s mean coverage is 30%, low but not zero, so “unserved” would overstate it. And the risk word describes the cluster’s median, not every zone-year in it: individual zone-years can sit in any band, and a median close to a band’s edge can flip the word.

TipIn practice

Check every name against the heatmap and the profile table before you use it, and keep the rule beside the name in anything you publish. A name is a summary someone else will quote without the table. The thresholds, 40% and half a standard deviation, are choices too: a different one could change a name.

Pause and think

Tunja’s rural water scored in the high-risk band in its early years (Section 18.3). Before reading on, which cluster do you expect its rural zone to fall in, and what could put it somewhere else?

Answer

The rural cluster is the obvious guess. But k-means places a zone-year by all nine features at once, and several of them, such as the rural share and the altitude, belong to the municipality, so they are the same for both of Tunja’s zones. If those sit far from the rural cluster’s centre, they can outweigh the water.

21.3 Where Tunja lands

The cluster that holds all 16 of Tunja’s zone-years, urban and rural, now has a name: cluster 1 (cities and large towns), “Cities and large towns, no-risk water”. For the urban zone, it fits. For the rural zone, the label hides most of what Section 18.3 found. Its water scored in the high-risk band every year from 2010 to 2014, before the analysis’s window opens in 2016, and inside the window its utility-supplied IRCA is never in the no-risk band, and is in the medium band in three of the eight years.

Two things are at work. The risk word describes the cluster’s median, not each member, as Section 21.2 warned. And k-means puts each zone-year in the cluster whose centre is nearest over all nine features at once (Section 20.1), adding up the squared differences feature by feature. Table 21.2 splits that sum in two. On its own four features, IRCA, piped-water coverage, sewer coverage and poverty, Tunja’s rural zone sits nearer the centre of cluster 4 (rural), and two of those, the coverages, are fills (Section 19.2). The other five, rural share, altitude, rainfall, municipal performance and water transfers, are the municipality’s, the same for both zones. On those it sits far nearer cluster 1’s centre, by enough to outweigh the first four. The other two centres are farther still.

Table 21.2: Where Tunja’s rural zone stands: its mean z-score over its zone-years beside two cluster centres, and its squared distance to each. Filled: imputed for every year.
Feature Tunja, rural Centre 1 Centre 4
IRCA −0.2 −0.5 +0.8
Piped-water coverage (filled) −1.1 +0.1 −1.3
Sewer coverage (filled) −1.4 +0.1 −1.3
Poverty +0.3 −0.8 +1.0
Squared distance, its own four 5.0 1.7
Rural share −2.1 −1.1 +0.3
Altitude +1.6 +0.1 +0.1
Rainfall −1.3 −0.2 +0.0
Municipal performance +1.5 +1.1 −0.2
Water transfers −2.3 −1.3 +0.3
Squared distance, the shared five 5.4 18.8
Squared distance, all nine 10.5 20.5

It has company: 33% of that cluster’s zone-years are rural zones. A label describes a kind; to say anything about one member, go back to its rows.

21.4 Outcomes by cluster

Now the outcomes, which the clustering never used (Figure 21.2). Each line is a cluster’s median within each year, so a shift that moves every kind in one year cannot pass for a difference between kinds. Infant mortality is compared only in 2017–2019 and 2021–2023: its method changes in 2017, and 2020 looks like another source (the data’s README, caveat 8).

Three small line charts by year, one line per cluster. Cluster 1, cities and large towns, has the lowest infant mortality and the highest Saber 11 scores. Cluster 2, highland towns, has the highest infant mortality. Clusters 3 and 4, lowland towns and rural, have the lowest Saber 11 scores. Dropout medians stay within a few percent and dip in 2020 for every cluster.
Figure 21.2: Median outcome by cluster and year: infant mortality in 2017–2019 and 2021–2023 only, Saber 11 in its published years. Descriptive only: not causal effects of water safety.

Gaps in the lines are years with no comparable value, drawn as gaps rather than joined across. Infant mortality and dropout are measured for a whole municipality but repeated on both zone rows, so they are counted once per municipality in each cluster and year: for these two the unit is the municipality-year, one municipality in one year, while the features and Saber 11 stay per zone-year.

Figure 21.3 puts the outcomes on the features’ scale: each outcome standardised within its year, a cell being the cluster’s mean z, with the mean itself in brackets.

Heatmap of the four named clusters by three outcomes: infant mortality, Saber 11 and school dropout. Each cell prints the cluster's mean outcome in z units and, in brackets, the mean itself. All the cells are pale compared with the features' heatmap.
Figure 21.3: Cluster mean outcome in z units, with the mean in brackets. Red is above average: better for Saber 11, worse for the other two. Descriptive only.

The paleness is the finding. No cluster’s mean outcome is more than 0.65 standard deviations from the average, against 1.32 for the features that formed them: the kinds differ less in outcomes than in what defines them. Part of this is by design: k-means chose the clusters to separate the features, so they should differ more on those than on anything else. Within that: the lowest recorded infant mortality is in cluster 1 (cities and large towns) and the highest in cluster 2 (highland towns); the best Saber 11 scores are in cluster 1 (cities and large towns); and the lowest dropout is in cluster 2 (highland towns).

Pause and think

Suppose unsafe water did harm infants’ health a great deal. Would you still expect the outcome heatmap to be paler than the features’?

Answer

Yes. k-means chose the clusters to separate the features, so they differ most on those by construction. The outcomes differ only as far as they go with the features, and small places’ noisy rates blur them further. Paleness alone is no evidence that water does not matter.

The highest recorded infant mortality is not in the cluster with the worst water, cluster 4 (rural). That is a good moment to slow down rather than to explain it away, because two things in the data push the same way. The first is small numbers. The rates of cluster 2 (highland towns) come from the smallest places of any kind, a median of 10,680 people per municipality-year, where a handful of deaths moves a rate a long way.

The second is what is missing. From 2017 the series looks like observed rates rather than modelled estimates, so a municipality where no infant died that year should show a rate of 0, yet no zero appears anywhere: a blank may be a recorded zero. 50% of this kind’s municipality-years are blank, against 32% of all clustered municipality-years. Leaving the blanks out drops what may be zeros, which raises small places’ averages most. That is selection on the outcome: whether a row is kept depends on the value being measured, so the rows kept are not typical. It may account for part of this kind’s place at the top. Small numbers, reporting and many other differences between the kinds all sit between a cluster’s water and its outcomes, and Chapter 22 takes that up directly.

21.5 Every variable, cluster by cluster

Means and medians hide overlap. Box plots show it: how each variable spreads within each cluster (Figure 21.4). Each box holds the middle half of a cluster’s values, with a solid line at the median and a dashed line at the mean, and its whiskers reach from the 5th to the 95th percentile, so they span the middle 90% of values. Features cover every zone-year 2016–2023; outcomes count in their years, municipal ones once per municipality.

Box plots of IRCA, utility-supplied (0–100), one per cluster. Medians: cluster 1 (Cities and large towns, no-risk water) 1.6; cluster 2 (Highland towns, no-risk water) 4.0; cluster 3 (Lowland towns, low-risk water) 9.2; cluster 4 (Rural, low-coverage, high-risk water) 35.2.
Figure 21.4: Every feature and outcome, cluster by cluster: pick a variable. Boxes hold the middle half of each cluster’s values; whiskers the middle 90%.

The boxes make the overlap visible. Clusters differ sharply on the features that built them and overlap on the outcomes, and a gap between two clusters’ boxes is a description of the data, not an effect of water safety.

So the water analysis’s second question has an answer, and it is descriptive: infant mortality, exam scores and dropout do differ between the kinds of place, exam scores most, but by no more than 0.65 standard deviations in any cluster’s mean, and for every outcome the middle 90% of each kind’s values overlaps every other kind’s.

21.6 Building the interpretation

The normal route copies the finished outcome rules and names, as in Section 20.5:

Ask your coding agent
Copy ../quarto-workshop/checkpoints/08-water-interpret/scripts/analysis.py over scripts/analysis.py (keep mine as scripts/analysis-before-08.py), run it with `uv run python scripts/analysis.py`, and show me the cluster names.
Expect it to run

copies · runs · writes outputs/outcomes_by_year.csv and outputs/cluster_profiles.csv

It should show four names, the ones in Table 21.1.

To build them instead (optional), two prompts: the outcome rules, then the profiles and names.

Ask your coding agent
In scripts/analysis.py, add outcome rules: infant_mortality only in 2017–2019 and 2021–2023; infant_mortality and dropout_rate are municipal, so count each municipality once per `cluster` and `year`; saber11_mean stays per zone-year. Save outputs/outcomes_by_year.csv (median, n). Run it.
Ask your coding agent
Write outputs/cluster_profiles.csv: rows, municipalities, mean raw and z features, outcome medians and n under the same rules. Name each cluster by a rule: setting, relative coverage ("low-coverage" if mean z_water_coverage ≤ −0.5), median IRCA band. Show me the names.

Whichever route you took, outputs/outcomes_by_year.csv should hold no infant-mortality rows for 2016 or 2020: the rules keep only the years whose method matches.

Both routes then add the interpretation to the clusters page:

Ask your coding agent
Add to clusters.qmd a heatmap of the cluster mean z-scores (features × clusters, blue to red) and one line chart per outcome from outputs/outcomes_by_year.csv, with gaps, not joined lines, in missing years. Use the cluster names and colours. Render it.
You do

reload clusters.html in your browser

The page should show the heatmap, four clusters by nine features, and one chart per outcome, with the infant-mortality lines broken at 2020 and Saber 11’s at its unpublished years, such as 2018 and 2021.

Pause and think

While it renders: describe one cluster using two features that are not both coverage, then check whether its name says the same.

Answer

One good answer, from Figure 21.1: cluster 4, “Rural, low-coverage, high-risk water”, stands out on piped-water coverage (−1.3 SD) and poverty (+1.0 SD). Its name carries the coverage but not the poverty, one more reason to publish the profile beside the name.

ImportantTry it

Take one route through the outcome rules and names, then extend the clusters page.

You’re done when the agent shows a name for every cluster, and clusters.html shows the heatmap and the outcomes by cluster and year.

Stuck for two minutes? Paste this into your agent. It backs up your work and brings the project to this point; then carry on from here.

Catch me up: follow ../quarto-workshop/checkpoints/08-water-interpret/README.md

21.7 Explorations in 3-D (optional)

The slide deck this book grew from has five more 3-D views of the outcomes the clustering never used, by cluster and year: dropout as terrain, clusters as hulls, the years stacked, clusters as orbits and dropout in Voronoi columns. They are exploratory pictures rather than evidence, and they run live in the published slides, starting with dropout as terrain.

Summary

  • A cluster’s profile, its mean z-score on each feature, describes it as a combination of traits that no single variable could have found.
  • A written rule names each cluster from its profile (setting, relative coverage, median IRCA band), so anyone can check the name.
  • “Low-coverage” is relative and the risk word describes a median: Tunja’s rural zone, never no-risk inside the window, sits in a cluster named for no-risk water, carried there by what it shares with its urban zone.
  • The outcomes, never used to form the clusters, differ between the kinds less than the features do, partly by design, and their distributions overlap.
  • The highest recorded infant mortality is not where the water is worst, and small places and blank rates make that ranking fragile.
  • All of it is description: a gap between kinds is not an effect of water safety.

Yet a description like this invites a causal reading. Chapter 22 tests the link inside each kind of place, where poverty, rurality and coverage are more alike than across the country.

Further reading

  • examples/water/scripts/analysis.py, name_rule(), profiles() and outcomes_by_year(), in the repository.
  • examples/water/data/README.md, caveat 8, on why infant mortality is compared only in 2017–2019 and 2021–2023.
  • matplotlib, boxplot: what the boxes, whiskers and means show (checked 2026-11-12).