Synthetic Musings

Who Does the Clearing?

Ravi Kalia — Thu, 17 Sep 2026 05:00:00 GMT

Who Does the Clearing?

A macrophage population clears apoptotic cells at some rate, and that rate is usually all an assay reports. The number is a sum over cells, and a sum has no memory of which cells produced it.

Watch a clearance assay for long enough and the same total uptake can arrive two ways. Experienced macrophages can take on more, so clearance concentrates in the cells that have already eaten. Or experienced macrophages can slow down while fresh cells absorb what they leave, so clearance spreads out. Both trajectories are consistent with continual efferocytosis as it is described, and both give the same population curve. Separating them needs a model of the per-cell event history and a design that assigns the history rather than observing it.

Everything here is simulated. No experimental data were used, and no result below is a claim about a real macrophage population — the point is what an analysis can and cannot recover when the generating process is known exactly.

1 Efferocytosis

Efferocytosis is the recognition, engulfment and disposal of an apoptotic cell by a phagocyte (Doran et al. 2020). It runs in stages that are worth keeping separate:

Find-me. Nucleotides and lipids released by the dying cell recruit the phagocyte (Ravichandran 2011).
Eat-me. Exposed phosphatidylserine, read directly or through a bridging molecule, marks the target for engulfment (Poon et al. 2014).
Internalisation. The target is drawn into a phagosome.
Processing. The phagolysosome degrades the cargo, and the phagocyte metabolises what it recovers.

These are not a strict queue. A macrophage can bind one target while still digesting another, so binding, internalisation and processing overlap in time.

Macrophages clear targets repeatedly rather than once, and what happens after the first meal is the part that matters here. Processing the first corpse changes the cell: mitochondrial fission supports continued uptake (Wang et al. 2017), arginine recovered from the cargo is metabolised into polyamines that promote further uptake (Yurdagul et al. 2020), and loss of the phagocyte’s UCP2 protein blocks continued clearance specifically (Park et al. 2011). Against that, a cell carrying undigested cargo has finite capacity.

So a cell’s own history could push its next uptake either way. Prior uptake could prime it, or saturate it. Both are described in the literature, and an assay that reports a population mean cannot tell which one is running.

Neighbours matter too, in two different ways that are easy to conflate. A neighbour that eats a target removes that target, which lowers the opportunity in the local neighbourhood. Separately, metabolites released by apoptotic cells act on surrounding tissue as signals in their own right (Medina et al. 2020). The first is depletion; the second is signalling; they are distinguishable only if opportunity is measured.

2 Same mean, different mechanism

When targets are limited, the population curve is set by supply, and the allocation underneath it is free.

Two rules draw on one fixed pool of targets. Under priming, cells with prior experience take a larger share; under satiety they take a smaller one and the naive cells absorb the remainder. The pool is what runs out, so both rules eat through the same supply on the same schedule.

Code

CP or Tucker

Ravi Kalia — Tue, 15 Sep 2026 05:00:00 GMT

CP or Tucker

parafac(X, rank=3) and tucker(X, rank=[3, 3, 3]) take the same array, hand back factor matrices, and differ in the call by one integer against three — so they look like two settings of one algorithm. They are not. CP returns components that, under a condition on the factors, are the only ones that fit the array; Tucker returns a basis per mode and a small core, fixed only up to a rotation that slides between them.

The gap shows up when someone reads the output. A CP component can be named and argued about. A Tucker factor column is a coordinate: rotate it, push the inverse rotation into the core, and the fit does not change. Naming a Tucker column picks one of infinitely many equally good answers.

Every worked form here is three-way. The algebra for general , with unfoldings, HOSVD, tensor train and the t-product, is in Tensor Factorizations and Tensor Inverses. What these compressions cost on real video and audio is in Uses of Tensor Factorizations.

1 The two arrays

Two shapes recur, and they pull in opposite directions.

A ratings table with a time axis — user by movie by month — gets asked what the recurring patterns are. The answer is meant to be read out: this group of users, these films, these months.

A colour image is height by width by colour, and gets asked how few numbers can stand in for it. Nobody reads a picture’s factors.

A three-way array. Each axis is a mode, and no two modes are interchangeable.

2 Figure data

The volumes in the figures are synthetic and no decomposition is fitted to them.

The three CP components are outer products of Gaussian bumps on a grid, each centred somewhere else in it, so a component reads as one blob. They stand in for concepts recovered from a ratings cube, which is far too large and too sparse to draw.
The Tucker figure is a layout: a full cube, a small core, and three random heatmaps standing in for factor matrices. It is not a fit of the CP volumes.
The rotating cubes are labelled boxes, not data at all.
src/make_cubes.py writes all five images. The page has no executable cells.

Which decomposition answers which question is settled by the algebra, so a MovieLens-shaped table or a public-domain film clip would not change the answer. What the compression costs does need measured data, and is measured in Uses of Tensor Factorizations.

3 Notation

For a three-way :

Mode. One axis. Mode 1 has length , mode 2 has length , mode 3 has length .
Outer product. is the array with entries . An array that can be written this way is rank-1.
Mode- product. multiplies every mode- fiber of by , leaving the other modes alone.
Mode- unfolding. , the matrix whose columns are the mode- fibers.

4 CP

CP — CANDECOMP/PARAFAC — writes the array as a sum of rank-1 terms:

One integer sets the model. Each term carries one vector per mode, so a term is a triple: a weighting of users, a weighting of movies, a weighting of months.

Three rank-1 terms and the array they add up to. Each term is an outer product of three Gaussian bumps, so it occupies one compact region. Two of the three sit close enough to merge into one lump in the sum: separating them takes a decomposition, not a threshold.

4.1 Uniqueness

CP’s value is that the triples are pinned down. Write for the k-rank of the factor matrix — the largest such that every columns of are linearly independent. Kruskal’s condition is

When it holds, the rank- CP decomposition is unique up to permuting the terms and rescaling the three vectors inside a term against each other. No rotation is available. That is why a CP component survives being given a name: a second analyst fitting the same array recovers the same triples in a different order.

Matrix factorization has no such property. for any invertible , so PCA needs an extra rule — maximise variance, or rotate to a varimax criterion — before a component means anything. The third mode is what makes that rule unnecessary.

4.2 Storage

components on an array need numbers, with each scaled into one of its own three vectors rather than stored. At on that is 108 against the dense 1728.

5 Tucker

Tucker compresses each mode against its own budget:

is , is , is . The triple is the multilinear rank, and it is three separate dials: the ranks of the three unfoldings.

A array, a core, and the three factor matrices (), () and ().

CP is the special case where and is zero off its superdiagonal. That shared ancestry is why the two library calls look alike.

The dials move independently. A colour image has three colour slices and a good compression keeps all three, while the height and width modes give up most of their length.

Each picture mode gives up most of its length while the colour mode, three slices deep, keeps all three. The shrunk box is the core: the relabel to marks where its axes stop counting pixels and start counting components.

5.1 Rotational freedom

For any invertible ,

and the same holds in modes 2 and 3. The fit does not change. Requiring orthonormal columns, as the higher-order SVD does, narrows to an orthogonal matrix but does not remove it: the factors are a basis for a subspace, and the subspace is what Tucker identifies.

Truncated HOSVD is the cheap non-iterative way to get that basis: take the SVD of each unfolding and keep left singular vectors. It is not the best multilinear-rank- approximation, but for an -way array its error is within of the best one. That is close enough to start an iterative fit from.

5.2 Storage

numbers. At on that is , against the dense 1728.

Two parameter counts on one toy array settle nothing: 108 for CP at and 180 for Tucker at compare two models that fit the array to different accuracies. Storage separates the two only at fixed error, and at fixed error the winner depends on whether the array really is a short sum of rank-1 terms.

6 Choosing

The question decides:

“What are the parts, and what do they mean?” CP — but check where Kruskal’s condition actually bites before leaning on it. Generic factors have full k-rank, so for it reduces to and holds for every . It bites when exceeds a mode’s length, and when a fit returns near-collinear columns, whose k-rank is numerically lower than the exact arithmetic says.
“How few numbers can stand in for this?” Tucker. The per-mode budget is the feature — spend where the array has structure, and leave a short mode alone.
“Which modes carry the variation?” Tucker, read as multilinear rank. The three unfolding ranks answer directly, and the factors need no interpretation to do it.
“How many components are there?” Neither, not directly. Determining tensor rank is NP-hard, so is chosen by fitting a range and watching the residual and the stability of the factors across restarts.

A useful order when both are on the table: fit Tucker first, because it always has a best fit and gives the multilinear rank cheaply, then fit CP inside the compressed core if the components are going to be read.

7 Constraints

CP:

A best rank- approximation need not exist for . The infimum can be approached by factors that grow without bound and cancel — the ALS iterates diverge while the residual keeps falling. Rising factor norms are the tell.
ALS gets stuck on long plateaus where the residual barely moves for hundreds of iterations and then drops. Stopping on “no progress” stops early.
Kruskal’s condition is sufficient, not necessary. Failing it means uniqueness is not guaranteed, not that the fit is wrong.

Tucker:

The core is , exponential in the number of modes. At eight modes with the core alone is numbers. Tensor train exists to avoid that.
The factors are bases. A heatmap of is not a picture of concepts, and reading it as one is the mistake CP exists to prevent.

Uniqueness. Or. Compression. CP. Names. Parts. Tucker. Spends. Budget. Ask. First.

8 References

Kolda, T. G., and Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review 51(3). The survey both models are usually cited from.
Kruskal, J. B. (1977). Three-way arrays: rank and uniqueness of trilinear decompositions. Linear Algebra and its Applications 18(2). The k-rank condition.
De Lathauwer, L., De Moor, B., and Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications 21(4). HOSVD and the bound.
de Silva, V., and Lim, L.-H. (2008). Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications 30(3). Why a best rank- fit can fail to exist.
Hillar, C. J., and Lim, L.-H. (2013). Most tensor problems are NP-hard. Journal of the ACM 60(6). Tensor rank among them.
Tensor Factorizations and Tensor Inverses — the algebra for general , plus tensor train, t-SVD and four tensor inverses.
Uses of Tensor Factorizations — what these compressions cost on a film clip, a conv kernel and a dense layer.

Probabilistic PCA

Ravi Kalia — Sun, 13 Sep 2026 05:00:00 GMT

Probabilistic PCA

Probabilistic PCA estimates a noisy low-dimensional signal with a Gaussian model: its maximum-likelihood fit recovers PCA’s principal subspace, while its posterior reconstruction shrinks toward the mean.

A classical PCA projection gives us coordinates and a reconstruction error. If we also want uncertainty about the unobserved signal, we need assumptions about how the measurements were generated; that is the role of the probability model.

Six Views of PCA derives the geometric optimization. We use that subspace result here and distinguish three objects: the fitted distribution of observations, the posterior latent coordinates, and the reconstructed signal.

1 Gaussian latent-variable model

Suppose a few unobserved quantities influence many measurements. Write one observation as

contains observed features; contains latent, or unobserved, coordinates, with .
maps latent coordinates to features. Its columns need not be orthonormal.
is the observation mean. We assume independent observations with common parameters.
is the noise variance in every feature direction. This equal-variance assumption is called isotropic noise; it refers to the chosen measurement coordinates.

Independent Gaussian contributions add their covariances, giving

The signal covariance has rank at most . Noise adds variance in all directions, so the observation distribution has full rank even though its signal lies in a smaller subspace.

This is the PPCA model introduced by Tipping and Bishop (1999). Fixing the latent covariance to leaves a rotation ambiguity: and describe the same observation distribution whenever .

2 Maximum-likelihood fit

2.1 Covariance convention

For complete observations, the fitted mean is . Let contain the centred observations as rows, and define

The denominator is , as required by this Gaussian likelihood. Eigenvalues computed using must be multiplied by before using the PPCA formulas; the eigenvectors are unchanged.

Apart from a constant, the log likelihood is

The determinant term penalizes an excessively diffuse distribution. The trace term penalizes observed variation that the model assigns too little variance to.

2.2 Principal subspace and residual variance

At the maximum, the model aligns its signal subspace with the leading sample eigenvectors. Let denote its covariance eigenvalues: the first can exceed , while the remaining all equal .

With that alignment, minimizing the negative likelihood reduces to

Differentiating gives for retained directions and an average for the discarded directions. Thus

Here and is any orthogonal matrix. The construction matches retained covariance eigenvalues and replaces discarded ones with their average.

If , the columns of span the classical principal subspace. The scales of those columns differ from unit PCA axes.
If , the corresponding loading vanishes, so the effective signal dimension is smaller than the requested .
If the discarded variance is zero, the solution lies at a singular zero-noise boundary; it is not a nonsingular Gaussian density.

The leading-subspace result follows from the likelihood optimization in Tipping and Bishop. It is agreement between optimizers under a model, not an identity between the likelihood and PCA’s reconstruction error for every .

3 Posterior coordinates and reconstruction

3.1 Conditional distribution

For a new measurement, we infer with fitted parameters held fixed. Let ; completing the square in the Gaussian density gives

The posterior expresses uncertainty about this observation’s latent coordinates. It does not include uncertainty in the fitted parameters , , or .

Choose to compare directly with classical scores . At the maximum-likelihood fit,

Latent coordinates use the model’s unit-variance prior scale. They are not the ordinary PCA scores .

3.2 Signal reconstruction

The posterior mean of the noise-free signal is

Classical PCA retains unchanged in the selected subspace. PPCA multiplies it by : when a retained direction has little variance beyond the fitted noise, the model attributes less of an extreme measurement to signal.

For a fixed principal subspace with positive retained eigenvalues, the reconstruction approaches the classical orthogonal projection as . The latent mean instead approaches , up to the latent rotation; its rescaling does not disappear.

3.3 Worked calculation

Consider a centred sample with eigenvalues under the convention and axes equal to the coordinate axes. This is an illustrative covariance, chosen to make the residual average explicit, not an empirical dataset.

For , we obtain and . For an observation , compare the three outputs:

Object	Value
Classical PCA score
Posterior latent mean
Classical reconstruction
Posterior signal mean
Posterior latent variance

Both reconstructions lie on the first coordinate axis. The posterior signal estimate is closer to the mean, while its latent coordinate is expressed on a different scale.

4 Reproducible sensor example

4.1 Data and fit

We generate 500 synthetic paired sensor readings from , , , and independent sensor noise with standard deviation . The example represents two calibrated sensors responding to one shared quantity with equal noise variance.

There is no external collector: the code generates both the signal and the noise. We ask whether the fitted model recovers the shared direction and changes reconstruction as derived; a real application would need to check the equal-noise assumption before trusting its uncertainty estimates.

rng = np.random.default_rng(19)
n, d, k = 500, 2, 1
W_true = np.array([[2.2], [0.9]])
mu_true = np.array([1.0, -1.0])
z_true = rng.normal(size=(n, k))
observations = mu_true + z_true @ W_true.T + rng.normal(scale=0.8, size=(n, d))
mu = observations.mean(axis=0)
X = observations - mu
_, singular, Vt = np.linalg.svd(X, full_matrices=False)
eigenvalues = singular**2 / n
V = Vt[:k].T
# Fix a plotting convention without consulting labels or the true latent z.
V *= np.where(V[np.argmax(abs(V), axis=0), np.arange(k)] < 0, -1, 1)
noise = eigenvalues[k:].mean()
W = V * np.sqrt(eigenvalues[:k] - noise)
covariance = W @ W.T + noise * np.eye(d)
print("sample covariance eigenvalues:", eigenvalues)
print(f"fitted noise variance: {noise:.4f}; generating value: {0.8**2:.4f}")
alignment = abs(float(V[:, 0] @ W_true[:, 0])) / np.linalg.norm(W_true)
print(f"absolute cosine with generating direction: {alignment:.4f}")
assert np.allclose(np.linalg.eigvalsh(covariance)[::-1], eigenvalues)

sample covariance eigenvalues: [6.0691 0.6283]
fitted noise variance: 0.6283; generating value: 0.6400
absolute cosine with generating direction: 1.0000

The fitted noise variance is 0.6283, compared with the generating value 0.6400. With and , only one eigenvalue is discarded, so this PPCA model can reproduce both eigenvalues of any positive-definite sample covariance.

Covariance agreement here checks the implementation; it is not a goodness-of-fit test for Gaussianity or a proof about the true sensor noise.

4.2 Reconstruction check

We compute both estimates for every observation and check the shrinkage equation. A fixed probe, three units along the fitted first axis and one unit perpendicular to it, makes the two reconstructions easy to distinguish in the figure.

Code

Six Views of PCA

Ravi Kalia — Sun, 13 Sep 2026 05:00:00 GMT

Six Views of PCA

PCA’s maximum-variance and minimum-error definitions choose the same subspace; the projection geometry explains why, and the algebra connects both to truncated SVD.

When we replace a measurement table with fewer coordinates, we want to know what those coordinates preserve. The six formulations answer that question for a fixed matrix; centring and scaling determine which matrix we have asked them to summarize.

Eigenvectors and eigenvalues and SVD geometry provide the linear-algebra background.

1 Projection geometry

We can compare a line that keeps the most spread with a line that leaves the shortest perpendicular residuals. Both choose the direction called the first principal component, or PC1.

The synthetic cloud used here: 200 draws of a scalar latent , mapped to and perturbed by independent noise in both coordinates, then column-centred. It stands in for any measurement pair where one underlying quantity drives both readings and the instrument adds isotropic noise — two sensors on one process, two assays of one sample. Two dimensions let us draw the residuals. The example checks an algebraic identity; it does not validate a sensor model on real measurements.

Code

GAPDH as a NET suppressor

Ravi Kalia — Thu, 10 Sep 2026 05:00:00 GMT

GAPDH as a NET suppressor

A labelled metabolome with three COVID groups will classify. That is the easy trick. It does not name the enzyme that lets neutrophils throw nets of DNA into a lung that is already failing.

Li et al. (2023) asked that enzyme question. They measured metabolites in neutrophils from 75 people — severe COVID-19, mild COVID-19, or no infection — then they blocked one glycolytic enzyme in healthy cells and watched nets form. The public spreadsheet is the first experiment. The mechanism is the second. Neutrophil Metabolomes as a Matrix is the layout of the spreadsheet: Mild does not sit on the Control–Severe line. This review is the follow-up: what GAPDH is doing, why pH — not extra NADPH — is the trigger, and which array shape those inhibitor tables would need before a tensor factorization is the right tool.

The deposited cohort is Metabolomics Workbench ST002477: 75 people, one snapshot each, 285 unique LC-MS peak-area features, CC BY 4.0. The numbers are chromatogram peak areas, not fluxes and not NET counts. The inhibitor tables — live-cell movies, pH dyes, C tracing — were not deposited as a cube.

Neutrophils throw DNA nets and do not come back

Most white cells in blood are neutrophils. They live hours to a couple of days. Their job is to find bacteria and fungi and kill them in one of three ways: swallow them, dump bleach-like oxidants, or throw out a net of their own DNA, studded with enzymes, that traps microbes outside the cell. Brinkmann et al. (2004) showed those nets kill bacteria. The same structures turn up in lungs that are already failing in severe COVID-19.

Throwing a net is not a reversible state. The nuclear envelope breaks, the chromatin unpacks, and DNA leaves the cell. The cell is gone. Immunologists call that NETosis. A short-lived cell in low-oxygen tissue still has to make ATP — energy — and NADPH, the reducing power for the oxidant burst. It also has to keep the net programme off until something warrants the cost. Metabolism is not only fuel. It is the switch.

What still needs a name is the switch itself. The paper’s claim is that one enzyme in glycolysis is holding it.

GAPDH is a valve in glycolysis, not a housekeeping stamp

The enzyme that holds the switch is GAPDH. Cells burn glucose in a chain of enzymes called glycolysis. About halfway down that chain sits glyceraldehyde-3-phosphate dehydrogenase. It takes a three-carbon sugar, oxidises it, and in the same step reduces to NADH and releases a proton, . Textbooks call it housekeeping because every cell has it. In a neutrophil it behaves more like a valve. As long as sugar is flowing through GAPDH, the cell stays on the glycolytic path and keeps making acid. Block the valve and two things happen at once. Unprocessed sugar phosphates pile up upstream. Proton production falls, so the inside of the cell becomes more alkaline.

A bottleneck, in plain terms

Think of a pipe with a narrow stretch. Slow that stretch and liquid backs up on one side and runs dry on the other. GAPDH is that stretch. Li et al. treat it as a rate-limiting node whose flow keeps nets off. They do not treat it as one more loading on a severity axis.

A useful diagnostic of a block is a ratio. DHAP (dihydroxyacetone phosphate) sits just upstream of GAPDH. BPG (1,3-bisphosphoglycerate) sits just downstream. If GAPDH is slow, DHAP rises relative to BPG. Li et al. saw that DHAP:BPG ratio elevated in neutrophils from people with severe COVID-19. That is consistent with reduced GAPDH activity. It is not a proof that reactive oxygen species oxidised the enzyme in those patients. A snapshot ratio is a clue. It is not a knockout.

So they did the knockout the only way primary human neutrophils allow: they added a drug.

Blocking GAPDH is enough to make a net

Heptelidic acid — also sold as koningic acid; one molecule, two names — is a covalent GAPDH inhibitor made by the fungus Trichoderma koningii. Iodoacetate is a second, chemically different inhibitor. Either one, added to neutrophils taken from healthy donors, is enough to make NETs.

Watch the traffic after the valve closes. Upper-glycolysis intermediates — DHAP, fructose-1,6-bisphosphate (FBP), fructose-6-phosphate (F6P) — rise. Lower-glycolysis products — pyruvate, BPG — fall. The /NADH ratio rises, as expected once GAPDH stops reducing .

Glucose-6-phosphate can still peel off into the pentose-phosphate pathway, a shunt that makes NADPH. That shunt gets busier after GAPDH is blocked. It is easy to guess that extra NADPH then drives the oxidant burst, and the oxidants make the net. That is how PMA, a classic lab trigger, makes nets. It is not how heptelidic acid does it.

Neutrophils make a burst of superoxide using NADPH oxidase (NOX). PMA needs that burst. Blocking G6PD — the first enzyme of the oxidative shunt — or blocking NOX with diphenyleneiodonium (DPI) stops PMA nets. The same blockers leave HA nets intact. DPI can even speed HA nets. HA also failed to raise mitochondrial ROS or collapse the GSH/GSSG ratio, the usual fingerprints of an oxidant crisis. So the HA path is not “more NADPH, more oxidants, more nets.”

DHAP that cannot go through GAPDH spontaneously becomes methylglyoxal, a sticky aldehyde. That molecule blunts the respiratory burst. HA-treated cells are worse at making oxidants, not better.

The GAPDH reaction itself produces . Stop the reaction and the cell makes less acid. Intracellular pH rises. Buffer the outside with HEPES, or stop the cell exporting leftover protons with cariporide — an inhibitor of the exchanger neutrophils use to set pH — and the pH rise is gone, and so are death and NET formation. The metabolic pile-up is not gone. DHAP and BPG stay where HA put them. pH is downstream of the block. The metabolome of an HA-treated cell still looks blocked even when the cell is saved.

Neutrophil elastase is a protease stored in granules. At higher pH it works better. Inhibitors of elastase (AZD9668, BAY-678, GW311616a) do not stop the membrane from becoming leaky. They stop DNA from leaving. The last mechanical step of the net is enzyme cutting, not a passive leak.

Severe COVID-19 serum is full of inflammatory cytokines. TNF- pushes glycolysis forward. HA still blocks at GAPDH in that setting. Inflammatory drive and the GAPDH block are separable.

The causal order is linear until elastase:

flowchart TD
  g6p[G6P] --> upper[Upper glycolysis]
  g6p --> ppp[PPP cycling]
  upper --> dhap[DHAP and FBP]
  dhap --> gapdh[GAPDH blocked]
  dhap --> ppp
  gapdh --> lower[Lower glycolysis falls]
  gapdh --> protons["H+ production falls"]
  protons --> ph[Intracellular pH rises]
  ph --> ne[Neutrophil elastase]
  ne --> dna[DNA extrusion]

pH sits downstream of the metabolic block. Rescuing pH does not restore DHAP or BPG. A metabolome of an HA-treated cell therefore still looks “blocked” even when the cell does not throw a net. NET yield is a separate assay, not a hidden column of the concentration table.

The pathway drawing is the same order as the graph: glycolysis along the chain, pH and elastase on the finish, PPP on the shunt and not on the pH arrow.

Code

Neutrophil Metabolomes as a Matrix

Ravi Kalia — Thu, 10 Sep 2026 05:00:00 GMT

Neutrophil Metabolomes as a Matrix

A classifier will separate these three COVID neutrophil groups. That is the easy trick. Clustering, which does not get the labels, does not recover the same partition.

Seventy-five people, 285 metabolites, three tags on rows. The usual next move is to average each tag and draw a line through those three points, then treat the leftover as a second disease direction. That geometry is always defined. It is not a model of the people. One Control profile already sits far from the Control mean. The clouds overlap. A line through the means never sees that.

Li et al. (2023) isolated neutrophils from 75 people, extracted two million cells each, and deposited LC-MS peak areas as Metabolomics Workbench ST002477 (CC BY 4.0). They wanted the cell that kills, not plasma. The numbers are relative ion intensities, not counts and not fluxes. The job that fits the file is whether the 75 rows form three blobs that match the labels.

1 Neutrophil metabolomes

Plasma metabolomes mix liver, muscle, and lunch. This table is one cell type: the neutrophil, the cell that does the killing.

Neutrophils eat microbes, dump oxidants, and throw DNA nets (NETs). That work runs on glycolysis and the pentose-phosphate pathway (Morrison, Watts, Sadiku, and Walmsley 2022). In severe COVID-19 these cells pile up in failing lungs. If their fuel is off, killing and damage move together. Li et al. saw amino-acid, redox, and central-carbon pools shift, and showed GAPDH can suppress NET formation. Peak area is not that flux. It is still the public readout of the cell.

Erythrose 4-phosphate sits on the pathway that powers the oxidant burst (Britt et al. 2022). Hypotaurine and glycine sit near redox handling. Those names only help after the 75 people have been treated as 75 people.

2 Source table

75 profiles: 19 Control, 30 Mild, 26 Severe. 287 named rows, 285 unique vectors. Chromatogram peak areas from a targeted Q Exactive method, median-normalised per sample. Nonnegative and continuous. Not integer counts.

metabolite	PMN_74 (C)	PMN_75 (C)	PMN_02 (M)	PMN_03 (M)	PMN_48 (S)	PMN_49 (S)
Hypotaurine	6.97e+08	3.66e+08	5.58e+08	1.39e+08	9.88e+08	6.45e+08
Glycine	1.07e+08	8.54e+07	7.55e+07	8.73e+07	9.30e+07	1.09e+08
Erythrose 4-phosphate	1.06e+06	5.94e+06	2.67e+06	3.12e+06	3.40e+04	1.85e+05
1,3/ 2,3-Bisphosphoglycerate	6.71e+04	1.11e+05	1.73e+06	3.79e+07	2.28e+05	5.03e+06
Creatine	9.06e+09	6.27e+09	9.80e+09	7.02e+09	8.04e+09	1.49e+10
Uridine monophosphate	4.36e+07	1.39e+06	1.54e+06	1.98e+06	1.64e+06	1.68e+06

Relative peak areas, two samples per group. The range is normal for ion intensity. It is not a count table.

3 Matrix layout

Each person is one row. Each metabolite is one column. . Control, Mild, and Severe partition those rows. They do not add a third index.

A tensor mode can vary on its own. Here each person sits in one group. sample × metabolite × group leaves two slices empty for every person. CP on that cube would spend rank explaining holes the design made.

flowchart LR
  subgraph nested [Nested design]
    samples["75 samples"] --> matrixX["X: 75 x 285"]
    samples --> partition["19 Control / 30 Mild / 26 Severe"]
  end
  subgraph crossed [Crossed design]
    donor["donor"] --> cube["donor x condition x metabolite"]
    condition["condition"] --> cube
    metabolite["metabolite"] --> cube
  end

Group would be a third mode if the same people were measured in all three states. This cohort is one snapshot per person. Raw chromatograms could still be sample × retention time × m/z. Those files are not in the public table.

4 Clustering

The labels are a hypothesis about density: three blobs, 75 points. Clustering tests that without being handed the tags.

is borrowed from the label count. It is not chosen by silhouette, BIC, or a gap statistic. Those criteria need a held-out split or a stable bootstrap. 75 rows, 285 columns, and groups of 19 / 30 / 26 do not give one. Hyperparameter Tuning in Clustering is the case where a validation split exists. This file is not that case.

Three algorithms, scikit-learn defaults, same log1p/IQR matrix :

k-means. Three spherical averages. This is averaging, without using the COVID tags.
Ward agglomerative. Merge by increase in within-cluster sum of squares.
Diagonal Gaussian mixture. One variance per metabolite. A full covariance per component cannot be fit: each would want parameters on 75 rows.

Agreement with the COVID labels is the adjusted Rand index (ARI). Chance is 0. A match is 1.

Code

A NumPy-like Tensor in Pure Python

Ravi Kalia — Thu, 10 Sep 2026 05:00:00 GMT

A NumPy-like Tensor in Pure Python

Nested Python lists already hold a grid. They do not carry a shape we can reshape. A tiny class that stores a flat list and a shape tuple gives NumPy’s layout methods with no NumPy.

We already write nested lists and call len for the first axis. The other axes are nested loops. NumPy’s .shape is that layout made explicit. shape, ndim, and size are attributes, not calls.

1 Storage

This post uses a synthetic array of range(24).

Provenance:

The 24 integers are generated, not measured.
The array stands in for a small 3-way table (batch × row × column).
Size 24 was chosen so each entry can keep one colour in the figures.
A measured table of that shape is larger; the layout questions are the same.

The constructor flattens nested lists and records a shape. It rejects a shape whose product is not len(data).

from math import prod


def _flatten_shape(data):
    if not isinstance(data, list):
        return [data], ()
    if not data:
        return [], (0,)
    parts = [_flatten_shape(item) for item in data]
    shape = parts[0][1]
    if any(s != shape for _, s in parts):
        raise ValueError("ragged nested list")
    flat = [v for f, _ in parts for v in f]
    return flat, (len(data),) + shape


def _unravel(index, shape):
    out = []
    for size in reversed(shape):
        out.append(index % size)
        index //= size
    return tuple(reversed(out))


def _ravel(multi, shape):
    index = 0
    for i, n in zip(multi, shape):
        index = index * n + i
    return index


def _pad_key(key, ndim):
    if not isinstance(key, tuple):
        key = (key,)
    if key.count(Ellipsis) > 1:
        raise IndexError("at most one ellipsis")
    if Ellipsis in key:
        loc = key.index(Ellipsis)
        n_fill = ndim - (len(key) - 1)
        if n_fill < 0:
            raise IndexError("too many indices")
        key = key[:loc] + (slice(None),) * n_fill + key[loc + 1 :]
    if len(key) > ndim:
        raise IndexError("too many indices")
    if len(key) < ndim:
        key = key + (slice(None),) * (ndim - len(key))
    return key


def _axis_int(i, n):
    if isinstance(i, bool) or not isinstance(i, int):
        raise TypeError("integer index expected")
    if i < 0:
        i += n
    if i < 0 or i >= n:
        raise IndexError("index out of bounds")
    return i


class Tensor:
    def __init__(self, data, shape=None):
        if shape is None:
            data, shape = _flatten_shape(data)
        data = list(data)
        shape = tuple(shape)
        if prod(shape) != len(data):
            raise ValueError("shape does not match size")
        self._data = data
        self._shape = shape

2 Counts

Four numbers describe the same layout:

shape — length of each axis: (2, 3, 4).
ndim — how many axes: 3.
size — product of the shape: 24.
len — first axis only: 2, not 24.

    @property
    def shape(self):
        return self._shape

    @property
    def ndim(self):
        return len(self._shape)

    @property
    def size(self):
        return prod(self._shape)

    def __len__(self):
        return self._shape[0]

3 Reshape

reshape returns a new Tensor with the same _data and a new shape. The product must equal .size. The list is copied; this is not a NumPy view.

    def reshape(self, *shape):
        if prod(shape) != self.size:
            raise ValueError("size mismatch")
        return Tensor(self._data, shape)

Each colour is one entry. The strip is storage order.

Figure 1: Layout of a tensor after reshape to a column, a row, and a . The storage strip stays 0 through 23.

The column, the row, and the keep that order.

4 Transpose

Default transpose reverses axes. becomes . Each multi-index is permuted and written into a new flat list.

    def transpose(self, *axes):
        if not axes:
            axes = tuple(range(self.ndim - 1, -1, -1))
        new_shape = tuple(self._shape[a] for a in axes)
        out = [None] * self.size
        for i, value in enumerate(self._data):
            old = _unravel(i, self._shape)
            new = tuple(old[a] for a in axes)
            out[_ravel(new, new_shape)] = value
        return Tensor(out, new_shape)

Figure 2: The same tensor after transpose. Layout is ; the storage strip is no longer 0 through 23.

The box matches the last reshape. The strip does not.

5 Indexing

__getitem__ reads values. Three forms. Every result is a copy.

Simple — an integer. That axis drops. t[0] has shape (3, 4). t[0, 2, 3] is 11.
Slicing — a slice. That axis stays. t[:, 1] is the middle row of both batches, shape (2, 4). t[0, :2, 1:3] has shape (2, 2).
Fancy — a list of integers gathers those positions. t[[0, 1]] is both batches. Two lists pick pairs, not a grid: t[[0, 1], [2, 0], 0] is [8, 12]. The block is a slice: t[:2, :2, 0].

    def __getitem__(self, key):
        key = _pad_key(key, self.ndim)
        kinds = []
        coords = []
        for k, n in zip(key, self._shape):
            if isinstance(k, list):
                coords.append([_axis_int(i, n) for i in k])
                kinds.append("fancy")
            elif isinstance(k, slice):
                coords.append(list(range(*k.indices(n))))
                kinds.append("slice")
            elif isinstance(k, int) and not isinstance(k, bool):
                coords.append([_axis_int(k, n)])
                kinds.append("int")
            else:
                raise TypeError("index must be an int, a slice, or a list of ints")

        fancy = [i for i, kind in enumerate(kinds) if kind == "fancy"]
        if fancy:
            length = len(coords[fancy[0]])
            if any(len(coords[i]) != length for i in fancy):
                raise IndexError("fancy lists must have the same length")
            if max(fancy) - min(fancy) + 1 != len(fancy):
                raise IndexError("fancy axes must be adjacent")

        out_shape = []
        ax = 0
        while ax < self.ndim:
            if fancy and ax == min(fancy):
                out_shape.append(length)
                ax = max(fancy) + 1
                continue
            if kinds[ax] == "slice":
                out_shape.append(len(coords[ax]))
            ax += 1
        out_shape = tuple(out_shape)

        data = []

        def fill(ax, p, multi):
            if ax == self.ndim:
                data.append(self._data[_ravel(tuple(multi), self._shape)])
                return
            kind = kinds[ax]
            if kind == "int":
                fill(ax + 1, p, multi + [coords[ax][0]])
            elif kind == "slice":
                for i in coords[ax]:
                    fill(ax + 1, p, multi + [i])
            elif ax == min(fancy):
                for q in range(length):
                    fill(ax + 1, q, multi + [coords[ax][q]])
            else:
                fill(ax + 1, p, multi + [coords[ax][p]])

        fill(0, None, [])
        if not out_shape:
            return data[0]
        return Tensor(data, out_shape)

6 Constraints

No broadcasting, no boolean mask, no newaxis, no views. Fancy axes must be adjacent. Enough to see why NumPy treats shape as data. NumPy to JAX is the real API.

7 Class

Markdown fences do not join. reshape, transpose, and __getitem__ are ordinary methods on Tensor:

from math import prod


def _flatten_shape(data):
    if not isinstance(data, list):
        return [data], ()
    if not data:
        return [], (0,)
    parts = [_flatten_shape(item) for item in data]
    shape = parts[0][1]
    if any(s != shape for _, s in parts):
        raise ValueError("ragged nested list")
    flat = [v for f, _ in parts for v in f]
    return flat, (len(data),) + shape


def _unravel(index, shape):
    out = []
    for size in reversed(shape):
        out.append(index % size)
        index //= size
    return tuple(reversed(out))


def _ravel(multi, shape):
    index = 0
    for i, n in zip(multi, shape):
        index = index * n + i
    return index


def _pad_key(key, ndim):
    if not isinstance(key, tuple):
        key = (key,)
    if key.count(Ellipsis) > 1:
        raise IndexError("at most one ellipsis")
    if Ellipsis in key:
        loc = key.index(Ellipsis)
        n_fill = ndim - (len(key) - 1)
        if n_fill < 0:
            raise IndexError("too many indices")
        key = key[:loc] + (slice(None),) * n_fill + key[loc + 1 :]
    if len(key) > ndim:
        raise IndexError("too many indices")
    if len(key) < ndim:
        key = key + (slice(None),) * (ndim - len(key))
    return key


def _axis_int(i, n):
    if isinstance(i, bool) or not isinstance(i, int):
        raise TypeError("integer index expected")
    if i < 0:
        i += n
    if i < 0 or i >= n:
        raise IndexError("index out of bounds")
    return i


class Tensor:
    def __init__(self, data, shape=None):
        if shape is None:
            data, shape = _flatten_shape(data)
        data = list(data)
        shape = tuple(shape)
        if prod(shape) != len(data):
            raise ValueError("shape does not match size")
        self._data = data
        self._shape = shape

    @property
    def shape(self):
        return self._shape

    @property
    def ndim(self):
        return len(self._shape)

    @property
    def size(self):
        return prod(self._shape)

    def __len__(self):
        return self._shape[0]

    def reshape(self, *shape):
        if prod(shape) != self.size:
            raise ValueError("size mismatch")
        return Tensor(self._data, shape)

    def transpose(self, *axes):
        if not axes:
            axes = tuple(range(self.ndim - 1, -1, -1))
        new_shape = tuple(self._shape[a] for a in axes)
        out = [None] * self.size
        for i, value in enumerate(self._data):
            old = _unravel(i, self._shape)
            new = tuple(old[a] for a in axes)
            out[_ravel(new, new_shape)] = value
        return Tensor(out, new_shape)

    def __getitem__(self, key):
        key = _pad_key(key, self.ndim)
        kinds = []
        coords = []
        for k, n in zip(key, self._shape):
            if isinstance(k, list):
                coords.append([_axis_int(i, n) for i in k])
                kinds.append("fancy")
            elif isinstance(k, slice):
                coords.append(list(range(*k.indices(n))))
                kinds.append("slice")
            elif isinstance(k, int) and not isinstance(k, bool):
                coords.append([_axis_int(k, n)])
                kinds.append("int")
            else:
                raise TypeError("index must be an int, a slice, or a list of ints")

        fancy = [i for i, kind in enumerate(kinds) if kind == "fancy"]
        if fancy:
            length = len(coords[fancy[0]])
            if any(len(coords[i]) != length for i in fancy):
                raise IndexError("fancy lists must have the same length")
            if max(fancy) - min(fancy) + 1 != len(fancy):
                raise IndexError("fancy axes must be adjacent")

        out_shape = []
        ax = 0
        while ax < self.ndim:
            if fancy and ax == min(fancy):
                out_shape.append(length)
                ax = max(fancy) + 1
                continue
            if kinds[ax] == "slice":
                out_shape.append(len(coords[ax]))
            ax += 1
        out_shape = tuple(out_shape)

        data = []

        def fill(ax, p, multi):
            if ax == self.ndim:
                data.append(self._data[_ravel(tuple(multi), self._shape)])
                return
            kind = kinds[ax]
            if kind == "int":
                fill(ax + 1, p, multi + [coords[ax][0]])
            elif kind == "slice":
                for i in coords[ax]:
                    fill(ax + 1, p, multi + [i])
            elif ax == min(fancy):
                for q in range(length):
                    fill(ax + 1, q, multi + [coords[ax][q]])
            else:
                fill(ax + 1, p, multi + [coords[ax][p]])

        fill(0, None, [])
        if not out_shape:
            return data[0]
        return Tensor(data, out_shape)


t = Tensor(list(range(24)), (2, 3, 4))
assert t.shape == (2, 3, 4)
assert len(t) == 2
assert t.reshape(4, 3, 2)._data == t._data
assert t.transpose()._data != t._data
assert t[0, 2, 3] == 11
assert t[:, 1].shape == (2, 4)
assert t[[0, 1], [2, 0], 0]._data == [8, 12]
assert t[:2, :2, 0].shape == (2, 2)

Storage. Order. Survives. Reshape. Transpose. Breaks. It. Indexing. Copies. Out.

8 References

NumPy, The N-dimensional array (ndarray).
NumPy, Indexing on ndarrays.
NumPy to JAX — this blog; shape, ndim, size, and reshape on the real array type.

What the Immune System Sees in an AAV Vector

Ravi Kalia — Fri, 04 Sep 2026 05:00:00 GMT

What the Immune System Sees in an AAV Vector

A gene therapy arrives as a vector. The body has no category for that. What reaches the bloodstream is a protein shell around 25 nanometres across with a single strand of DNA folded inside, assembled from the same genes a wild virus uses, and the receptors that meet it evolved to find exactly that shape. They cannot read intent.

Four arms of the immune system answer. They read three different things: the capsid shell, the DNA folded inside it, and the protein that DNA goes on to make. The shell is read twice — intact from the outside, and again in fragments from the inside, after a cell has taken a particle up and broken it apart. Which arm dominates decides whether enough genomes reach a nucleus, whether the patient’s liver enzymes climb in week three, and whether a second dose is ever possible.

Figure 1: The route the rest of this covers. Ribbon height is the surviving fraction of the dose, stepping down at each of the six stations; the end of it is the 7.6% that reaches a nucleus in a patient with no response running. Numbers on the ribbon match the cards.

1 Recombinant AAV

Adeno-associated virus is a small, non-enveloped parvovirus. It carries about 4.7 kilobases of single-stranded DNA and cannot copy itself without a helper virus. In the wild it causes no known disease, which is most of why it became the delivery vehicle for in-vivo gene therapy.

A recombinant AAV — rAAV — keeps the shell and throws away the contents:

The capsid is unchanged. Sixty copies of three related proteins, VP1, VP2 and VP3, packed into an icosahedron. This is the part the therapy needs, because the outside of the shell is what picks which tissue the particle enters.
The genome is replaced. Everything between the two inverted terminal repeats — the short hairpin sequences at each end that mark where packaging starts and stops — is swapped for a promoter and the therapeutic gene.
The particle is replication-defective. The genes for making more virus are gone, so a transduced cell produces the therapeutic protein and no progeny virus.

Two features of that construction matter to the immune system, and neither is a design choice anyone would make freely.

The shell is a viral protein whether or not it carries viral genes. Around 30 to 60 percent of adults carry antibodies against common serotypes such as AAV2 from ordinary childhood exposure, and those antibodies bind a therapeutic capsid exactly as well.
The cargo is unmethylated bacterial-style DNA. Plasmid DNA grown in E. coli is rich in CpG dinucleotides that the mammalian genome has largely lost, and the manufacturing process leaves that signature in the packaged genome.

Figure 2: A The shell a recombinant vector keeps. B The cargo it replaces: everything between the two ITRs is swapped for a promoter, the therapeutic gene and a polyA, and the CpG marks are left by the manufacturing route rather than chosen.

2 What the host can sense

The four responses look like four separate topics. They collapse to one question — which part of the particle is exposed, and when.

Innate sensing

Sensor TLR2 on the cell surface, TLR9 inside the endosome

Clock Minutes to hours

Antibodies

Sensor Neutralising antibody blocks entry, binding antibody opsonises

Clock Any time, and for years afterwards

CD8⁺ T cells

Sensor Capsid peptides displayed on MHC class I

Clock Four to eight weeks

Complement

Sensor C3 settling on the capsid surface, then the cascade

Clock Minutes

These four colours carry through every figure and through the widget’s four switches.

Exposed	Where	Sensor	Arm
Capsid, outside surface	Blood, cell surface	Pre-existing antibody, TLR2, complement C3	Humoral, innate, complement
Capsid ssDNA, unmethylated CpG	Endosome, during uncoating	TLR9	Innate
Capsid peptides	Cell surface, on MHC class I	CD8 T cell receptor	Cellular
Transgene product	Cell surface, on MHC class I	CD8 T cell receptor	Cellular

Reading the table down is reading the route. The shell is exposed first and is sensed first; the DNA is exposed only after the particle is inside a compartment where it can begin to open; the peptides appear only after a capsid has been broken up, which takes hours, and the T cells that recognise them take days to weeks to expand.

3 Timing

Minutes. Complement activation and TLR2 engagement. Both act on the intact particle in circulation, so both scale directly with how many particles are in circulation.
Hours. TLR9 sensing inside the endosome, and the type I interferon response it drives. The particle has to be taken up first.
Days. Innate signalling licenses antigen-presenting cells, which prime naive T cells.
Weeks. Capsid-specific CD8 T cells expand and kill transduced cells. This is the arm that takes away expression the patient already had.
Any time. Pre-existing neutralising antibodies act before the first two, and a post-dose antibody response closes the door on a second administration for years.

Figure 3: Bar position is when an arm acts, not how strong it is. The readout on the right of each lane is what a clinician measures on that clock.

4 The route

Six barriers stand between the syringe and a working episome (Figure 1). Each removes a fraction of the dose. The widget runs that attrition as an explicit chain: switch an arm on, and the particles it removes drop out at the station where that arm acts.

Things to try, and what each should show:

Turn everything off. About one genome in thirteen reaches a nucleus — 7.6% of the dose. Almost all of that loss is trafficking, not immunity. The endosome is the single most expensive step, with entry across the plasma membrane and the race against the proteasome close behind, and all three are expensive in a patient with no response to the vector at all.
Turn on antibodies with the titre at 1:1000. The figure falls to 0.15%, about one genome in 660. Almost the whole dose goes before a particle is ever inside a cell: opsonised capsids cleared at the sinusoid, then neutralising antibody blocking entry at the cell surface. The inflammatory lamps stay dark, because the particles never got far enough to be sensed from inside.
Turn on CD8 T cells alone. The nucleus count does not move at all. Expression at week 8 falls to 44 anyway. This arm does not stop delivery; it removes the cells that already received it.
Turn on complement and drag the dose from to vg/kg. The share reaching a nucleus falls from 5.4% to 4.0% while the genomes delivered rise roughly sevenfold, and the complement lamp goes from amber to red. Both halves of the dose problem are in one picture.

7.6%Reaches a nucleusNo response running, 10¹³ vg/kg

0.15%At a titre of 1:1000About one genome in 660

30–60%Of adults are seropositiveFrom ordinary childhood exposure

4–8 wkALT and AST riseWhen CD8⁺ T cells find the cells

4.1 Model

The numbers are synthetic and illustrative. They are not fitted to any trial, and no patient produced them.

The generating process is a product of survival fractions, one per station:

Each is a baseline trafficking fraction multiplied by a penalty for whichever arms act at station . The baselines are set so a seronegative patient with no active response lands at 7.6% of the dose reaching a nucleus, the right order for the few percent that reach target-cell nuclei in practice, and the dose and titre terms move in the direction the literature reports. Expression at week 8 multiplies the nucleus count by a T-cell survival term.

What it stands in for: a single intravenous, liver-directed dose in an adult. It is a teaching device for the ordering and coupling of the four arms — which one acts where, and what happens when two act together. It is not a predictor of any individual outcome.

5 Innate sensing

Minutes to hours

Toll-like receptors are pattern receptors: each one binds a molecular shape that is common in pathogens and rare in the host, and converts binding into a signalling cascade. Two of them see AAV, at different places, and they drive different cytokines.

Figure 4: A TLR2 reads the intact shell at the plasma membrane of a Kupffer cell. B TLR9 reads the released DNA inside the endosome of a plasmacytoid dendritic cell. Both route through MyD88 and split at the transcription factor.

5.1 TLR2

TLR2 sits on the cell surface, and on the endosomal membrane, and binds the structural capsid proteins of the intact particle. Human endothelial cells and Kupffer cells — the resident macrophages lining the liver’s blood channels — carry it, which puts it exactly where an intravenous dose goes first.

Binding recruits MyD88, an adaptor protein, which activates the transcription factor NF-B. NF-B turns on the pro-inflammatory cytokine genes: TNF-, IL-6 and IL-1. The consequence is a general inflammatory alarm, raised by the outside of the particle, with no reference to what the particle carries.

5.2 TLR9

TLR9 is inside the endosome, facing the compartment’s interior, and it binds unmethylated CpG dinucleotides in single-stranded DNA. That is a bacterial and viral signature: the mammalian genome depletes CpG and methylates most of what is left, so unmethylated CpG in an endosome means something foreign has begun to open.

The capsid must start uncoating for the DNA to become visible. TLR9 sensing therefore lags TLR2 sensing, and it reports on particles that got at least as far as an endosome.
Signalling goes through MyD88 again, but in plasmacytoid dendritic cells it branches to IRF7 and drives type I interferons, IFN- and IFN-, at high levels.
Interferon does two things at once. It suppresses transgene expression directly, and it matures the antigen-presenting cells that go on to prime the T-cell response. The innate arm is what licenses the adaptive one.

CpG content is a manufacturing variable, not a fixed property of the vector. Codon choices and regulatory sequences can be redesigned to strip CpG motifs out of the expression cassette without changing the protein, and CpG-depleted vectors are one of the levers the field pulls on this response.

6 Antibodies

Any time, and for years afterwards

B cells make antibodies against the capsid — often long before the therapy, from ordinary exposure to wild AAV. They act on the particle while it is still outside a cell, and they split into two functionally different classes.

Figure 5: A Neutralising antibody covers the receptor footprint, so an undamaged particle cannot dock. B Binding antibody attaches elsewhere and leaves the Fc tail out for a Kupffer cell to grip. Either way the particle never enters a hepatocyte, and complexes like these also start the classical complement pathway.

6.1 Neutralising antibodies

Neutralising antibodies bind the outer surface of the capsid at or near the sites the particle uses to attach. The blocking is steric — the antibody is physically in the way of the glycan the capsid binds and of AAVR, the receptor that carries it into the cell. The particle is intact and undamaged and simply cannot dock.

Neutralisation is close to all-or-nothing at the level of a single particle, so titre translates into dose loss steeply.
Titres are reported as the reciprocal dilution at which neutralisation is still measurable, so 1:1000 is a high titre and 1:5 is a low one. Many trials exclude patients above a threshold as low as 1:5, because a small amount of circulating antibody removes a large fraction of the dose.
This is the reason a second dose usually fails. The first dose raises a durable neutralising response against the capsid it was delivered in, and re-dosing with the same serotype years later meets a titre far above any enrolment threshold.

6.2 Binding antibodies

Binding antibodies attach to the capsid without covering a receptor site. The particle could still dock, but it does not get the chance: the antibody’s Fc tail is now sticking out, and Fc receptors on macrophages and Kupffer cells grip it.

The effect is opsonisation — coating a particle to mark it for eating. A capsid tagged this way is pulled into a phagocyte and degraded. The dose is cleared from circulation faster, and the liver’s resident macrophages do most of that clearing, which is one reason an intravenous dose is a liver problem before it is anything else.

An assay that only measures neutralisation misses this entirely. A patient can be seronegative by a neutralisation assay and still clear a dose quickly through binding antibodies alone. Total-antibody assays are run alongside for that reason.

7 Capsid-specific T cells

Four to eight weeks

The first three arms act on the particle. This one acts on the patient’s own cells, and it is the arm that takes away expression that was already working.

Figure 6: A capsid that never reached the nucleus becomes the antigen that destroys the cell that took it up.

The chain has four links, and the first one is a trafficking failure rather than an immune event:

A capsid fails to reach the nucleus. Most do. It is tagged with ubiquitin and fed to the proteasome, the cell’s general-purpose protein shredder.
The proteasome cuts it into short peptides. These are the ordinary raw material of the class I presentation pathway — every cell continuously advertises samples of its own protein contents this way.
Peptides are loaded onto MHC class I and displayed on the cell surface. A hepatocyte that took up a vector is now showing capsid fragments to anything that walks past.
A capsid-specific CD8 T cell recognises the complex and releases perforin, which opens the target membrane, and granzymes, which enter and trigger apoptosis.

The observable signature is specific and well documented: a rise in the liver enzymes ALT and AST at roughly four to eight weeks after infusion, coinciding with a fall in transgene expression. It looks like liver toxicity and it is really the immune system finding cells that took up the dose. Prophylactic corticosteroids, started before the enzymes rise, are the standard countermeasure, and their timing is set by that four-to-eight-week window.

Two points are easy to get backwards.

The antigen is usually the capsid, not the transgene. The capsid arrived by the thousand per cell and is presented immediately; the transgene product is a protein the patient may already be tolerant to. A transgene-directed response does happen, and it matters most when the patient’s own gene is deleted, so the therapeutic protein is foreign to them in the way a viral protein is.
The dose is already delivered when this arm acts. Steroids cannot recover killed cells. They prevent the killing of cells that are still expressing.

8 Complement

Minutes

Complement is a cascade of plasma proteins that amplifies: a small trigger cleaves a protein, the fragments cleave more, and the output climbs steeply with input. Both the classical pathway, triggered by antibody bound to a capsid, and the alternative pathway, which turns over continuously and settles on surfaces, activate on AAV.

Figure 7: Every step splits into a soluble alarm signal and a surface-bound fragment. Only the second branch reaches a membrane.

The cascade produces two kinds of output, and they cause different problems.

Anaphylatoxins. Cleaving C3 and C5 releases the small fragments C3a and C5a into the plasma. These recruit and activate neutrophils and mast cells and drive systemic inflammation. This is the fast, whole-body reaction, and it is the one that follows an infusion within hours.
The membrane attack complex. The larger fragments assemble C5b-9, a ring that inserts into a lipid membrane and opens a pore. On vascular endothelial cells, that damage exposes the surfaces that start clotting, and the clinical consequence is microvascular thrombosis with falling platelets — the picture reported as thrombotic microangiopathy after high-dose systemic AAV.

Complement is the arm most tightly coupled to dose, for two reasons that compound. The cascade is an amplifier, so it responds non-linearly to how much surface is presented to it; and circulating antibody-capsid complexes trigger the classical pathway, so a patient with any pre-existing titre starts the cascade with a much lower threshold. Dose and serostatus are not independent risks.

9 Dose

Every arm is dose-sensitive, and none is dose-sensitive in the same way.

Arm	What rises with dose	Shape
Complement	Surface presented to an amplifying cascade	Steeply non-linear
TLR2, TLR9	Particles sensed per cell	Roughly proportional
Antibody clearance	Saturates once titre is exhausted	Flattens at high dose
CD8 T cells	Capsid peptides displayed per cell	Proportional, then a threshold for killing

The trap is that the reason to raise the dose and the reason not to are the same number. Delivery is inefficient, so a therapeutic effect needs a large dose; a large dose is what turns the amplifying arms on. Systemic doses in the to vg/kg range are where the severe complement and liver events have been reported, and they are also where enough genomes arrive for some indications to work at all.

10 Clinical signatures

Each arm shows up as something a clinician measures, on its own clock. Reading backwards from the observation to the mechanism is most of how these events are managed.

Observation	Timing	Arm
Fever, raised IL-6 and TNF-	Hours	TLR2, innate
Raised type I interferon signature	Hours to days	TLR9, innate
Falling platelets, haemolysis, renal impairment	Hours to days	Complement, MAC
No expression at all despite a full dose	Immediate	Pre-existing neutralising antibody
ALT and AST rise with falling expression	4 to 8 weeks	Capsid-specific CD8 T cells
Second dose has no effect	Years	Post-dose neutralising antibody

The countermeasures line up with the same table: pre-screening for neutralising titre before enrolment, CpG depletion in cassette design, prophylactic corticosteroids over the T-cell window, and complement inhibition where the cascade is the presenting problem.

11 Constraints

The four arms are not independent. Innate signalling licenses the adaptive response, antibody-capsid complexes trigger complement, and complement fragments opsonise capsids for the same phagocytes that binding antibodies recruit. The widget treats the arms as separable because that is what makes the ordering visible, and the separability is the simplification.
Serotype changes the numbers, not the mechanisms. AAV8 and AAV9 differ from AAV2 in tropism, in seroprevalence, and in how strongly each arm engages. The route is the same.
Route of administration changes the exposure. Delivery to the eye, the central nervous system or a single muscle presents far less capsid to the circulation than an intravenous dose, and the complement and antibody arms weaken accordingly.
Preclinical models under-report this. Mice have different TLR9 distribution and different complement handling from humans, and no colony has the pre-existing anti-AAV antibody that a third to a half of adult patients arrive with.

Shell. Sensed. DNA. Sensed. Peptides. Presented. Dose. Amplifies. Screen. Titre. First.

12 References

Manno, C. S. et al. (2006). Successful transduction of liver in hemophilia by AAV-Factor IX and limitations imposed by the host immune response. Nature Medicine. The trial that identified the capsid-specific CD8 T-cell response and the ALT rise.
Zhu, J., Huang, X. and Yang, Y. (2009). The TLR9-MyD88 pathway is critical for adaptive immune responses to adeno-associated virus gene therapy vectors in mice. Journal of Clinical Investigation. The CpG-TLR9 link.
Faust, S. M. et al. (2013). CpG-depleted adeno-associated virus vectors evade immune detection. Journal of Clinical Investigation. CpG content as a manufacturing lever.
Hösel, M. et al. (2012). Toll-like receptor 2-mediated innate immune response in human nonparenchymal liver cells toward adeno-associated viral vectors. Hepatology. TLR2 sensing of the capsid.
Pillay, S. et al. (2016). An essential receptor for adeno-associated virus infection. Nature. Identification of AAVR, the entry receptor neutralising antibodies block access to.
Mingozzi, F. and High, K. A. (2013). Immune responses to AAV vectors: overcoming barriers to successful gene therapy. Blood. Review covering the humoral and cellular arms together.
Verdera, H. C., Kuranda, K. and Mingozzi, F. (2020). AAV vector immunogenicity in humans: a long journey to successful gene transfer. Molecular Therapy. Clinical synthesis, including complement and dose.

Why So Many Matrix Factorizations

Ravi Kalia — Thu, 03 Sep 2026 05:00:00 GMT

Why So Many Matrix Factorizations

A matrix is like a complicated 3D object: what you see depends entirely on where you stand. Just as rewriting a polynomial can instantly reveal its roots, factoring a matrix isolates the exact properties you need to solve a specific problem. We don’t choose between Cholesky, QR, or SVD because one is inherently superior to the others. We choose them because the jobs differ wildly. One day you need to solve least squares; the next, you need a rank- summary or a spectrum analysis. Each factorization is simply a different mathematical lens, perfectly ground to bring a specific application into focus.

1 Quadratic forms

The same parabola has four writings. Each writing makes one geometric fact cheap to read.

Standard — the -intercept is .
Vertex — the min or max is .
Factored — the roots are .
Matrix — the already holds . Cholesky of that is completing the square.

The curve in the widget is synthetic: are slider values, not fitted data. It stands in for any 1-D quadratic whose job changes with the writing.

Pick Vertex from the form menu and the widget marks the turning point . Pick Factored and it marks the roots when the curve crosses the axis, and says so when it does not. Pick Standard and it marks the intercept ; pick Matrix and it shows the itself. Each writing makes one feature the thing you can read off.

2 Which factorization

The benefit is not the smallest leading term. Cholesky is the cheapest factor of an SPD covariance and is not defined on the photograph. We pick a factorization for the object it hands to the next step: a coefficient, a draw, a weight, a rank- image, a spectrum, or an inverse that exists.

Table 1: Use case, not a ranking.

Factorization	Matrix it needs	Downstream job	What we take
QR	tall , full column rank	OLS	from , without
Cholesky	SPD	correlated draws; repeated SPD solves	, or two triangular solves
LU	square, not SPD	general (KKT, any invertible )	the solve
SVD	any shape, any rank	compression;	rank- summary, or the pseudoinverse
Eigen	square symmetric, modest	spectrum / PCA of a covariance we trust	and
NMF		parts or topics

NMF is iterative, . The Golub leukemia matrix is worked in Matrix Factorizations as Optimization Problems. For a Gram matrix , SVD of is the stable writing of the same spectrum. Flop-count models are Golub & Van Loan / Trefethen & Bau for dense , not wall-clock.

3 QR

Householder is orthogonal. Back-substitution on solves

without forming . Matrix Factorizations as Optimization Problems treats the same factorization on the wheat markers, where is large. The design here is ordinary OLS and well conditioned.

3.1 Diabetes

Source: Efron, Hastie, Johnstone, Tibshirani (2004), Least Angle Regression; the copy shipped with sklearn.datasets.load_diabetes.
Measurements: 442 patients × 10 baseline variables (age, sex, BMI, blood pressure, six serum assays), quantitative disease-progression response. sklearn standardises each column to mean 0 and squared length 1.
This post asks: recover two ways — normal equations vs QR — and report the coefficient gap.
A wrong : a progression score that ranks patients in the wrong order.
Why QR: the job is tall OLS. here, so the two solves will agree. QR is the factorization that remains accurate on a later design whose is large.

Code

Claude Agent SDK in Python

Ravi Kalia — Mon, 31 Aug 2026 05:00:00 GMT

Claude Agent SDK in Python

messages.create() returns one model turn. An agent that reads a file, edits it, and runs a test needs many turns. You append the tool_use, run the function, append the tool_result, and call again.

The Agent SDK wraps Claude Code so you skip that loop. Use query() for one shot. Use ClaudeSDKClient to keep talking.

query() starts a child process, works, and exits. ClaudeSDKClient starts the process once and leaves it up. A later prompt can say “the first one”.

%%{init: {"theme": "base", "themeVariables": {"fontFamily": "ui-sans-serif, system-ui, sans-serif", "lineColor": "#C96442", "edgeLabelBackground": "#FAF7F2", "clusterBkg": "#F4F0E8", "clusterBorder": "#C96442"}}}%%
flowchart TB
  Prompt["user prompt"] --> Options["ClaudeAgentOptions"]
  Options --> Query["query() — new session"]
  Options --> Client["ClaudeSDKClient — held session"]
  Query --> Create["messages.create"]
  Client --> Create
  Create --> Stop{"stop_reason"}
  Stop -->|"tool_use"| Tools["run Read / Edit / Bash"]
  Tools --> Append["append tool_result"]
  Append --> Create
  Stop -->|"end_turn"| Result["ResultMessage"]

  classDef cream fill:#F4F0E8,stroke:#C96442,stroke-width:2px,color:#191919
  classDef terracotta fill:#C96442,stroke:#8F3F24,stroke-width:2px,color:#FAF7F2
  classDef dark fill:#191919,stroke:#C96442,stroke-width:2px,color:#FAF7F2
  classDef peach fill:#E8A087,stroke:#C96442,stroke-width:2px,color:#191919

  class Prompt,Options cream
  class Query terracotta
  class Client dark
  class Create,Tools,Append peach
  class Stop,Result terracotta

query() shuts the child process down after one turn. The client keeps it. The diamond is stop_reason. Loop while the model wants a tool. Stop when it is done.

1 Agent SDK

Claude Code already does this in a terminal. You type a prompt. Claude reads a file or runs a command. It looks at the output. It decides whether to continue. The Agent SDK starts that process from Python. The package is claude-agent-sdk.

The child process:

starts and stops Claude Code
offers Read, Edit, Write, Bash, Glob, Grep
checks permissions (permission_mode, allowed_tools)
calls the model again after each tool result

messages.create() is one HTTP request. You send the history. You get one turn back. Claude does not open a shell. It does not edit files. It does not call itself again. A tool_use block means you run the function and send the next request. That package is the Anthropic Python SDK. Write the loop there. Let Claude Code write it here.

2 query vs ClaudeSDKClient

Two ways in. Both take ClaudeAgentOptions: system prompt, tools, permissions. Both are asyncio. Wrap them in async def main() and start with asyncio.run(main()). That asyncio.run starts the event loop. It is not a client method.

query() — one prompt, no follow-up. Opens a session, works, closes. No interrupt(). To continue later, pass continue_conversation=True or a resume id.
ClaudeSDKClient — the next prompt depends on the last answer. The async with block holds the connection. A second client.query() keeps the session. No resume. interrupt() works.

Open the client with async with ClaudeSDKClient(options) as client. Python connects on enter and disconnects on leave. After client.query(), read receive_response() until the turn ends. Then send the next prompt.

2.1 run

Other libraries ship client.run(prompt). One call sends and waits. ClaudeSDKClient has no .run(). It splits the turn so you can interrupt.

await client.query(prompt) — write the user turn. Returns when the write lands. Tools may still be running.
async for message in client.receive_response() — read until ResultMessage.

%%{init: {"theme": "base", "themeVariables": {"fontFamily": "ui-sans-serif, system-ui, sans-serif", "lineColor": "#C96442", "edgeLabelBackground": "#FAF7F2", "clusterBkg": "#F4F0E8", "clusterBorder": "#C96442"}}}%%
flowchart TB
  subgraph missing ["Guessed one-shot"]
    Run["client.run(prompt)"]
  end
  subgraph present ["Actual turn"]
    Q["query(prompt) — write"]
    Gap["interrupt() window"]
    Recv["receive_response() — wait"]
    Done["ResultMessage"]
    Q --> Gap --> Recv --> Done
  end
  Run -.-> Q

  classDef cream fill:#F4F0E8,stroke:#C96442,stroke-width:2px,color:#191919
  classDef terracotta fill:#C96442,stroke:#8F3F24,stroke-width:2px,color:#FAF7F2
  classDef dark fill:#191919,stroke:#C96442,stroke-width:2px,color:#FAF7F2
  classDef peach fill:#E8A087,stroke:#C96442,stroke-width:2px,color:#191919

  class Run peach
  class Q terracotta
  class Gap dark
  class Recv peach
  class Done terracotta

The dashed edge is run(). It is missing. Send, optionally cancel, then wait.

3 One-shot query

Use query() for one prompt. ClaudeAgentOptions sets the system prompt, the tools, and whether edits apply without asking.

Code

Fingerprint Algorithms: From Ridge Formation to On-Device Embeddings

Ravi Kalia — Mon, 31 Aug 2026 05:00:00 GMT

Fingerprint Algorithms: From Ridge Formation to On-Device Embeddings

You unlock a phone with your thumb a hundred times a week. It works when your thumb is wet, turned forty degrees off, pressed too hard, or only half on the glass. It fails when you offer a different finger.

The strange part is that the phone has no picture of your thumb. There is nothing in it to compare a photograph against. At enrolment it threw the picture away and kept something much smaller, and every unlock since has been a comparison between two of those smaller things.

So the interesting question is not “how do you match a fingerprint”. It is what do you keep, and what survives when the capture is bad. This post builds four answers to that question, using 397 real inked fingers from NIST, and scores all four against one shared gallery of two hundred of them. One of the four is a modern metric-learning network. It does not win, and the reason it does not win turns out to be the most useful thing here.

If you have trained a classifier, you already have most of the machinery for this. Matching is classification: each enrolled finger is a class, and a probe image has to be assigned to one of them. What makes it strange is the shape of the training data. There is one example per class, the classes are created after the model ships — you enrol a new thumb and nobody retrains anything — and there are as many classes as there are enrolled fingers. Those three conditions break the usual recipe of a softmax over a fixed label set, which is why the work moves out of the decision rule and into the representation. So the post compares four answers to “what do you keep”, under one decision rule held fixed.

1 A pattern nobody chose, not even your genome

Friction ridges form on the volar pads between roughly week 10 and week 17 of gestation. The basal layer of the epidermis is a stiff sheet growing on a soft foundation, and it is being compressed as the pad underneath it swells and then regresses. A stiff sheet under compression on a soft bed does not stay flat. It buckles, into a regular corrugation, at a wavelength set by the mechanics rather than by any instruction.

Kücken and Newell (2005) fit that picture to fetal pad geometry: ridge direction follows the lines of the compressive stress field, and the pattern type — loop, whorl, arch — falls out of the pad’s shape and timing. The same periodic stripes can be produced by a Turing reaction–diffusion system, which is why chemistry is the intuitive guess, but the model that matches the embryology is mechanical.

Where the ridge spacing comes from

For a stiff film of thickness and modulus bonded to a soft substrate of modulus , compression buckles the film at the wavelength that minimises the total energy — bending the film wants long waves, deforming the substrate wants short ones. The balance gives the classical wrinkling result

What matters is not the constant but that depends only on a thickness and a stiffness ratio, so a patch of skin has a ridge period whether or not anything told it what period to have. Every filter in this post has to be tuned to that period, and measuring it per print is the first thing the code does.

Once the epidermis keratinises the pattern is fixed for life, scars aside. And identical twins, who share a genome, do not share ridge detail — the minute arrangement of endings and forks differs between them, and between your own two index fingers. That is what makes this a usable identifier: the thing being measured is a developmental accident, not an inherited plan, so it is effectively a per-finger random draw that then never changes.

A biological signature in the molecular sense is a short list of features that stands in for a condition. A fingerprint template is the same move performed on skin. The engineering problem is to make the standing-in deterministic, so that two captures of one finger reduce to the same object and two captures of different fingers do not.

2 397 fingers, twice each, and what it cost to standardise them

Everything measured here comes from one cache, so here is what is in it before anything is asked of it.

What it is. NIST’s MINEX III validation imagery: 801 raw 8-bit grayscale scans of inked fingerprint cards at 500 dpi, from the usnistgov/minex repository (commit dc57b22), released into the public domain as a US Government work. Retrieved 2026-09-12; the built cache is 20.3 MB, sha256 426f1ccf….
Who made it and why. NIST publishes it so that vendors’ minutiae extractors and matchers can be validated against a common set before certification. It exists to be a benchmark, which is exactly why it is safe to use as one, and why the results here are not a comment on anyone’s product.
The structure that makes it a test. Files are named _ — a001_02 is set A, subject 001, right index. Sets A and B are two separate impressions of the same finger. Two impressions is the minimum that makes identification measurable: enrol one, search with the other, and ask which enrolled finger comes back first. 397 fingers have both impressions, giving 794 images; the remaining seven scans have no mate in the other set and are dropped.
What this post asks of it. Rank-1 identification against a gallery of two hundred fingers, and an equal error rate for the one-against-one case. Nothing about latents, and nothing about live sensors — these are inked cards.
What a wrong answer costs. In the deployed version of this problem, a false accept opens someone else’s phone and a false reject locks you out of your own. Those two errors trade against each other, which is why one accuracy number is never enough and the tables here carry both.

Each scan is put on one scale and one centre by src/fetch_data.py, and that step makes a trade which bites one of the four methods later:

Code

Tensor Inverses: Independence, Completion, Identification

Ravi Kalia — Mon, 31 Aug 2026 05:00:00 GMT

Tensor Inverses: Independence, Completion, Identification

You already undo a mixed recording, fill a blank rating, and pick plant settings from a desired output. For a square, full-rank matrix those three jobs share one inverse. A tensor does not. You pick a product, then extra structure, and a leftover still remains — permutation and scale, a rank, a lag.

A meeting transcript, a recommendation, and a dryer batch each hang on which leftover you keep.

Tensor Factorizations and Tensor Inverses has the algebra. Uses of Tensor Factorizations has the compression case. Tensor Inverses in Practice works four other problems and lists the products. This post takes three more — unmixing speech, filling missing ratings, identifying a dryer — and shows that even after you pick a product, the inverse is not unique.

1 Words used here

Mode, flatten, rank, null space, inverse, and pseudoinverse are defined in Tensor Inverses in Practice. Four more words appear below.

Cumulant. A moment of the data with the Gaussian part subtracted. The fourth-order cumulant of a vector is a tensor. Independent non-Gaussian sources make that tensor diagonal.
Joint diagonalization. One orthogonal change of basis that makes several matrices as diagonal as they can be at once. JADE does this to the slices of the cumulant.
Sampling operator. The map that keeps the observed cells of an array and throws the rest away. Its null space is every array that is zero on those cells.
Impulse-response tensor. For a linear plant, is how output responds to input at lag .

/Users/ravikalia/Code/github.com/ml-blog/.venv-tensor-factorizations/lib/python3.12/site-packages/tensorly/solvers/nnls.py:103: SyntaxWarning: invalid escape sequence '\l'
  .. math:: \lambda_s, \lambda_r
/Users/ravikalia/Code/github.com/ml-blog/.venv-tensor-factorizations/lib/python3.12/site-packages/tensorly/solvers/admm.py:99: SyntaxWarning: invalid escape sequence '\_'
  .. math:: dual\_var = dual\_var + (Ax + Bx_{split} - c)

2 Why the inverse is not unique

A matrix inverse undoes matrix multiplication. There is one matrix product, so there is one inverse when the matrix is square and full rank.

A tensor has several products. Each product has its own inverse. That is the algebra in the earlier posts. This post is about the leftover that stays after you pick one.

Independence. The mixing matrix is recoverable only up to permutation and scale, and only if the sources are independent with at most one Gaussian. Two signed permutations of both undo the mix. They are different matrices.
Missing entries. The sampling operator has a huge null space. Infinitely many arrays match the observed ratings. Rank and the ALS start pick one filling. Two ranks fill the same hole with two numbers.
MIMO identification. Input-output data underdetermines a full parameter tensor. A lag length picks one impulse-response tensor. Inverting that tensor for a control input is a second, different inverse. Two products, two settings.

In real work you solve, you do not invert. That line is already in Tensor Inverses in Practice. This post builds the inverse anyway, so you can see the leftover.

3 Data

Three sources. Two are downloaded. The speech mix is constructed from real recordings, so the recovery can be scored.

3.1 Three voices, one mix

What it is. Four seconds each of three Wikimedia recordings of Aesop’s The North Wind and the Sun: British English, Received Pronunciation (P. Roach / International Phonetic Association, CC BY-SA 3.0), Swedish, and Foochow (GnuDoyng, public domain). Resampled to 8 kHz mono.
Who recorded them and why. Phonetic archives keep this fable so accents can be compared. The recordings were not made for unmixing.
What this post asks of them. Mix the three tracks with a known matrix (seed 7) and try to get the voices back. The mix is constructed. The waveforms are not.
The business problem. A meeting recorder, a hearing aid, or a call centre hears several talkers on one microphone. The product is a transcript, a boosted voice, or a compliance log that follows one speaker.
What solving buys. Each voice comes back on its own channel. A transcript can attribute a sentence. A hearing aid can raise one talker and lower the others.
What a wrong inverse costs. Words land on the wrong speaker. A quote in minutes, a medical note, or a legal record names the person who did not say it.
Why a tensor inverse. The fourth-order cumulant of the mix is a tensor. Independent non-Gaussian sources make it diagonal. Inverting that structure is how you recover the mixing matrix. The inverse is unique only up to permutation and scale.

The three sources, then the three mixed channels.

English (RP)

Swedish

Foochow

Mixed channel 1

Mixed channel 2

Mixed channel 3

Code

Tensor Inverses in Practice

Ravi Kalia — Sun, 30 Aug 2026 05:00:00 GMT

Tensor Inverses in Practice

Four real problems where the operator is a tensor. For each one, this post names the product that applies, takes the inverse in that product, and measures what flattening the tensor into a matrix would have cost.

Two earlier posts give the background. Tensor Factorizations and Tensor Inverses has the algebra. Uses of Tensor Factorizations has the compression case. The first one shows that a tensor has no default product. Because of that, it has no default inverse either.

This post answers the question that leaves open. For a given problem, which product applies, and what does flattening cost?

The table of inverses comes first. Then the four problems. Then all the numbers side by side.

1 Words used here

Some of these words have a different everyday meaning. This is what they mean in this post.

Mode. One axis of an array. A array has three modes.
Order. How many modes an array has. A vector has order 1, a matrix order 2.
Frontal slice. For a three-mode array , the matrix . There is one for each value of .
Flatten. Rearrange a tensor into a matrix or a vector. Other papers say unfold or vectorise for the same idea. This post says flatten throughout.
Operator. Anything that takes an input and returns an output. Here it is usually a tensor that takes a tensor and returns a tensor.
Rank. How many independent directions an operator really uses. An operator on nine dimensions can have rank 6.
Null space. The set of inputs that an operator sends to zero. An operator with a null space larger than has no inverse.
Inverse. The operator that undoes another operator.
Pseudoinverse. What you use when no inverse exists. It undoes the operator on the directions the operator keeps, and returns zero on the directions the operator destroys.
Condition number, . How much an operator can magnify an error. If , an error in the twelfth digit of the input can reach the first digit of the output.
Residual. How far a computed answer sits from the answer it should give.
Covariance. A table saying how strongly each pair of measurements moves together.
Precision matrix. The inverse of a covariance matrix. Here the word does not mean accuracy.

2 The kinds of tensor inverse

A matrix inverse undoes matrix multiplication. There is one matrix product, so there is one matrix inverse.

Tensors work differently. There are several tensor products, and each one has its own inverse. You cannot use one in place of another, and none of them is the tensor inverse.

Inverse	Product it inverts	Order and shape it needs	Reach for it when	What it costs
Mode- pseudoinverse	mode- product	any order	you want to recover a factor, a mixing matrix, or a change of basis along one mode:	it inverts a matrix that acts on one mode. It does not invert the tensor. Readers confuse these two more often than anything else in this table
t-inverse / t-pseudoinverse	t-product	third order, with a meaningful third mode	the third mode is time, wavelength, or a frequency grid: video, hyperspectral cubes, some fMRI layouts	one FFT along mode 3, then matrix inverses. Each slice is inverted on its own, so a single bad slice ruins the whole answer
Einstein-product inverse	Einstein product	even order, square:	the tensor really is a linear map that takes a tensor and returns one: stiffness and compliance, or a PDE written out on a grid	it is the same object as a matrix, and it exists exactly when that matrix can be inverted
Multilinear Moore–Penrose	Einstein product	even order, rectangular or rank-deficient	same setting, but the operator is not square or not full rank	the one object that satisfies the four Penrose conditions; at order 2 it is the ordinary matrix pseudoinverse
Tucker- or TT-format inverse	mode products, TT contraction	any order, low multilinear or TT rank	the full operator will not fit in memory, and you need the inverse stored in the same compressed form	only approximate. The inverse usually needs a higher rank than the operator
CP-based approximate inverse	—	any order	a rough, cheap approximation is enough and you never need the exact inverse	nothing guarantees that it exists or that it is unique. Treat it as an approximation, not an inverse
Kronecker-separable precision	—	any order	you are modelling a covariance over a multi-way array (the tensor-normal or matrix-normal model)	not a tensor inverse at all. You are assuming that the inverse splits mode by mode. It is in this table because people mistake it for one

2.1 Two rules that come with the table

In real work you solve, you do not invert. This is the same rule as for matrices. Tensor Krylov methods, such as t-product GMRES, find in without ever building (El Guide et al. 2021). This post builds the inverse anyway, so that you can see what it looks like. That is a choice made for teaching, not advice for production code.

Using the wrong product is a mistake of kind, not a mistake of accuracy. Suppose comes out small. That tells you nothing about . The two products multiply along different modes. They are different equations that happen to use the same letters. Computing in higher precision will never turn one into the other.

3 Data

Four problems and four sources. Two are downloaded. One is a set of published material constants. One is simulated, because the real data needs a signed agreement.

3.1 An off-axis carbon/epoxy ply

What it is. Five elastic constants for a unidirectional T300/5208 carbon-fibre/epoxy ply at room temperature: GPa along the fibres, GPa across them, GPa, , . These are typical published values for that material, written out in full so you can check the arithmetic below. Nothing here turns on the third significant figure.
Who measures them and why. Materials labs pull, press and twist small test pieces cut from cured panels. Structural engineers use the results to predict how a part will bend before anyone builds it. The measurement exists to feed a stiffness model.
What this post asks of them. Build the fourth-order stiffness tensor and rotate the ply off the fibre axis. Then invert it to get compliance, so that a known stress returns the strain it caused.
What being wrong costs. Compliance turns a load into a deflection. An error here means an aircraft panel or a pressure vessel is sized against the wrong strain. That is a part that breaks, not a plot that looks wrong.
Why a tensor inverse. A stiffness tensor really is a linear map. It takes a symmetric matrix and returns a symmetric matrix. This is exactly what the Einstein product describes. It is also the one problem here where flattening works, so it is a fair place to start.

3.2 Pavia University

Provenance. A hyperspectral image of the University of Pavia campus in northern Italy, pixels over 103 spectral bands at 1.3 m ground resolution. The ROSIS sensor recorded it from an aircraft. Prof. Paolo Gamba released it for research; he works at the Telecommunications and Remote Sensing Laboratory at the University of Pavia. Graña, Veganzones and Ayerdi at UPV/EHU then put it into the Hyperspectral Remote Sensing Scenes collection, which serves it today.
Collector and motive. An aircraft flight, not a survey and not a web scrape. The team flew the scene and released it so that hyperspectral classification methods would share one benchmark. Hundreds of remote-sensing papers now use it.
What this post uses. A crop. src/fetch_data.py writes it to data/paviaU_crop.npz, which is committed. The official host blocks scripted downloads, so the script takes the whole scene from an open mirror, cuts out the crop, and throws the rest away.
What this post asks of it. Blur the cube with an operator that smears space and mixes neighbouring bands, then try to get it back two ways.
What being wrong costs. Hyperspectral analysis identifies a material from the shape of a pixel’s spectrum. Suppose a method sharpens the picture but leaves the bands mixed. It then reports the wrong material, and reports it with confidence. A mineral survey, a crop-disease map or a pollution estimate would rest on a spectrum that belongs to no real material.
Why a tensor inverse. The blur mixes the bands into each other. You therefore cannot treat each band as a separate problem. The t-product is built for this case: a third mode whose order means something, and slices that mix.

3.3 A synthetic four-dimensional scan

It is synthetic, and this is how it is made. A array. There are four regressors: a constant, two on-off task blocks with different periods, and a slow linear drift. Each regressor has its own Gaussian blob somewhere in space. The array is the design matrix times those blobs, plus independent Gaussian noise with . The random seed is 7.
What it stands in for, and why it is simulated. A task-fMRI session of the kind in ADNI or the CMU StarPlus set. Those need an application and a signed agreement, so a render cannot download them. There is a better reason too. This section has to recover a map that it already knows, so that it can score the result. No real scan comes with the answer attached.
What being wrong costs. A task map that puts activity in the wrong voxels becomes a published claim about which part of the brain does a job. The claim would be wrong.
Why a mode- pseudoinverse. Time is one mode out of four, and the design matrix acts only on that mode. Regress along that mode, treating the three space modes as samples, and the whole solve is . The alternative is written out below.

3.4 Chicago incident counts

Provenance. The City of Chicago open data portal, “Crimes — 2001 to Present” (Socrata resource ijzp-q8t2), a public extract of the police department’s records-management system.
Collector and motive. Officers of the Chicago Police Department enter each reported incident, and the city publishes the records under its open-data programme. These are reports, not offences. How often people report a crime varies by area and by offence type, and the portal says so.
What this post uses. All of 2023, for community areas 1–24 and the eight most common offence types. The counts go into a array: week by area by offence type. The bins are 52 whole weeks. The one leftover day of the year is thrown away, so that the last bin is not longer than the others. src/fetch_data.py writes the array to data/chicago_counts.npz, which is committed, so a render never calls the API. The cell below prints the incident count.
What this post asks of it. Estimate the covariance of the weekly array. A forecast needs that covariance before it can give any error bars.
What being wrong costs. Cities use covariance estimates to plan patrols and staffing. A model that is confident about the wrong areas sends officers to the wrong areas.
Why a structured inverse. Flattened, one week is a 192-dimensional vector, and there are only 52 weeks. The full covariance therefore cannot be inverted. A precision matrix that splits mode by mode can be fitted from the same data. As that section explains, splitting it is an assumption you are making, not an inverse you are taking.

4 Compliance: the Einstein product

Who does this. A mechanical engineer setting up a finite-element run on an anisotropic laminate.

flowchart LR
  S["σ
stress, 3 × 3"] --> C["ℂ
stiffness
3 × 3 × 3 × 3"]
  C -->|"invert under ⊛"| SS["𝕊 = ℂ⁺
compliance"]
  SS --> E["ε = 𝕊 ⊛ σ
strain, 3 × 3"]

Figure 1: Stress in, strain out. The operator is a fourth-order stiffness tensor, and the inverse is taken under the Einstein product.

Hooke’s law in tensor form is . Summing over the last two indices of against a matrix is the Einstein product.

So takes a matrix and returns a matrix. Compliance is the operator that undoes it.

Build the ply, rotate it off the fibre axis, and look at the operator as a matrix.

Code

Claude API with the Anthropic Python SDK

Ravi Kalia — Sat, 29 Aug 2026 05:00:00 GMT

Claude API with the Anthropic Python SDK

The official anthropic Python SDK talks to Claude through one method, client.messages.create(). Roles, content blocks, tools, and retrieved documents are fields on that call — not separate endpoints.

1 Messages

Anthropic() reads ANTHROPIC_API_KEY from the environment. Every request is stateless: resend the full messages list each turn. This post uses claude-sonnet-5, the current Sonnet API id.

On that model, messages accepts two roles:

user — the human turn, or a tool_result you are sending back.
assistant — a prior model turn you are replaying.

System instructions are a top-level system parameter, not a role in that list — a string, or a list of text blocks. max_tokens is required.

The Opus-tier models take a third role. On claude-opus-5, claude-opus-4-8, claude-fable-5 and claude-mythos-5 you can append {"role": "system", "content": ...} inside messages to give the model a new operator instruction part-way through a conversation. Rewriting the top-level system would say the same thing and throw the cached prefix away with it; a message appended at the end does not. It cannot be the first entry, and it has to be either the last one or followed by an assistant turn. claude-sonnet-5 rejects the role.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system="Answer in one sentence.",
    messages=[{"role": "user", "content": "What is a content block?"}],
)

A string content is shorthand for one text block. The response content is always a list; do not assume response.content[0].text. Streaming, timeouts, and input_json_delta are covered in Waiting, and Not Waiting. Schema-constrained JSON is covered in Stop Parsing Prose.

2 Content blocks

Each block has a type. Inspect block.type before reading any other field. claude-sonnet-5 may emit a thinking block before text; a tool call adds another.

Documented input and output types used in this post:

text — ordinary prose. String content expands to this.
image — a source object (base64, url, or file / file_id).
document — plain text, PDF, or custom content; optional citations.
search_result — a retrieved hit you already have, also citation-capable.
tool_use — model output: id, name, input.
tool_result — your follow-up: tool_use_id plus content.
thinking / redacted_thinking — reasoning blocks. Pass them back unchanged on the next turn if the API returned them.

for block in response.content:
    if block.type == "text":
        print(block.text)
    elif block.type == "tool_use":
        print(block.name, block.input)

Unknown types should be ignored or forwarded, not parsed as text.

3 Tools

A client tool is a function you run. Pass name, description, and a JSON Schema input_schema. The model returns stop_reason="tool_use" and one or more tool_use blocks. You execute the function, then send a user message whose content is a tool_result whose tool_use_id matches.

A server tool (web_search, web_fetch, code_execution, and similar) runs on Anthropic’s infrastructure. You declare it by type and name; you do not execute it or invent a tool_result for it.

The loop for a client tool is always: request → tool_use → your code → tool_result → request again, with both the assistant turn and the result appended.

3.1 Stop reasons

stop_reason says why the model stopped. It has six values, not two:

end_turn — finished on its own. Read the text.
stop_sequence — hit a stop sequence you configured. Also finished.
tool_use — wants tools run. The loop above.
max_tokens — cut off part-way. Retry with a bigger cap; do not read it as an answer.
pause_turn — a server tool hit its own iteration limit. Send the history back unchanged to resume.
refusal — declined on safety grounds. Only here is stop_details filled in; it is null for the other five.

A loop written as while response.stop_reason == "tool_use" exits on the other five. Two of those exits are right — end_turn and stop_sequence really are finished. The remaining three are not, and each hands the caller a partial answer with no error. The Orchestrator-Worker Loop branches on all six and sets the budgets that stop a run going forever.

3.2 Tool runner

The SDK will run that loop for you. Decorate a function with @beta_tool, pass it to client.beta.messages.tool_runner(...), and call runner.until_done(). It dispatches the calls, appends the results, and returns the final message. The helper lives on client.beta.messages, so it needs the beta namespace.

Write the loop by hand once, to see what the runner is doing. Use the runner in production.

4 A tool-use round trip

The dict below is synthetic. Three invented refund rows stand in for an internal policy store a support bot would query. A real policy table would be confidential, and it would be out of date by the time you read this. Neither matters here: what the example shows is the shape of the blocks going in and out, not the rows. No live system is called; this post has no API key, so the cell is not executed.

import anthropic

POLICIES = {
    "standard": "Refunds within 30 days of purchase, unused items only.",
    "plus": "Refunds within 60 days; opened software is excluded.",
    "enterprise": "Refunds require a ticket and manager approval within 90 days.",
}


def lookup_policy(plan: str) -> str:
    return POLICIES.get(plan, "No policy found for that plan.")


client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

tools = [
    {
        "name": "lookup_policy",
        "description": "Return the refund policy text for a subscription plan.",
        "input_schema": {
            "type": "object",
            "properties": {
                "plan": {
                    "type": "string",
                    "enum": ["standard", "plus", "enterprise"],
                    "description": "The customer's subscription plan.",
                }
            },
            "required": ["plan"],
        },
    }
]

messages = [
    {"role": "user", "content": "What is the refund window on the Plus plan?"},
]

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system="Answer only from lookup_policy. If the tool returns nothing, say so.",
    tools=tools,
    messages=messages,
)

if response.stop_reason == "tool_use":
    messages.append({"role": "assistant", "content": response.content})
    results = []
    for block in response.content:
        if block.type == "tool_use":
            results.append(
                {
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": lookup_policy(block.input["plan"]),
                }
            )
    messages.append({"role": "user", "content": results})
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=1024,
        system="Answer only from lookup_policy. If the tool returns nothing, say so.",
        tools=tools,
        messages=messages,
    )

print(response)
text = next(b.text for b in response.content if b.type == "text")
print(text)

The objects below are hand-written to show the two Message shapes that loop produces. They are not a captured transcript.

First return, stop_reason="tool_use":

Message(
    id='msg_01SimulatedFirstTurn000000001',
    type='message',
    role='assistant',
    model='claude-sonnet-5',
    content=[
        ToolUseBlock(
            id='toolu_01SimulatedLookup00000001',
            name='lookup_policy',
            input={'plan': 'plus'},
            type='tool_use',
        )
    ],
    stop_reason='tool_use',
    usage=Usage(input_tokens=412, output_tokens=56),
)

Second return, after tool_result:

Message(
    id='msg_01SimulatedSecondTurn00000001',
    type='message',
    role='assistant',
    model='claude-sonnet-5',
    content=[
        TextBlock(
            type='text',
            text='The Plus plan allows refunds within 60 days; opened software is excluded.',
        )
    ],
    stop_reason='end_turn',
    usage=Usage(input_tokens=481, output_tokens=24),
)

Check stop_reason before reading text. This exchange only produces two of the six: tool_use means there is no final answer yet, end_turn is the text turn. A loop that meets the other four still needs the branches above.

5 RAG

The SDK has no vector store and no embeddings client. Retrieval is your job; Claude only sees what you put on the next request.

Three patterns the Messages API actually supports:

Stuff the hits. Put retrieved text in the user turn as document or search_result blocks, then ask the question as a text block.
Citations. Set citations={"enabled": True} on those documents (all or none in the request). The model can emit citation blocks pointing at spans. Citations cannot be combined with structured outputs (output_config.format).
Retrieval as a tool. Expose search_kb (or similar) as a client tool. Claude decides when to call it; you run the retriever and return a tool_result. Same loop as lookup_policy above.

A stable document prefix can carry cache_control={"type": "ephemeral"} so later turns reuse it at cache-read prices. Changing the prefix, the tool list, or the model id misses the cache.

5.1 Embeddings and stores

Anthropic does not offer an embedding model. The docs name Voyage AI as the provider to start with (voyageai package, VOYAGE_API_KEY). Use voyage-4 for a general index; input_type="document" on ingest and input_type="query" at search. Assess other vendors if the domain needs it.

The SDK still has no vector store. Typical indexes for the Voyage vectors:

pgvector — default when you already run Postgres. Hosted as RDS, Cloud SQL, Azure Database for PostgreSQL, Supabase, or Neon.
Pinecone — managed, serverless. Anthropic’s RAG cookbook uses it.
Qdrant or Weaviate — self-host or their clouds when you want an OSS engine and hybrid (keyword + vector) search. Weaviate vs Pinecone compares the two; Retrieval-augmented generation shows the FAISS-shaped core and why its recall bounds the answer.

5.2 Hosting

Claude itself is not only api.anthropic.com. The same Python package exposes first-party plus cloud hosts:

Anthropic() — first-party Messages API.
AnthropicAWS — Claude Platform on AWS (anthropic[aws]).
AnthropicBedrockMantle — Amazon Bedrock (anthropic[bedrock]). AnthropicBedrock still works, but it is the older bedrock-runtime InvokeModel path; new code wants the Mantle client.
AnthropicVertex — Google Vertex AI (anthropic[vertex]).
AnthropicFoundry — Microsoft Foundry.

Who runs the service decides what you get. Claude Platform on AWS is Anthropic-operated: AWS sign-in and billing, the first-party API surface, same-day feature parity, and plain model ids. Bedrock and Vertex are partner-operated, so each has its own prices, its own release lag, and a smaller feature set — and Bedrock prefixes the model id (anthropic.claude-sonnet-5). Foundry bills through the Microsoft Marketplace at first-party rates.

Voyage embeddings are a separate key: Voyage’s API, or Voyage on AWS Marketplace. The vector store is a third bill — Pinecone, Qdrant Cloud, Weaviate Cloud, or pgvector on the Postgres you already pay for.

6 Audio

The Messages API has no speech endpoint and no audio content block. The OpenAI-compatible surface ignores audio and input_audio fields.

Claude Code /voice dictation is a CLI product feature. It is not client.messages.create().

Spoken I/O is a second vendor. Anthropic’s cookbook builds a low-latency voice loop with ElevenLabs for STT and TTS around a streaming Claude turn. Keep Claude on text; send transcripts in and pipe tokens out to ElevenLabs. Voice AI architectures (2026) compares ElevenLabs to the rest of that market.

7 CLAUDE.md

CLAUDE.md is a Claude Code convention: a markdown file the coding agent reads at session start and holds as project context. Analogues are AGENTS.md and editor rule files. This repository’s root CLAUDE.md is one — render commands, register, freeze rules.

client.messages.create() does not load it. Nothing in the Python SDK opens the working tree or injects a file by that name.

For an API app, put the same rules in system (optionally as cached text blocks). For a coding agent, keep them in CLAUDE.md and let the agent host load them. Do not expect a claude_md= parameter.

Messages. Stay. Stateless. Voyage. Embeds. Stores. Host. Voice. Needs. ElevenLabs.

8 References

Anthropic: Messages API
Anthropic: Tool use
Anthropic: Citations
Anthropic: Embeddings (Voyage)
Anthropic: Python SDK
Anthropic cookbook: RAG
Anthropic cookbook: ElevenLabs voice
Claude Code: Memory / CLAUDE.md
Claude Certified Architect, Week 1: The Orchestrator-Worker Loop — this blog; the loop that branches on every stop_reason this post lists.

Claude Certified Architect, Week 1: The Orchestrator-Worker Loop

Ravi Kalia — Sat, 29 Aug 2026 05:00:00 GMT

Claude Certified Architect, Week 1

An agent loop is a while loop around one API call. Each time round, the model tells you why it stopped. There are six things it can say, and a loop that only handles one of them will hand back a half-finished answer and call it done.

These are notes from the first of a few weeks studying for the Claude Certified Architect – Foundations exam. Its biggest topic is Agentic Architecture and Orchestration.

The full implementation is orchestrator.py, in this post’s directory. Apart from the single-call example, every code block is copied straight out of it; the only change is the indentation.

I learned to think this way at SoFi (Social Finance, Inc.), building apps that use LLMs to take over repetitive regulatory work. Nobody reads the loop’s output there before it goes into a filing. A wrong answer that looks finished does more damage than a crash.

1 Scope

This week covers one pattern properly instead of skimming all six exam domains.

Domain: Agentic Architecture and Orchestration — 27% of the exam, the largest single block.
Pattern: orchestrator-worker — one model plans and splits the work, then hands each piece to a worker loop that can reach only the tools it needs.
Surface: the Claude Messages API and a hand-written loop. The SDK’s client.beta.messages.tool_runner handles this loop for you and is the right default in production; writing it out by hand once is what makes the rest of the material make sense.
Model: claude-opus-5 with adaptive thinking, which is on by default. Depth and spend are set with output_config.effort — low through max — not with a token budget; budget_tokens is rejected on this model.

2 Prompting versus architecture

Prompting and agent design break in different places. Moving from one to the other changes what you are on the hook for.

	Ad-hoc prompting	Agentic architecture
Unit of work	One request, one response	A loop over many requests
Who decides the next step	A human reading the output	Your control flow, reading `stop_reason`
Primary failure surface	Prompt wording	State handling, tool boundary, budgets
Effect of a bad output	A human notices	A malformed argument reaches a database
Cost ceiling	One call	Unbounded until you impose one

This does not make prompting matter less. It moves where it matters. The prompt is still a lever, and a wider one than plain text: Soft Prompts vs. Hard Prompts starts a prompt on GPT-2’s vocabulary and, sixty gradient steps later, it has drifted somewhere no sentence can reach. But no prompt of either kind tells your code what to do when the model stops mid-sentence.

So the model’s output stops being an answer. It becomes an instruction to your code.

3 A single Messages API call

The API is stateless, so every call carries the whole conversation. This is the call the loop wraps.

import anthropic

client = anthropic.Anthropic()  # ANTHROPIC_API_KEY, or an `ant auth login` profile

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    system="Answer in one sentence.",
    messages=[{"role": "user", "content": "Why should an agent loop check stop_reason?"}],
)

print(response.stop_reason)          # "end_turn"
print(response.usage.output_tokens)

for block in response.content:       # a list of typed blocks, not a string
    if block.type == "text":
        print(block.text)

Three things in that response matter later.

stop_reason rides on every response. The next sections are the six values it can take, and what your code must do about them.
content is a list of typed blocks, not a string. On claude-opus-5 thinking is on by default, so the list can hold thinking blocks beside text ones — the loop checks block.type, and appends the list rather than the text.
messages is the entire history, sent again every call. Nothing is kept server-side.

The rest of that call — the roles messages takes, the content-block types, how a tool is defined, how retrieved text goes in as a document block — is surveyed in Claude API with the Anthropic Python SDK. This post assumes that surface and stays on the loop around it.

The loop that wraps this call is the one LLM Agents from First Principles builds from scratch, against a raw model with its own text parsing. The Messages API changes what is hard: a tool call arrives as a typed tool_use block instead of text to be picked apart, so the parsing step disappears and the branching step is what is left to get right.

4 The four-stage lifecycle

One turn of an agent loop has four stages. Three of the four are code you own; only the first is prompting.

flowchart LR
    A["Stage 1: initial prompt
system + tools + history"] --> B{"Stage 2
why did it stop?"}
    B -->|end_turn / stop_sequence| Z["Return to caller"]
    B -->|refusal| R["Stop here:
read stop_details"]
    B -->|max_tokens| T["Retry bigger,
then fail loudly"]
    B -->|pause_turn| P["Resume,
capped"]
    B -->|tool_use| C["Stage 3: tool execution
check, run, catch"]
    C --> D["Stage 4: state hand-off
append content + results"]
    D --> A
    P --> A

4.1 Stage 1: initial prompt

The request carries the system prompt, the tool definitions, and the entire history — the API is stateless.

Put the parts that never change first (fixed system text, a tool list in a fixed order) and the parts that change every call last, so the cache still hits.
A timestamp or a request id in the system prompt quietly breaks the cache on every call. If usage.cache_read_input_tokens is zero on calls that should share a prefix, that is what happened.

4.2 Stage 2: stop reason check

The model tells you why it stopped. Everything else in the loop hangs off this one check, and it is the part most code gets wrong.

4.3 Stage 3: tool execution

Each tool_use block names a tool and carries already-parsed JSON input.

Check the input before you run anything. strict: true on the tool definition makes the API reject bad input before it ever reaches your handler.
One turn can carry several tool_use blocks. Running them concurrently is the point of that, though this loop dispatches them in order to keep the error path readable.
Never match on the raw JSON text; models escape Unicode and forward slashes differently.

4.4 Stage 4: state hand-off

The results go back as the next user turn, and this stage has three hard rules.

Append the whole response.content list, not just its text. Dropping the tool_use and thinking blocks makes the next request invalid.
Every tool_use_id from the assistant turn needs a matching tool_result. Miss one and the next request is rejected.
All results from one turn go in a single user message. Splitting them trains the model out of making parallel tool calls.

5 Stop reasons

Six values. Each one needs different code.

`stop_reason`	Meaning	Required action
`end_turn`	Finished naturally	Return the text
`stop_sequence`	Hit a configured stop sequence	Return the text
`tool_use`	Wants one or more tools run	Run them, hand the results back, go round again
`max_tokens`	Output cut off part-way	Retry with a bigger cap, then fail loudly
`pause_turn`	A server-side tool hit its own limit	Send the history back unchanged to resume, and cap the resumes
`refusal`	Turned the request down on safety grounds	Stop here; read `stop_details`

stop_details is filled in only for refusal. It is null for every other reason, so check before you read it.

The loop gives every value its own outcome, and sets limits the model never sees:

class Outcome(str, Enum):
    """How a loop ended. Every value maps to one branch in `run_loop`."""

    COMPLETED = "completed"  # stop_reason end_turn / stop_sequence
    TRUNCATED = "truncated"  # stop_reason max_tokens, retry exhausted
    REFUSED = "refused"  # stop_reason refusal
    TURN_LIMIT = "turn_limit"  # our own ceiling, not the model's
    STALLED = "stalled"  # same tool call repeated, no progress
    ABORTED = "aborted"  # ToolAbort from a handler
    UNKNOWN_STOP = "unknown_stop"  # a stop_reason this code has never seen


@dataclass(frozen=True)
class Budget:
    """Ceilings the model cannot see and therefore cannot talk its way past."""

    max_turns: int = 12
    max_tool_calls: int = 40
    #: `pause_turn` means a server-side tool loop hit its own iteration cap.
    #: Resuming is correct; resuming without a bound is an infinite loop.
    max_pause_resumes: int = 5
    #: One retry at a larger `max_tokens` before declaring truncation.
    max_token_retries: int = 1
    #: Identical (name, input) calls tolerated before the run is called stalled.
    max_repeats: int = 2

The checks are a flat ladder, and nothing drops off the bottom. A reason this code has never seen — one the API adds after it ships — stops the run instead of being taken for success:

stop = response.stop_reason
log.debug("turn %d stop_reason=%s", result.turns, stop)

if stop in ("end_turn", "stop_sequence"):
    result.outcome = Outcome.COMPLETED
    return result

if stop == "refusal":
    # `stop_details` is populated only for refusals; guard before use.
    details = response.stop_details
    result.outcome = Outcome.REFUSED
    result.detail = getattr(details, "explanation", None) or "no explanation"
    log.error(
        "refused (%s): %s",
        getattr(details, "category", None),
        result.detail,
    )
    return result

if stop == "max_tokens":
    # Output was cut mid-sentence. Continuing as if it were complete is
    # how a half-written tool argument reaches a database.
    if token_retries < budget.max_token_retries:
        token_retries += 1
        turn_max_tokens *= 2
        messages.pop()  # discard the truncated turn before retrying
        result.text = previous_text  # ... and the text it contributed
        log.warning("truncated; retrying at max_tokens=%d", turn_max_tokens)
        continue
    result.outcome = Outcome.TRUNCATED
    result.detail = f"still truncated after {token_retries} retry(ies)"
    return result

if stop == "pause_turn":
    # A server-side tool (web search, code execution) hit its own
    # iteration limit. Re-send the history unchanged -- the API sees the
    # trailing server_tool_use block and resumes; an added "continue"
    # message confuses it.
    pause_resumes += 1
    if pause_resumes > budget.max_pause_resumes:
        result.outcome = Outcome.TURN_LIMIT
        result.detail = f"paused {pause_resumes} times without finishing"
        return result
    continue

if stop != "tool_use":
    # A reason added to the API after this code was written. Stopping
    # is the honest response; falling through would invent an answer.
    result.outcome = Outcome.UNKNOWN_STOP
    result.detail = f"unhandled stop_reason {stop!r}"
    log.error(result.detail)
    return result

Note the messages.pop(). The cut-off turn is thrown away before the retry. Leave it in and the retry just adds a second turn carrying on from half a sentence.

6 Tool execution and errors

A broken tool is something you tell the model, not an exception that tears down the loop. Whatever goes wrong — no such tool, bad arguments, the handler blew up, it took too long — comes back as a tool_result marked is_error: True. The model reads it and tries something else.

started = time.monotonic()
try:
    output = _call_with_deadline(
        lambda: tool.handler(**block.input),
        tool.timeout_s,
    )
except TimeoutError:
    log.warning("%s exceeded %gs; abandoned", tool.name, tool.timeout_s)
    return _error_result(
        block.id,
        f"{tool.name} did not finish within {tool.timeout_s:g}s and was "
        "abandoned. Narrow the request and try again.",
    )
except ToolFailure as exc:
    log.warning("%s failed: %s", tool.name, exc)
    return _error_result(block.id, str(exc))
except ToolAbort:
    raise
except Exception as exc:  # noqa: BLE001 - a handler bug must not kill the run
    log.exception("%s raised", tool.name)
    return _error_result(
        block.id,
        f"{tool.name} failed with {type(exc).__name__}: {exc}",
    )

log.debug("%s ran in %.2fs", tool.name, time.monotonic() - started)

return {
    "type": "tool_result",
    "tool_use_id": block.id,
    "content": output if isinstance(output, str) else json.dumps(output),
}

ToolAbort is the exception: dead credentials, a missing binary, a tripped breaker. The model cannot fix any of those, so the run stops instead of retrying against something that is already broken.

Error results carry the same tool_use_id as any other:

def _error_result(tool_use_id: str, message: str) -> dict[str, Any]:
    return {
        "type": "tool_result",
        "tool_use_id": tool_use_id,
        "content": message,
        "is_error": True,
    }

7 Loop limits

Two limits in the tool_use branch stop a run going forever: a hard cap on tool calls, and a check for repeats.

# --- stop_reason == "tool_use" -------------------------------------
calls = [b for b in response.content if b.type == "tool_use"]
granted = {t.get("name") for t in tools}
tool_results: list[dict[str, Any]] = []
terminal: tuple[Outcome, str] | None = None
for block in calls:
    if result.tool_calls >= budget.max_tool_calls:
        terminal = (
            Outcome.TURN_LIMIT,
            f"exceeded {budget.max_tool_calls} tool calls",
        )
        break

    # An agent that repeats a call verbatim is not making progress; it
    # is usually re-reading a resource whose result it misread. Say so
    # in-band before the turn ceiling burns the whole budget.
    signature = (block.name, json.dumps(block.input, sort_keys=True))
    seen_calls[signature] = seen_calls.get(signature, 0) + 1
    if seen_calls[signature] > budget.max_repeats:
        # Answer the block anyway -- every tool_use id in the turn needs
        # a result or the history cannot be resent -- then stop.
        tool_results.append(
            _error_result(
                block.id,
                f"{block.name} has already been called with these exact "
                "arguments and returned the same result. Use a different "
                "approach or state what is blocking you.",
            ),
        )
        terminal = (
            Outcome.STALLED,
            f"{block.name} repeated {seen_calls[signature]} times",
        )
        break

    try:
        result.tool_calls += 1  # counted only when it actually runs
        tool_results.append(registry.dispatch(block, granted))
    except ToolAbort as exc:
        terminal = (Outcome.ABORTED, str(exc))
        log.error("aborted by %s: %s", block.name, exc)
        break

# However the turn ended, every tool_use id in it needs a result or
# the history cannot be resent -- so answer the blocks that never ran
# before handing it back. This is why the loop breaks rather than
# returning from inside the loop.
answered = {r["tool_use_id"] for r in tool_results}
tool_results.extend(
    _error_result(b.id, "Not run: the loop stopped part-way through this turn.")
    for b in calls
    if b.id not in answered
)

# All results go back in one user message. Splitting them across
# several messages trains the model out of parallel tool calls.
messages.append({"role": "user", "content": tool_results})

if terminal is not None:
    result.outcome, result.detail = terminal
    return result

The repeat check is the cheaper of the two. An agent calling the same tool with the same arguments is stuck, usually re-reading something whose answer it misread the first time. Saying so in the tool result gives it a chance to recover before the turn limit eats the whole budget.

8 Delegation as a tool call

The orchestrator has one tool, and it starts a worker loop. Because handing off work is a plain tool call and not a special case, it gets that error path: a worker that gets stuck, cut off, or refused comes back as an is_error result the planner can work around.

def delegate(task: str, tools: list[str]) -> str:
    spend["workers"] += 1
    if spend["workers"] > max_workers:
        # Its own ceiling, deliberately below the planner's tool-call cap.
        # Sharing that cap made this unreachable: run_loop stops before it
        # dispatches the call that would exceed it.
        raise ToolAbort(f"worker budget exhausted after {max_workers} workers")

    unknown = [name for name in tools if name not in registry.names]
    if unknown:
        raise ToolFailure(f"no such tool(s): {', '.join(unknown)}")

    worker = run_loop(
        client,
        system=worker_system,
        messages=[{"role": "user", "content": task}],
        tools=registry.params(tools),  # run_loop enforces this grant
        registry=registry,
        budget=worker_budget,
        model=model,
    )
    if not worker.ok:
        # Surfaced to the planner as a tool failure, with the outcome name
        # so it can distinguish "ran out of turns" from "was refused".
        raise ToolFailure(
            f"worker ended as {worker.outcome.value} ({worker.detail}). "
            f"Partial output: {worker.text[:500] or '(none)'}",
        )
    return worker.text

Two things fall out of this.

The worker starts clean. It sees only the task string, so a long planner history never bloats worker requests.
A worker gets only what it needs. The planner names the tools that worker may call, and run_loop derives the allowed set from the definitions it sent, so a call to anything else comes back refused. Narrowing the definitions alone would not do it: a model can name a tool it was never shown, either by hallucinating a name from an earlier result or because injected text told it to.

9 Anti-patterns and production failure modes

All eight pass a demo and fail in production. The first two hurt most in regulatory work, where nothing about a cut-off answer looks different from a finished one.

while response.stop_reason == "tool_use". The most common bug by far. A max_tokens truncation, a refusal, and a pause_turn all exit the loop as though the model had finished, and the caller gets a partial answer with no error. Branch on every value by name.
Unhandled schema validation errors. A handler that raises on a bad argument tears down the loop and throws away a turn you could have saved. Worse, catching it and dropping the block leaves a tool_use_id unanswered, and the next request comes back as a 400 that names neither the tool nor the turn. Always send a result, and set is_error.
Infinite execution loops. Three separate causes: no turn limit, resuming pause_turn forever, and a model retrying the same call forever. All three need separate limits. One max_turns hides the other two until the bill shows up.
Appending only the text. messages.append({"role": "assistant", "content": response.text}) loses the tool_use and thinking blocks. It breaks on the next request, well away from the line that caused it.
Splitting parallel tool results. The API accepts each tool_result in its own user message, and doing that quietly teaches the model to stop batching calls. Things get slower over a session; the API never complains.
One broad except. Catching APIStatusError alone lumps the errors worth retrying (429, 5xx, connection) in with the ones that will never succeed (400, 404). Catch the specific ones first.
A volatile cache prefix. A clock, a UUID, or an unsorted json.dumps() in the system prompt or tool list breaks the prefix on every call. The bill triples and nothing errors.
Uniform tool grants. Give every worker the full tool list and one bad call can reach anything.

10 Takeaways

Treat stop_reason as six named cases, not a yes or no.
A broken tool sends a message back. A broken dependency raises. Use different exception types for the two.
Limit turns, tool calls, pause resumes, and repeated calls separately. They run away in four different ways.
Give a worker the smallest tool set its job needs, and one self-contained instruction instead of the whole history.
Use the SDK’s tool runner in production. Write the loop by hand once, to see what it is doing for you.
Check caching with usage.cache_read_input_tokens. Do not assume you put the breakpoint in the right place.

Stop. Reasons. Branch. Errors. Return. Inline. Budgets. Bound. Loops. Delegate. Narrowly.

11 References

Handling stop reasons — Anthropic docs; what to do for each stop reason.
Building effective agents — Anthropic engineering; where orchestrator-worker sits among the agent patterns.
anthropic-sdk-python — the SDK, including client.beta.messages.tool_runner.
Claude Certified Architect – Foundations exam guide (Anthropic) — where the 27% figure comes from.
Soft Prompts vs. Hard Prompts — this blog; the hard-versus-soft prompt distinction.
LLM Agents from First Principles — this blog; the same loop built from scratch against a raw model.
Claude API with the Anthropic Python SDK — this blog; the request surface this loop wraps.
orchestrator.py — the full code the blocks here are copied from.

GitHub vs GitLab: Origins, Dominance, and AI-Native Hosts

Ravi Kalia — Sat, 29 Aug 2026 05:00:00 GMT

GitHub vs GitLab: Origins, Dominance, and AI-Native Hosts

GitHub and GitLab differ by who built them, who owns them, and who leads today — and four AI-native platforms are now reshaping git hosting around them. The comparison assumes GitHub as the default workflow most open-source projects and individual developers already use.

1 Origins and ownership

Both platforms wrap the same Git protocol. Their histories explain why one feels like a social network for code and the other like a single-vendor DevOps stack.

GitHub (2008): Tom Preston-Werner, Chris Wanstrath, P. J. Hyett, and Scott Chacon; San Francisco. Bootstrapped profitably, then VC-backed. Microsoft acquired it for $7.5B (closed Oct 2018); wholly owned subsidiary since, with Copilot as the clearest AI bet on the platform.
GitLab (2011 OSS, 2014 Inc.): Dmitriy Zaporozhets started the project in Ukraine; Sytse Sijbrandij built the company from the Netherlands. Y Combinator 2015. Public on Nasdaq (GTLB) since Oct 2021 IPO; Sijbrandij executive chairman.

2 Dominance today

Git won the version-control war. The competition is between forges — services that host remotes, run CI, and gate merges.

GitHub leads on adoption: ~38% SCM market share vs GitLab ~16%; roughly 2–3× on surveys and open-source gravity (Stack Overflow 2025: ~81% GitHub vs ~36% GitLab).
GitLab’s wedge: enterprise DevOps — integrated CI/security, self-managed installs, regulated buyers; higher spend per seat despite lower headcount share.
Third place: Bitbucket (~9% revenue) inside Atlassian/Jira shops.

For most teams starting today, GitHub is the path of least resistance: contributors already have accounts, and the Actions marketplace covers most CI without a second vendor.

3 Product split

Both are Git forges: remotes, PRs/MRs, review, CI, permissions. The split is how much of the delivery pipeline each ships in the box.

GitHub: Actions marketplace, Pages, Dependabot, Copilot on-host. Wins on ecosystem depth and the assumption that the repo lives on GitHub.
GitLab: CI, registry, scanning, environments in one product; GitLab Duo in-UI. Wins on single-vendor DevOps and self-hosting.

Pick GitHub when open-source reach and third-party Actions matter. Pick GitLab when you want the full stack on your infrastructure without stitching point tools.

4 AI-native platforms

None of these replaces Git overnight. They change where code is hosted, how fast agents can read it, and what gets versioned besides commits. Legacy forges were built for human-paced PRs; agent fleets generate read traffic and merge volume those systems were not designed for.

Cursor Origin (Aug 2026 beta): git forge inside Cursor — repos, PRs, code browse, GitHub sync/mirror; paid plans. Agent-scale hosting inside the editor; GitHub remains system of record for mirrored repos.
Harness Code Repository (Aug 2026): SCM for agent-scale PR volume; paired AI Code Review (risk-ranked diffs); MCP + CLI; OPA gates; free tier; migrates from GitHub/GitLab.
Entire (Jul 2026 preview): Thomas Dohmke (ex-GitHub CEO); distributed Git network — regional mirrors (US/EU/AU) offload agent clone/pull traffic from GitHub; native repos on roadmap; session memory in-repo.
Zed DeltaDB (beta 2026): CRDT delta stream — versions every edit and linked agent conversation, not just commits; runs alongside Git; Delta client for multiplayer threads.

Git. Won. GitHub. Leads. GitLab. Integrated. Agents. Stress. Origin. Harness. Entire. Delta.

5 References

GitHub — Wikipedia (founding, Microsoft acquisition)
History of GitLab — GitLab Handbook
Code review tooling market — Jon Moshier (SCM share figures)
Cursor Origin documentation
Harness Agent-Ready Code Repository — blog
Entire distributed Git network — The New Stack
Introducing DeltaDB — Zed Blog

Uses of Tensor Factorizations

Ravi Kalia — Sat, 29 Aug 2026 05:00:00 GMT

Uses of Tensor Factorizations

The main application of tensor factorizations is compression: images and video, or the parameter tensors inside a neural net — a VGG conv kernel, a transformer dense map.

The algebra is in Tensor Factorizations and Tensor Inverses. A post on tensor inverses is coming.

NumPy for CP, Tucker, and truncated HOSVD is first. Then the two compressions, then an unmixing cube: flattening loses the dyes.

1 Tensor data

Film clip. Five seconds of the 1962 Lawrence of Arabia theatrical trailer (public domain, Wikimedia Commons). Colour, picture and sound. Cut at 00:00:55 (train explosion, then the cut that follows). Files: media/clip.mp4, still.png, frames.npy, clip.wav.
Conv kernel. Synthetic CP rank-16 weight plus noise, seed 7. Stands in for one VGG-16 conv5 layer. The 512-channel counts are closed-form, not a downloaded VGG.
Dense map. Synthetic TT-rank-4 matrix plus noise. Stands in for a transformer projection. The counts use the same formula.
Mixing cube. Synthetic table, three dyes plus noise: sample × emission colour × excitation colour. Seed 7. Stands in for a fluorescence excitation–emission stack.

Figures and the slider read widget-data/curves.json, written from the same seed.

2 Factorizations

CP, Tucker, and truncated HOSVD in NumPy. The algebra is in Tensor Factorizations and Tensor Inverses. Later sections call these same functions.

2.1 Toy cube

This section uses a synthetic array.

Rank-2 CP: two outer products of Gaussian bumps, plus i.i.d. Gaussian noise at times the clean scale, seed 7.
Stands in for a small 3-way table (sample × feature × condition). The size is so the factors fit on one screen.
Objective: recover the two components. Report .
A flattened SVD mixes the two bumps. An overspecified CP rank invents a third. Either error mis-assigns which condition drives which feature.
Tensor methods fit because Kruskal uniqueness (CP) and per-mode truncation (HOSVD / Tucker) are properties of this layout, not of any matrix obtained from it.

Code

NumPy to JAX: The Array That Learned New Tricks

Ravi Kalia — Fri, 28 Aug 2026 05:00:00 GMT

NumPy to JAX: The Array That Learned New Tricks

NumPy stores numbers in arrays whose operations run in compiled code. JAX keeps that array API and adds compilation, automatic differentiation, and accelerator backends.

1 History

Python had two earlier array libraries:

Numeric — fast on small arrays.
Numarray — features needed for large astronomical images.

The two APIs were incompatible. Travis Oliphant merged them and released NumPy 1.0 in 2006 (Guide to NumPy).

That common type is what pandas, SciPy, and scikit-learn exchange. Harris et al. (2020), “Array programming with NumPy”, traces the downstream impact.

2 NumPy primitives

Four constructors and operations cover most work:

np.array — Python values to an array.
np.arange — regular sequence.
.reshape() — change dimensions without changing values.
elementwise arithmetic — no Python loop.

Array metadata:

.shape — length of each axis.
.ndim — number of axes.
.size — total element count.
.dtype — storage type.

The new dimensions after reshape must still multiply to .size.

Indexing:

grid[1, 2] — one value.
grid[1, :] — one row.
grid[:, 2] — one column.

Vectorized math: state the array operation; NumPy runs it in compiled routines.

3 Reductions and broadcasting

Reductions collapse many values to summaries: .mean(), .sum(), .min(), .max().

On a matrix, axis picks the direction:

grid.sum(axis=0) — column totals.
grid.sum(axis=1) — row totals.

Broadcasting subtracts a scalar mean from every element.

The check should be near zero. Residual is floating-point rounding.

These values are synthetic: five numbers chosen for a centering demo, not measured.

4 JAX

JAX mirrors the NumPy API:

import jax.numpy as jnp

x = jnp.arange(12).reshape(3, 4)
y = jnp.sin(x) + x**2

Additions relative to NumPy:

GPU / TPU execution for compatible work.
jax.jit — compile a function for repeated runs.
jax.grad — return a function that computes derivatives.

Frostig, Johnson, and Leary (2018), “Compiling machine learning programs via high-level tracing”, describes the tracing used for compilation.

Characteristic pattern:

import jax
import jax.numpy as jnp

def loss(weight):
    return (weight - 3.0) ** 2

fast_loss = jax.jit(loss)
loss_slope = jax.grad(loss)

print(fast_loss(1.0))   # 4.0
print(loss_slope(1.0))  # -4.0

jaxlib has no Pyodide build, so JAX cannot run in this page. The live cell below uses a finite-difference slope on the same scalar loss.

jax.grad(loss)(weight) returns the exact slope without a finite step.

5 Constraints

JAX transformations assume pure functions: output depends only on inputs; no hidden mutation of outside state.

Arrays are immutable. Replace x[0] = 10 with:

x = x.at[0].set(10)

Those rules make compilation and differentiation well-defined.

6 Choice of library

Use NumPy for general analysis on CPU.
Use JAX when gradients, jit, or accelerators dominate the workload.

Learn NumPy first. Its array model is the shared grammar; JAX reuses it.

Learn. NumPy. First. JAX. Reuses. Its. Grammar. Adds. Gradients. JIT.

7 References

Oliphant, T. E. Guide to NumPy — Numeric/Numarray merger and NumPy 1.0 (2006).
Harris, C. R. et al. (2020). Array programming with NumPy. Nature.
Frostig, R., Johnson, M. J. & Leary, C. (2018). Compiling machine learning programs via high-level tracing. MLSys.

Tensor Factorizations and Tensor Inverses

Ravi Kalia — Fri, 28 Aug 2026 05:00:00 GMT

Tensor Factorizations and Tensor Inverses

A tensor is an -way array. The case is a matrix: rows and columns only.

A matrix factorization or inverse needs a matrix. The standard reduction is a mode- unfolding , whose columns are the mode- fibers. SVD of that unfolding is the Eckart–Young optimum for the unfolding, not for . The left singular vectors of mix modes through ; there is no reason those vectors should be Kruskal factors, Tucker factors, or t-product singular tubes.

Kruskal uniqueness, multilinear rank, and the t-product algebra are properties of the multiway layout. They do not survive flattening. The constructions below keep the modes separate; Matrix Factorizations as Optimization Problems is the case of the same questions.

1 Synthetic tensor

This post uses a synthetic array.

Provenance:

Generated as a rank-3 CP model plus i.i.d. Gaussian noise, seed fixed.
Each generating vector is a localized Gaussian bump, not measured data. The three outer products sit in different corners of the cube, so a factorization that recovers the components is visible as three blobs.
The array stands in for a 3-way table (subject feature condition) whose modes should stay separate. A real table of that shape is high-dimensional and not drawable as voxels.
The objective is to recover the known CP/Tucker structure and to invert four different products on this object, or on a square unfolding of it.
A flattened SVD attributes variance to a mixed mode; an overspecified CP rank invents a fourth blob. Either error mis-assigns which condition drives which feature.
Tensor methods fit because Kruskal uniqueness and mode-wise compression are properties of this layout, not of any matrix obtained from it.

Code

From Dataset to Biological Signature

Ravi Kalia — Thu, 27 Aug 2026 05:00:00 GMT

1 Biological signature

A biological signature is a fixed gene list with weights and a decision threshold, fit to answer a specific classification question. Every reported performance number is conditional on the data having selected those genes.

This post runs Golub et al. (1999) ALL/AML microarray data end to end: recover the original train/test split, fit an elastic-net signature without leakage, and report honest external validation. The pipeline is described twice — ML vocabulary and statistical vocabulary.

Primer without code: What is a Biological Signature?.

2 Biology background

Acute lymphoblastic leukemia (ALL) and acute myeloid leukemia (AML) require opposite treatments. Pre-1999 diagnosis used morphology, cytochemistry, and surface markers.

Bone marrow stem cell gives two lines:

flowchart TD
  S["Bone marrow stem cell"] --> L["Lymphoid line
B-cells, T-cells"]
  S --> M["Myeloid line
red cells, platelets, granulocytes"]
  L -->|glitch| ALL["ALL
lymphoblasts"]
  M -->|glitch| AML["AML
myeloblasts"]

ALL: lymphoid line jammed; commonest childhood cancer.
AML: myeloid line jammed; adult disease; affects oxygen transport and clotting faster.

Diagnostic question: which line is stuck?

3 Data provenance

Source: Golub et al. (1999), Whitehead Institute / Dana-Farber. 72 acute leukemia patients; Affymetrix HU6800 array (7,129 gene probes). Archived hospital samples; goal was to show tumour type is legible in gene expression.

Matrix used here: Efron and Hastie’s CASI data page (leukemia_big.csv.gz), cached in data/.
Split key: original Whitehead files (golub_train.csv.gz, golub_independent.csv.gz) carry probe accessions and patient IDs the CASI copy dropped.
Objective: classify ALL vs AML from expression.
Downstream impact: treatment assignment (vincristine/steroids vs cytarabine/anthracycline). Target metric: performance at a different hospital, not peak training accuracy.

Imports, plot style, and loading the cached matrices