Thoughts on Unsupervised Learning

machine-learning
development
statistics
unsupervised-learning
clustering
dimensionality-reduction
Author

Ravi Kalia

Published

January 8, 2026

Made with ❤️ and Cursor
Cursor

1 Thoughts on Unsupervised Learning

The unsupervised learning methods arrange into one family tree, with compression as the trait they all share.

1.1 Unsupervised learning

Unsupervised learning (UL) discovers structure in data \(X\) without labels. Supervised learning maps inputs to targets; UL asks what shape the data has.

Two foundations:

  • Distance — compare examples (Euclidean distance, cosine similarity). Close points likely share properties.
  • Probability — model the data-generating distribution \(P(X)\).

1.2 Taxonomy

Algorithms grouped by how they manipulate feature space and probability density.

1.2.1 Feature manipulation

Change data representation.

  • Dimensionality reduction (compression) — lower-dimensional representation preserving structure.
    • Linear: PCA
    • Non-linear (manifold learning): autoencoders, t-SNE, UMAP
  • Dimensionality expansion (projection) — project to higher dimensions for separability or richness.
    • Random projections
    • Kernel methods (SVMs, RBFs)
    • Feature maps (CNN internal expansions)

1.2.2 Clustering

Group points by distance metrics.

  • Examples: K-Means, DBSCAN, hierarchical clustering

1.2.3 Probability density modeling

Learn the function describing data generation.

  • Explicit density — compute likelihood of a point (e.g., Gaussian mixture models)
  • Implicit density — sample without tractable likelihood (e.g., VAEs, GANs)

1.2.4 Self-supervised learning

Generate labels from the data itself (mask parts of input, predict them).

  • Examples: masked autoencoders (BERT), next-token prediction (GPT)

1.3 Generative AI

Generative AI is unsupervised learning at scale. Foundation models (GPT-4, Llama) model joint probability over token sequences via next-token prediction.

RLHF fine-tuning is supervised/RL; world knowledge comes from unsupervised pre-training.

1.4 Clustering as extreme reduction

Clustering is extreme, discretized dimensionality reduction.

Manifold learning maps input to a continuous latent space (e.g., 128 dimensions). Reduce to 1 dimension and discretize into \(K\) integer values → clustering.

A cluster assignment is a maximally compressed latent vector (integer in \(\{1,\ldots,K\}\)). The integer encodes group membership only; naming clusters turns them into classes. Most UL algorithms compress information into a coherent latent structure.

Clustering. Is. Compression. Reduction. Is. Compression. Labels. Never. Enter. It.