This guide shows how to turn raw documents into semantic embeddings using a pre-trained sentence-transformers model, reduce their dimensionality with UMAP, and then apply HDBSCAN to uncover topic clusters without labels. It walks through data loading, embedding generation, dimensionality reduction, clustering, and sampling of cluster contents in Python.

The article shows how packing specialized, retrieved knowledge into smaller AI agents can match or beat huge frontier models. It explains a structure—raw source extractions, concept entries, theses and a startup primer—plus embedding‐powered retrieval to feed just the right context at query time.

Pingkit is a toolkit designed for training reproducible, capacity-aware models using transformer activations. It offers features for extracting embeddings, training neural architectures, and creating custom probes tailored to specific research needs. The toolkit is integrated with Hugging Face models and provides various utilities for data processing and model training.