More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
You start by pulling 150 unlabeled posts from three 20-newsgroups categories—sci.space, sci.med and rec.autos—stripping out headers, footers and quotes so only raw text remains. After filtering out very short posts, you load the “all-MiniLM-L6-v2” model from sentence-transformers and encode each document into a 384-dimensional vector. Generating embeddings runs in seconds with a progress bar showing the workload.
Next you tackle dimensionality. High-dim vectors don’t work well for density clustering, so you apply UMAP with n_neighbors=15, n_components=5 and min_dist=0.0, shrinking each embedding down to five numbers. That step preserves local density patterns but makes the data easier to scan. The resulting shape goes from (150, 384) to (150, 5).
For grouping you pick HDBSCAN with a minimum cluster size of eight and min_samples=3. Feeding the reduced embeddings into HDBSCAN produces two clusters: one with 101 docs and the other with 49. No points end up labeled as noise. You then peek inside each group by printing three sample texts per cluster. Topic 0 mixes space-science and medical discussion (“blurring the distinction between real disease caused by Candida albicans…”), while Topic 1 clearly centers on auto performance (“diamond star cars…put out 190 hp in the turbo models…”).
The article points out that tweaking HDBSCAN’s hyperparameters—cluster size, sample threshold—can change how many groups you see. For a visual sanity check, you can plot all pairwise combinations of the five UMAP dimensions using matplotlib and seaborn. That scatterplot matrix helps you spot dense regions that HDBSCAN picked up and verify whether clusters line up with your intuitive topic boundaries.
Questions about this article
No questions yet.