Classification, regression, clustering, dimensionality reduction, anomaly detection

A researcher opens a panel of daily prices for four hundred stocks and asks a reasonable question: will this one go up tomorrow? That is one question among several a model can be asked, and picking the wrong family to ask it of is a more common failure than picking the wrong hyperparameters inside the right one. Before any estimator gets fitted, the useful exercise is naming the question precisely enough that the answer can be checked.

Machine learning in a research workspace splits into five families: classificationClassification A supervised learning task that assigns each observation to one of a fixed set of categories, such as up or down., regressionRegression A supervised learning task that predicts a continuous quantity, such as next-period return, rather than a category., clusteringClustering An unsupervised technique that groups observations by similarity without being told the correct groups in advance., dimensionality reductionDimensionality reduction Techniques that compress many correlated variables into fewer components while retaining as much of the original variation as possible., and anomaly detectionAnomaly detection Identifying observations that differ enough from the bulk of the data to be flagged as unusual, without a labelled definition of 'anomalous' to learn from.. The first two are supervised — they learn from a labelled outcome, a return or an up/down flag that already happened. The last three are unsupervised — they find structure in the data with no label to check against. That distinction changes what a result can mean, not just which scikit-learn class gets imported.

Classification: will it go up or down

Classification assigns each observation to one of a fixed set of categories. In finance the categories are usually binary: did the next-period return exceed a threshold, or not. The model does not output a return; it outputs a probability, and the workspace’s ML Models page turns that probability into a label using a threshold the researcher sets, not one baked into the estimator.

Fourteen classifiers are available, spanning simple and complex decision boundaries. Logistic Regression fits a linear boundary in the feature space and outputs a genuinely calibrated probability by construction — its output can be read as “the model believes this outcome has a 61% chance,” and that number tends to mean what it says. Random Forest and XGBoost Classifier build ensembles of decision trees and typically win on raw discriminative power, at the cost of needing a probability recalibration step if the reported number is going to be trusted as a probability rather than just a ranking. Gradient Boosting Classifier and Hist Gradient Boosting Classifier occupy similar territory with different training mechanics. AdaBoost Classifier reweights misclassified rows each round. SVC draws a maximum-margin boundary and, notably, needs its probability outputs generated by an internal randomised calibration pass rather than reading them directly off the decision function. KNN Classifier, Gaussian NB, LDA, QDA, Decision Tree Classifier and MLP Classifier round out the roster, each with a different assumption about how the classes are separated.

The metric that matters most for a classifier evaluated on financial data is ROC AUCROC AUC The area under the receiver operating characteristic curve; the probability a classifier ranks a random positive case above a random negative one., because financial class labels are rarely balanced — up-days and down-days are close to a coin flip in aggregate, but a five-day-ahead “large move” label might be 8% positive. Accuracy on an imbalanced label is close to meaningless; a model that always predicts “no large move” scores 92% accuracy and zero usefulness. ROC AUC asks a sharper question: across all possible thresholds, how well does the model rank positives above negatives? A value of 0.50 is a coin flip; 1.00 is perfect separation; anything reliably above roughly 0.55 on genuinely out-of-sample financial data is worth taking seriously, and anything above 0.65 deserves scrutiny for leakage before celebration.

Regression: how much, not just which direction

Regression predicts a continuous number — next period’s return, next period’s volatility, or a security’s fair-value spread from a peer group. Sixteen regressors are available in the workspace, more than classifiers, because return-level prediction is where a wider variety of loss functions and regularisation choices earns its place.

Linear Regression is the reference point every other regressor has to beat, imposing no regularisation at all. Ridge shrinks coefficients smoothly by penalising their squared magnitude; Lasso does the same with the absolute value and can set weak coefficients to exactly zero, which functions as automatic feature selection. Elastic Net blends the two penalties. Huber Regressor swaps the squared-error loss for one that grows linearly beyond a threshold, which matters specifically in finance: a model trained on daily returns with an ordinary least-squares loss lets a handful of crash days dominate the fit, because squaring a −12% day counts it 144 times as heavily as a −1% day. A Huber loss caps that influence without discarding the observation. Bayesian Ridge places a prior over the coefficients and returns a predictive variance alongside the point estimate. Decision Tree Regressor, Random ForestRandom forest An ensemble of decision trees trained on bootstrapped samples and averaged, robust to feature scaling and mixed variable types. Regressor, Extra Trees Regressor, Gradient BoostingGradient boosting An ensemble method that fits trees sequentially, each correcting the errors of the ones before it; usually the strongest baseline on tabular financial data. Regressor, Hist Gradient Boosting Regressor, XGBoost Regressor, SVR, KNN Regressor, AdaBoost Regressor and MLP Regressor cover the tree-based and non-linear end of the spectrum.

A worked example makes the loss-function choice concrete. Suppose five daily returns in a training window are 0.4%, −0.6%, 0.9%, −0.3% and −8.0% (a crash day). An ordinary least-squares fit minimises the sum of squared errors: the crash day contributes $(-8.0)^2 = 64.0$ to that sum before the other four days contribute a combined $0.16 + 0.36 + 0.81 + 0.09 = 1.42$. The crash day accounts for roughly 98% of the total squared error the model is trying to reduce, so the fitted line bends hard to accommodate one observation. A Huber loss with a modest threshold treats errors past that threshold linearly rather than quadratically, so the same crash day contributes proportionally to its size rather than to its square — the fit stays representative of the other four days instead of being dragged toward the outlier.

Regression on financial data is graded on R² and RMSE, and — because a positive-return-versus-negative-return call is often what a strategy actually consumes downstream — directional accuracy: the share of predictions that get the sign right, independent of magnitude. A model can have unremarkable R² and still be strategically useful if it gets direction right consistently more than half the time on true out-of-sample folds, which is the subject explored in depth in forecasting versus prediction.

Clustering: grouping without labels

Clustering has no outcome to predict at all — it groups observations by similarity in feature space, and the “similarity” it uses is whatever the features encode. Six clustering algorithms are available. K-Means partitions securities into a fixed number of groups by minimising distance to each group’s centroid, and needs that number chosen in advance. Hierarchical clustering builds nested groups and lets the researcher cut the resulting tree at any depth after the fact, trading the up-front k for a different judgement call. DBSCAN groups by density and — usefully for messy financial data — leaves genuine outliers unassigned rather than forcing every point into some cluster. Birch scales to larger universes by building a compact summary tree first. Spectral clustering works in the eigenspace of a similarity graph, which suits clusters that are not simply shaped in the original feature space. Gaussian Mixture assigns each point a probability of membership in each cluster rather than a hard assignment, useful when a security genuinely sits between two regimes rather than firmly inside one.

A common use in finance is clustering securities by return-history distance to build a sector-agnostic peer group, or clustering trading days by realised volatility and correlation structure to find regime boundaries the calendar does not respect. There is no ROC AUC or R² to report here — cluster quality is judged by internal separation statistics and, ultimately, by whether the grouping is stable and useful for the downstream question, not by a held-out accuracy figure.

Dimensionality reduction: compressing without losing the point

A universe of five hundred securities’ daily returns is a five-hundred-column feature space, and most of those columns move together. Dimensionality reduction finds a smaller number of components that capture most of the variation. PCA (Principal Component Analysis) finds orthogonal linear components ranked by variance explained — the first component in an equity universe is very often close to “the market,” and the amount of variance it alone captures is a genuine number worth reporting rather than assuming. Kernel PCA applies the same idea after a non-linear feature-space transform, useful when the true structure is not linear. Fast ICA looks for statistically independent components rather than merely uncorrelated ones, which suits separating genuinely distinct return-generating factors. t-SNE is built for visualisation rather than downstream modelling — it preserves local neighbourhood structure well enough to plot but distorts global distances, so a distance measured on a t-SNE plot is not a real distance.

Anomaly detection: finding what does not belong

Anomaly detection asks a narrower question than clustering: not “what groups exist,” but “which points do not belong to any of them.” Three methods are available. Isolation Forest isolates points by randomly partitioning the feature space and measuring how few splits it takes to isolate a given observation — genuine outliers isolate quickly because they sit far from the bulk of the data. Local Outlier Factor compares each point’s local density to its neighbours’, flagging points that sit in a sparser neighbourhood than their surroundings would predict, which catches local anomalies that a global method misses. One Class SVM learns a boundary around the dense region of “normal” data and flags anything outside it.

In a market-research context this maps onto identifying trading days with unusually decoupled cross-sectional behaviour, or securities whose recent return pattern has stopped resembling their own history — a signal worth investigating before deciding whether it is data quality, a genuine regime shift, or a corporate action the dataset has not yet absorbed.

Choosing a family starts with the question, not the data

The single most useful discipline before opening the ML Models page is writing the question in one sentence and checking which family actually answers it. “Will this go up” is classification. “How much will this move” is regression. “Which securities behave alike” is clustering. “Can I represent this cheaply” is dimensionality reduction. “What looks wrong here” is anomaly detection. Fitting a regressor to a question that is really about grouping, or a classifier to a question that is really about magnitude, produces a number — every estimator produces a number — but not an answer to the question actually being asked. The mechanics of choosing between the two dominant supervised styles, linear and tree-based, are covered in choosing between linear and tree-based models.