Hyperparameters that actually matter

A hyperparameterHyperparameter A setting chosen before training that controls how a model learns — a tree's depth, a regularisation strength — rather than a value the model fits from data. is a setting a researcher chooses before fitting a model, as distinct from a parameter the model learns from the data — a tree’s maximum depth is set in advance; the specific split thresholds it ends up using are learned. Every one of the workspace’s 43 machine learning models ships with sensible defaults, and a large share of research questions can be answered without touching them at all. But some hyperparameters are the difference between a model that generalises and one that has simply memorised its training window, and knowing which few actually matter — for a given family — is worth more than exhaustively searching all of them.

Regularisation strength: the parameter with the clearest story

For Ridge, Lasso and Elastic Net, the single most consequential hyperparameter is the regularisation strength, usually written $\alpha$ or $\lambda$:

$\min_\beta \; \sum_i (y_i - \hat{y}_i)^2 + \lambda \sum_j \beta_j^2$

At $\lambda = 0$ the model reduces to ordinary Linear Regression, free to fit every quirk of the training data. As $\lambda$ grows, coefficients shrink toward zero, trading training-fold fit for stability out of sample. The relationship between $\lambda$ and generalisation is not monotonic in the metric that matters — training error rises steadily as $\lambda$ increases, while test error typically falls, reaches a minimum, and then rises again once the shrinkage is aggressive enough to erase real signal along with the noise. A grid search across $\lambda$ values, scored on walk-forward folds rather than the training fit, is how that minimum gets located rather than guessed at.

Tree depth and leaf size: the parameters that control memorisation

For every tree-based estimator — Decision Tree Classifier/Regressor, Random ForestRandom forest An ensemble of decision trees trained on bootstrapped samples and averaged, robust to feature scaling and mixed variable types., Extra Trees, Gradient BoostingGradient boosting An ensemble method that fits trees sequentially, each correcting the errors of the ones before it; usually the strongest baseline on tabular financial data., Hist Gradient Boosting, XGBoost and AdaBoost’s base trees — the parameters that matter most are maximum tree depth and minimum samples per leaf, because these directly bound how finely the model can carve up the training data.

An unconstrained tree can, in principle, keep splitting until every leaf contains a single training row, at which point the tree has achieved perfect training accuracy by memorising the dataset rather than by learning anything transferable — the textbook definition of overfittingOverfitting A model or strategy that fits the noise in its training sample rather than a pattern that generalises, producing results that do not repeat out of sample.. Consider a training set of two thousand rows: a tree with max_depth = 3 has at most eight leaves and is forced to describe the data with eight broad rules, which limits how much noise it can fit but may also miss genuine finer structure. A tree with max_depth = 20 can, in the worst case, create up to $2^{20}$ leaves — vastly more than there are training rows — meaning most leaves would contain a single observation or none at all, a model with essentially zero bias and enormous variance.

For ensembles, the number of trees (n_estimators) interacts with depth differently depending on whether the ensemble is a Random Forest or a boosted model. Adding more independent trees to a Random Forest reduces variance and essentially never increases overfitting on its own, because averaging more independent, noisy estimates only sharpens the average. Adding more sequential trees to a boosted model is different: each additional tree fits the residual the previous ones left behind, so past a certain point additional trees begin fitting residual noise rather than residual signal, and the learning rate — the shrinkage applied to each new tree’s contribution — is what controls how quickly that happens.

$F_m(x) = F_{m-1}(x) + \eta \cdot h_m(x)$

A small learning rate $\eta$ (commonly 0.01–0.1) combined with a larger number of boosting rounds tends to generalise better than a large learning rate with few rounds, because each individual tree is prevented from overcorrecting to the specific residuals of the current training sample. This is the pairing to search jointly rather than independently — a learning-rate search that holds n_estimators fixed at its default will systematically under- or over-fit depending on which default happened to be chosen.

Neighbourhood and kernel parameters: the ones with a direct geometric meaning

KNN Classifier and KNN Regressor have one dominant hyperparameter, the number of neighbours $k$. A small $k$ (say, 3) makes predictions highly sensitive to individual nearby points — noisy, low-bias, high-variance. A large $k$ (say, 200 on a training set of 2,000 rows) averages over such a wide neighbourhood that the prediction approaches the global mean regardless of the query point’s actual location — smooth, high-bias, low-variance. SVC and SVR carry an analogous pair: the regularisation parameter $C$, which trades margin width against training-error tolerance, and the kernel’s own scale parameter (for an RBF kernel, $\gamma$), which controls how far a single training point’s influence reaches. A large $\gamma$ makes the decision boundary hug individual training points tightly; a small $\gamma$ makes it smooth and close to linear.

Clustering and dimensionality reduction: parameters that change what exists, not just how well it fits

K-Means and Gaussian Mixture both require the number of clusters $k$ chosen in advance, and unlike a supervised model’s hyperparameters, there is no held-out label to score $k$ against — the standard approach is to examine how within-cluster variance falls as $k$ increases and look for the point of diminishing returns, or to use a stability criterion across repeated runs. DBSCAN replaces $k$ with two different parameters — a neighbourhood radius ($\varepsilon$) and a minimum point count — and its behaviour is genuinely qualitative rather than a smooth trade-off: too small an $\varepsilon$ classifies almost everything as noise; too large merges clusters that should stay distinct. PCA’s equivalent decision is how many components to retain, typically chosen by the cumulative variance explained rather than searched against a downstream metric, since the components themselves have no notion of a labelled outcome to optimise against.

Search versus generalisation: what a grid search is actually testing

A hyperparameter grid search is not a search for the configuration that fits the training data best — that search always converges toward the least-regularised, most-flexible corner of the grid, for the same reason an unconstrained tree always achieves perfect training accuracy. It is a search for the configuration that generalises best, which means every candidate configuration in the grid must be scored on genuinely held-out folds, not on the data it was fit to. The workspace’s Model Tuning stage runs exactly this: search one estimator’s grid, evaluate each candidate on the same walk-forward folds used everywhere else in the pipeline, and report whether the winning configuration’s out-of-sample performance is stable across folds or concentrated in one lucky one.

A worked illustration: suppose a Random Forest Classifier’s max_depth is searched over {3, 5, 10, 20, None} on five walk-forward folds. If mean out-of-sample ROC AUC rises from 0.54 at depth 3 to 0.58 at depth 10 and then falls back to 0.55 at depth 20 and 0.53 at unlimited depth, the shape itself is the finding — it says the data supports roughly ten levels of interaction before the tree begins fitting fold-specific noise, and that conclusion is more durable than the single number 0.58 in isolation. If instead the standard deviation across folds at depth 10 is nearly as large as the mean improvement over depth 3, the improvement is not reliably distinguishable from fold-to-fold noise, regardless of which configuration technically scored highest on average.

The parameters worth searching versus the parameters worth accepting

Not every hyperparameter deserves a grid slot. Regularisation strength, tree depth, minimum leaf size, learning rate, number of neighbours and number of clusters change the bias-variance trade-off directly and are worth searching deliberately. Many of the remaining settings — a specific random seed, a solver algorithm, a convergence tolerance — affect reproducibility and computation time far more than they affect what the model actually learns, and searching them adds trial count to the ledger without adding insight. The discipline is the same one that governs model choice generally: search what changes the answer, record what was searched, and let the walk-forward fold spread — not the single best number — decide whether the search found something real.