How-to

Supervised Machine Learning: Regression Algorithms with Python – A Practical How‑To Guide

NerdIT · 22 Nov 2026 · 4 min read

Supervised Machine Learning: Regression Algorithms with Python – A Practical How‑To Guide

Supervised machine learning regression predicts a continuous target variable from input features. This guide walks you through the core concepts, popular algorithms, and a hands‑on Python workflow you can start today. By the end you’ll have a working model and know how to iterate for better results.

What is regression?

Regression is a type of supervised learning where the goal is to estimate a numeric value — like house price, sales forecast, or temperature — based on historical data. Unlike classification, which outputs a category, regression outputs a real number. The training data consists of feature columns (independent variables) and a target column (dependent variable). The model learns the relationship so it can predict the target for new, unseen feature sets.

Common regression algorithms

Choosing the right algorithm depends on data size, linearity, and interpretability needs. Here are the most widely used ones:

Setting up your Python environment

  1. Install Anaconda or Miniconda – gives you a clean conda environment.
  2. Create a new env: conda create -n reg-ml python=3.11 and activate it.
  3. Install core packages:
    conda install numpy pandas scikit-learn matplotlib seaborn jupyterlab
    
  4. (Optional) For boosting libraries:
    pip install xgboost lightgbm catboost
    
  5. Launch Jupyter: jupyter lab and start a new notebook.

Building a simple linear regression model

Below is a minimal end‑to‑end example using the California housing dataset (built‑in to scikit‑learn).

import pandas as pd
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Load data
data = fetch_california_housing(as_frame=True)
df = data.frame
X = df.drop('MedHouseVal', axis=1)
y = df['MedHouseVal']

# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train
model = LinearRegression()
model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)

# Evaluate
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f'MSE: {mse:.4f}, R²: {r2:.4f}')

Run the cell; you’ll see a baseline MSE and R². This is your starting point.

Evaluating model performance

Beyond MSE and R², consider:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_squared_error')
print('CV MSE:', -scores.mean())

If residuals show curvature, a linear model is insufficient — move to non‑linear algorithms.

Improving your model

  1. Feature engineering – create interaction terms, log‑transform skewed features, or add polynomial features (PolynomialFeatures).
  2. Regularisation – switch to Ridge (alpha=1.0) or Lasso (alpha=0.01) to shrink coefficients.
  3. Tree‑based models – try RandomForestRegressor(n_estimators=300, random_state=42) or XGBRegressor.
  4. Hyperparameter tuning – use GridSearchCV or RandomizedSearchCV on key parameters (e.g., max_depth, learning_rate).
  5. Ensemble stacking – combine predictions from multiple models for a final meta‑learner.

Each step should be validated with cross‑validation to avoid over‑fitting.

Next steps

Keep iterating, document your experiments, and build a portfolio notebook for each project. Consistent practice turns theory into job‑ready skills.

Browse all free courses