Linear Regression

Linear regression is the simplest and most interpretable ML algorithm. It assumes a linear relationship between input features and the target variable.

The Model

ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

ŷ = w₁x₁ + w₂x₂ + ... + wₙxₙ + b

Where:

• ŷ = predicted value
• w = weights (coefficients) — learned from data
• x = input features
• b = bias (intercept)

Cost Function: Mean Squared Error

We want to minimise the difference between predictions and actual values:

MSE = (1/n) × Σ(yᵢ - ŷᵢ)²

MSE = (1/n) × Σ(yᵢ - ŷᵢ)²

How Learning Works: Gradient Descent

1. Start with random weights
2. Calculate the error (MSE)
3. Compute gradient (direction of steepest increase)
4. Update weights in the opposite direction
5. Repeat until convergence

w = w - α × ∂MSE/∂w

w = w - α × ∂MSE/∂w

Where α (alpha) is the learning rate — how big each step is.

Assumptions of Linear Regression

1. Linearity — relationship between X and y is linear
2. Independence — observations are independent
3. Homoscedasticity — constant variance of residuals
4. Normality — residuals are normally distributed

Evaluation Metrics

Multiple vs Simple Linear Regression

• Simple: one feature (y = wx + b)
• Multiple: many features (y = w₁x₁ + w₂x₂ + ... + b)
• Polynomial: adds x², x³ terms for non-linear patterns