Little Bits of Artificial Intelligence

1. What is Ridge Regression (L2 Regularization)

Ridge Regression is an improved version of Linear Regression that helps when our model is too complex or when our data has too many features (variables). It works by adding a penalty to the model so that it doesn’t overfit the data.

The Core Idea:

• Regular linear regression tries to fit the best line through the data.
• But if we have too many variables, the model might memorize the data — which is bad!
• So, Ridge Regression adds a small penalty for having big values in the model’s coefficients.That penalty is based on the sum of the squares of the coefficients:

Loss=MSE+λ∑w^2

Where:

• MSE is Mean Squared Error (difference between actual and predicted values)
• w are the model’s weights (or coefficients)
• λ (lambda) is a parameter that controls how strong the penalty is

In Very Simple Terms:

Imagine we’re fitting a curve to data points using rubber bands. Without Ridge, the bands stretch too much to touch every point (overfitting). Ridge Regression adds some tension (penalty), pulling the bands back to a smoother shape.

2. Two Real-Life Examples:

1. Predicting House Prices

• We have features like: number of rooms, age of the building, distance to city center, school rating, crime rate, etc.
• Many of these variables are related, and using all of them might make the model overfit.
• Ridge regression helps keep the model general by reducing the impact of variables that don’t matter much.

2. Stock Market Prediction

• We’re using historical prices, moving averages, volumes, RSI, MACD, economic indicators, etc.
• Too many features can cause overfitting to past patterns.
• Ridge regression keeps the model stable and avoids learning noise from irrelevant or highly correlated indicators.

3. Understand the Classification Type from Data:

Step 1: Identify the Target Column (Label)

This is the column we’re trying to predict (also called the dependent variable or output).

Examples:

• Diagnosis (cancerous or not)
• Loan_Status (approved or not)
• Animal_Type (dog, cat, horse…)

Step 2: Check the Number of Unique Values in the Target Column

We can do this with basic Python:

import pandas as pd

df = pd.read_csv('your_data.csv')
print(df['target_column'].value_counts())
print("Unique classes:", df['target_column'].nunique())

Step 3: Interpret the Output

4. Real-Life Examples

5. Checklist to Know if Our Task is Classification

6. What is Multicollinearity?

Multicollinearity happens when two or more input features (independent variables) are highly correlated — meaning, they move together and carry similar information.

In simple terms: If one column can predict another column, then both may be giving the model redundant signals.

7. How to Detect Multicollinearity in a Dataset (Step-by-Step)

Step 1: Visual Inspection

Look at similar-sounding columns like:

• Height and Arm Span
• House Age and Years Since Renovation
• Total Rooms and Bedrooms

Why? Some columns might just be “renamed” versions of others or closely related.

Step 2: Correlation Matrix

Use Pearson correlation to find how strongly two features move together.

import pandas as pd

df = pd.read_csv("your_data.csv")
correlation_matrix = df.corr()
print(correlation_matrix)

Interpretation:

• Values close to +1 or -1 mean strong correlation
• Especially watch for values > 0.8 or < -0.8

Why? If Feature A and Feature B are 0.95 correlated, the model can’t tell which is actually causing the effect — this leads to unstable coefficients.

Step 3: Heatmap for Visual Aid (optional)

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(df.corr(), annot=True, cmap="coolwarm")
plt.show()

Step 4: Variance Inflation Factor (VIF) – Quantitative Check

This checks how much a variable is inflated due to correlation with other variables.

from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools.tools import add_constant

X = df.drop("target_column", axis=1)
X = add_constant(X)
vif_data = pd.DataFrame()
vif_data["Feature"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)

VIF Interpretation:

Real-Life Example

We’re predicting house price with:

• TotalSquareFeet
• LivingArea
• GarageArea

We notice:

• TotalSquareFeet and LivingArea have 0.92 correlation
• VIF for LivingArea is 12

Conclusion: Drop one of them or use Ridge Regression to reduce the impact of multicollinearity.

8. What Is “Interpretability”?

Interpretability refers to how easily a human can understand the reasoning behind a model’s predictions.It answers: “Why did the model make this decision?”

What Does It Mean When “Interpretability Is Not a Top Priority”?

It means we’re okay not fully understanding how much each input (feature) affects the output, as long as the model performs well

Example: Ridge Regression.

Ridge regression shrinks the coefficients of your features — so while it reduces overfitting, it also muddies the meaning of individual feature importance.

• In simple linear regression, if we see:
  

  Price=50×Area+10×Bedrooms

  → We can clearly say “Area” has 5× more effect than Bedrooms.

• But in Ridge regression, we might get:
  

  Price=12.4×Area+9.7×Bedrooms

  → Coefficients are shrunk to prevent overfitting, so it’s hard to trust individual importance.

Ask These Questions:

Real-Life Examples

9. What is Mean Squared Error (MSE)?

Mean Squared Error (MSE) is a way to measure how wrong our model’s predictions are — it’s a number that tells us how far off our predictions are from the actual values.

Formula:

        n
MSE=1/n ∑ (yi​−y^​i​)2
       i-1

• yi: the actual value
• y^i​: the predicted value
• n: number of data points

It means: Take each prediction, subtract the actual value, square the result, and average over all predictions.

Why Square the Errors?

• Avoids negative cancelation: Positive and negative errors don’t cancel each other out.
• Punishes larger mistakes more: A big error (say, 10) gets squared to 100. This ensures the model cares more about big mistakes.

Why is MSE used in Ridge Regression?

In Ridge Regression, the goal is to minimize the total error, but with control over the model’s complexity.

So the full Ridge Loss Function is:

Loss=MSE+λ∑w2

This means:

• MSE ensures the predictions are close to actual values.
• L2 penalty (λ∑w2) keeps the model from becoming too complex or overfitting.

Analogy:

Think of MSE as measuring how far our darts land from the bullseye. Ridge Regression says: “I want to hit the target (low MSE), but I also don’t want to throw wild, exaggerated darts (large coefficients).”

10. What Does λ (Lambda) Really Do?

Lambda (λ) controls the strength of the penalty on large model coefficients (weights).

• If λ = 0 → No penalty → Just normal Linear Regression
• If λ is very large → High penalty → Forces the weights to be small, maybe too small → Could lead to underfitting
• If λ is chosen carefully → We get a good balance between accuracy and simplicity

What Happens When We Tune λ?

Case 1: λ = 0 (No Penalty)

• Our model learns weights freely.
• It overfits the training data — it picks up noise and patterns that don’t generalize.
• Predicts well on training data, poorly on new/unseen data.

Case 2: λ = 10

• It penalizes large weights.
• Coefficients for unimportant features shrink.
• Helps the model focus on the most useful variables.
• Improves generalization.

Case 3: λ = 1000 (Too Much Penalty)

• Shrinks almost all weights to near zero.
• Our model becomes too simple, even ignoring important features.
• Results in underfitting.