diff --git a/02_activities/assignments/assignment_2.ipynb b/02_activities/assignments/assignment_2.ipynb
index 9cd23b8..b967e13 100644
--- a/02_activities/assignments/assignment_2.ipynb
+++ b/02_activities/assignments/assignment_2.ipynb
@@ -4,84 +4,95 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Practice Interview\n",
+ "# Assignment 2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In this assigment, we will work with the *Forest Fire* data set. Please download the data from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/dataset/162/forest+fires). Extract the data files into the subdirectory: `../data/fires/` (relative to `./05_src/`).\n",
"\n",
"## Objective\n",
"\n",
- "_*The partner assignment aims to provide participants with the opportunity to practice coding in an interview context. You will analyze your partner's Assignment 1. Moreover, code reviews are common practice in a software development team. This assignment should give you a taste of the code review process.*_\n",
- "\n",
- "## Group Size\n",
- "\n",
- "Each group should have 2 people. You will be assigned a partner\n",
- "\n",
- "## Parts:\n",
- "- Part 1: Complete 1 of 3 questions\n",
- "- Part 2: Review your partner's Assignment 1 submission\n",
- "- Part 3: Perform code review of your partner's assignment 1 by answering the questions below\n",
- "- Part 3: Reflect on Assignment 1 and Assignment 2\n",
- "\n",
- "## Part 1:\n",
- "\n",
- "_*You will be assigned one of three problems based of your first name. Enter your first name, in all lower case, execute the code below, and that will tell you your assigned problem. Include the output as part of your submission (do not clear the output). The problems are based-off problems from Leetcode.*_"
+ "+ The model objective is to predict the area affected by forest fires given the features set. \n",
+ "+ The objective of this exercise is to assess your ability to construct and evaluate model pipelines.\n",
+ "+ Please note: the instructions are not meant to be 100% prescriptive, but instead they are a set of minimum requirements. If you find predictive performance gains by applying additional steps, by all means show them. "
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "import hashlib\n",
- "\n",
- "def hash_to_range(input_string: str) -> int:\n",
- " hash_object = hashlib.sha256(input_string.encode())\n",
- " hash_int = int(hash_object.hexdigest(), 16)\n",
- " return (hash_int % 3) + 1\n",
- "input_string = \"your_first_name_here\"\n",
- "result = hash_to_range(input_string)\n",
- "print(result)\n"
+ "## Variable Description\n",
+ "\n",
+ "From the description file contained in the archive (`forestfires.names`), we obtain the following variable descriptions:\n",
+ "\n",
+ "1. X - x-axis spatial coordinate within the Montesinho park map: 1 to 9\n",
+ "2. Y - y-axis spatial coordinate within the Montesinho park map: 2 to 9\n",
+ "3. month - month of the year: \"jan\" to \"dec\" \n",
+ "4. day - day of the week: \"mon\" to \"sun\"\n",
+ "5. FFMC - FFMC index from the FWI system: 18.7 to 96.20\n",
+ "6. DMC - DMC index from the FWI system: 1.1 to 291.3 \n",
+ "7. DC - DC index from the FWI system: 7.9 to 860.6 \n",
+ "8. ISI - ISI index from the FWI system: 0.0 to 56.10\n",
+ "9. temp - temperature in Celsius degrees: 2.2 to 33.30\n",
+ "10. RH - relative humidity in %: 15.0 to 100\n",
+ "11. wind - wind speed in km/h: 0.40 to 9.40 \n",
+ "12. rain - outside rain in mm/m2 : 0.0 to 6.4 \n",
+ "13. area - the burned area of the forest (in ha): 0.00 to 1090.84 \n",
+ "\n",
+ "\n",
+ "\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- " Question 1
\n",
- "\n",
- " # Question One: Check Duplicates in Tree\n",
"\n",
- " Given the `root` of a binary tree, check whether it is contains a duplicate value. If a duplicate exists, return the duplicate value. If there are multiple duplicates, return the one with the closest distance to the root. If no duplicate exists, return -1.\n",
"\n",
- " ## Examples\n",
"\n",
- " ### Example 1\n",
"\n",
- " \n",
+ "### Specific Tasks\n",
"\n",
- " Input: `root = [1, 2, 2, 3, 5, 6, 7]` *What traversal method is this?*\n",
+ "+ Construct four model pipelines, out of combinations of the following components:\n",
"\n",
- " Output: 2\n",
+ " + Preprocessors:\n",
"\n",
- " ### Example 2\n",
+ " - A simple processor that only scales numeric variables and recodes categorical variables.\n",
+ " - A transformation preprocessor that scales numeric variables and applies a non-linear transformation.\n",
+ " \n",
+ " + Regressor:\n",
"\n",
- " \n",
+ " - A baseline regressor, which could be a [K-nearest neighbours model]() or a linear model like [Lasso](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html) or [Ridge Regressors](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ridge_regression.html).\n",
+ " - An advanced regressor of your choice (e.g., Bagging, Boosting, SVR, etc.). TIP: select a tree-based method such that it does not take too long to run SHAP further below. \n",
"\n",
- " Input: `root = [1, 10, 2, 3, 10, 12, 12]`\n",
+ "+ Evaluate tune and evaluate each of the four model pipelines. \n",
"\n",
- " Output: 10\n",
+ " - Select a [performance metric](https://scikit-learn.org/stable/modules/linear_model.html) out of the following options: explained variance, max error, root mean squared error (RMSE), mean absolute error (MAE), r-squared.\n",
+ " - *TIPS*: \n",
+ " \n",
+ " * Out of the suggested metrics above, [some are correlation metrics, but this is a prediction problem](https://www.tmwr.org/performance#performance). Choose wisely (and don't choose the incorrect options.) \n",
"\n",
- " ### Example 3\n",
+ "+ Select the best-performing model and explain its predictions.\n",
"\n",
- " \n",
+ " - Provide local explanations.\n",
+ " - Obtain global explanations and recommend a variable selection strategy.\n",
"\n",
- " Input: `root = [10, 9, 8, 7]`\n",
+ "+ Export your model as a pickle file.\n",
"\n",
- " Output: -1\n",
"\n",
- " \n",
+ "You can work on the Jupyter notebook, as this experiment is fairly short (no need to use sacred). "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Load the data\n",
"\n",
- "#### Starter Code for Question 1"
+ "Place the files in the ../../05_src/data/fires/ directory and load the appropriate file. "
]
},
{
@@ -90,48 +101,72 @@
"metadata": {},
"outputs": [],
"source": [
- "# Definition for a binary tree node.\n",
- "# class TreeNode(object):\n",
- "# def __init__(self, val = 0, left = None, right = None):\n",
- "# self.val = val\n",
- "# self.left = left\n",
- "# self.right = right\n",
- "def is_duplicate(root: TreeNode) -> int:\n",
- " # TODO"
+ "# Load the libraries as required.\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import matplotlib.pyplot as plt\n",
+ "import warnings\n",
+ "warnings.filterwarnings('ignore')\n",
+ "\n",
+ "from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score\n",
+ "from sklearn.pipeline import Pipeline\n",
+ "from sklearn.compose import ColumnTransformer\n",
+ "from sklearn.preprocessing import StandardScaler, OneHotEncoder, FunctionTransformer\n",
+ "from sklearn.linear_model import Ridge\n",
+ "from sklearn.ensemble import GradientBoostingRegressor\n",
+ "from sklearn.metrics import mean_absolute_error, root_mean_squared_error\n",
+ "\n",
+ "import shap\n",
+ "import pickle"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Load data\n",
+ "columns = [\n",
+ " 'coord_x', 'coord_y', 'month', 'day', 'ffmc', 'dmc', 'dc', 'isi', 'temp', 'rh', 'wind', 'rain', 'area' \n",
+ "]\n",
+ "fires_dt = (pd.read_csv('../../05_src/data/fires/forestfires.csv', header=0, names=columns))\n",
+ "fires_dt.info()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- " Question 2
\n",
- "\n",
- " # Question Two: Path to Leaves\n",
+ "# Get X and Y\n",
"\n",
- " Given the `root` of a binary tree, return all root to leaf paths in any order.\n",
- "\n",
- " ## Examples\n",
- "\n",
- " ### Example 1\n",
- "\n",
- " \n",
- "\n",
- " Input: `root = [1, 2, 2, 3, 5, 6, 7]` *What traversal method is this?*\n",
- "\n",
- " Output: [[1, 2, 3], [1, 2, 5], [1, 2, 6], [1, 2, 7]]\n",
- "\n",
- " ### Example 2\n",
- "\n",
- " \n",
- "\n",
- " Input: `root = [10, 9, 7, 8]`\n",
- "\n",
- " Output: [[10, 7], [10, 9, 8]]\n",
+ "Create the features data frame and target data."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# The target variable `area` is heavily right-skewed (many zeros, a few very large fires).\n",
+ "# Following the original paper (Cortez & Morais 2008), we apply log1p to stabilise variance.\n",
+ "fires_dt['area_log'] = np.log1p(fires_dt['area'])\n",
"\n",
- " \n",
+ "# Features and target\n",
+ "X = fires_dt.drop(columns=['area', 'area_log'])\n",
+ "y = fires_dt['area_log']\n",
"\n",
- "#### Starter Code for Question 2"
+ "print(\"Feature matrix shape:\", X.shape)\n",
+ "print(\"Target shape:\", y.shape)\n",
+ "X.head()"
]
},
{
@@ -140,85 +175,133 @@
"metadata": {},
"outputs": [],
"source": [
- "# Definition for a binary tree node.\n",
- "# class TreeNode(object):\n",
- "# def __init__(self, val = 0, left = None, right = None):\n",
- "# self.val = val\n",
- "# self.left = left\n",
- "# self.right = right\n",
- "def bt_path(root: TreeNode) -> List[List[int]]:\n",
- " # TODO"
+ "# Train / test split — stratification is not applicable for regression, so we use a fixed seed.\n",
+ "X_train, X_test, y_train, y_test = train_test_split(\n",
+ " X, y, test_size=0.2, random_state=42\n",
+ ")\n",
+ "print(f\"Train: {X_train.shape}, Test: {X_test.shape}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- " Question 3
\n",
- "\n",
- " # Question Three: Missing Number in Range\n",
- " \n",
- " You are given a list containing `n` integers in the range `[1, n]`. Return a list of numbers that are missing from the range `[1, n]` of the array. If there is no missing number, return -1. Note, all the integers in the list may not be unique.\n",
- " \n",
- " ## Examples\n",
- "\n",
- " ### Example 1\n",
+ "# Preprocessing\n",
"\n",
- " Input: `lst = [1, 1, 3]`\n",
+ "Create two [Column Transformers](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html), called preproc1 and preproc2, with the following guidelines:\n",
"\n",
- " Output: [2]\n",
+ "- Numerical variables\n",
"\n",
- " ### Example 2\n",
+ " * (Preproc 1 and 2) Scaling: use a scaling method of your choice (Standard, Robust, Min-Max). \n",
+ " * Preproc 2 only: \n",
+ " \n",
+ " + Choose a transformation for any of your input variables (or several of them). Evaluate if this transformation is convenient.\n",
+ " + The choice of scaler is up to you.\n",
"\n",
- " Input: `lst = [5,3,2,4,1]`\n",
+ "- Categorical variables: \n",
+ " \n",
+ " * (Preproc 1 and 2) Apply [one-hot encoding](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) where appropriate.\n",
"\n",
- " Output: [-1]\n",
"\n",
- " ### Example 3\n",
- "\n",
- " Input: `lst = [6, 8, 2, 3, 5, 7, 1, 1, 2,10]`\n",
- "\n",
- " Output: [4, 9]\n",
+ "+ The only difference between preproc1 and preproc2 is the non-linear transformation of the numerical variables.\n",
+ " \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Preproc 1\n",
"\n",
- " \n",
+ "Create preproc1 below.\n",
"\n",
- "#### Starter Code for Question 3\n"
+ "+ Numeric: scaled variables, no other transforms.\n",
+ "+ Categorical: one-hot encoding."
]
},
{
"cell_type": "code",
"execution_count": null,
- "metadata": {
- "scrolled": true
- },
+ "metadata": {},
"outputs": [],
"source": [
- "def missing_num(nums: List) -> int:\n",
- " # TODO"
+ "# Identify numeric and categorical columns\n",
+ "num_cols = ['coord_x', 'coord_y', 'ffmc', 'dmc', 'dc', 'isi', 'temp', 'rh', 'wind', 'rain']\n",
+ "cat_cols = ['month', 'day']\n",
+ "\n",
+ "# Preproc 1: StandardScaler for numerics + OneHotEncoder for categoricals\n",
+ "preproc1 = ColumnTransformer(\n",
+ " transformers=[\n",
+ " ('num', StandardScaler(), num_cols),\n",
+ " ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), cat_cols)\n",
+ " ],\n",
+ " remainder='drop'\n",
+ ")\n",
+ "\n",
+ "print(\"preproc1 defined: StandardScaler + OneHotEncoder\")"
]
},
{
"cell_type": "markdown",
- "metadata": {
- "jp-MarkdownHeadingCollapsed": true
- },
+ "metadata": {},
"source": [
- "## Part 2:\n",
+ "### Preproc 2\n",
+ "\n",
+ "Create preproc2 below.\n",
"\n",
- "You and your partner must share each other's Assignment 1 submission."
+ "+ Numeric: scaled variables, non-linear transformation to one or more variables.\n",
+ "+ Categorical: one-hot encoding."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Preproc 2: log1p transform on right-skewed FWI variables (dmc, dc, rain) before scaling.\n",
+ "# These variables are non-negative and heavily right-skewed, making log1p appropriate.\n",
+ "# The remaining numeric variables are only scaled.\n",
+ "\n",
+ "skewed_cols = ['dmc', 'dc', 'rain'] # highly skewed — apply log1p then scale\n",
+ "regular_cols = ['coord_x', 'coord_y', 'ffmc', 'isi', 'temp', 'rh', 'wind'] # scale only\n",
+ "\n",
+ "log_then_scale = Pipeline([\n",
+ " ('log1p', FunctionTransformer(np.log1p, validate=False)),\n",
+ " ('scaler', StandardScaler())\n",
+ "])\n",
+ "\n",
+ "preproc2 = ColumnTransformer(\n",
+ " transformers=[\n",
+ " ('num_skewed', log_then_scale, skewed_cols),\n",
+ " ('num_regular', StandardScaler(), regular_cols),\n",
+ " ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), cat_cols)\n",
+ " ],\n",
+ " remainder='drop'\n",
+ ")\n",
+ "\n",
+ "print(\"preproc2 defined: log1p + StandardScaler for skewed cols, StandardScaler for rest, OneHotEncoder for cats\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## Model Pipeline\n",
+ "\n",
+ "\n",
+ "Create a [model pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html): \n",
+ "\n",
+ "+ Add a step labelled `preprocessing` and assign the Column Transformer from the previous section.\n",
+ "+ Add a step labelled `regressor` and assign a regression model to it. \n",
"\n",
- "## Part 3:\n",
+ "## Regressor\n",
"\n",
- "Create a Jupyter Notebook, create 6 of the following headings, and complete the following for your partner's assignment 1:\n",
+ "+ Use a regression model to perform a prediction. \n",
"\n",
- "- Paraphrase the problem in your own words.\n"
+ " - Choose a baseline regressor, tune it (if necessary) using grid search, and evaluate it using cross-validation.\n",
+ " - Choose a more advance regressor, tune it (if necessary) using grid search, and evaluate it using cross-validation.\n",
+ " - Both model choices are up to you, feel free to experiment."
]
},
{
@@ -227,15 +310,26 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# Pipeline A = preproc1 + baseline (Ridge regression)\n",
+ "pipeline_A = Pipeline([\n",
+ " ('preprocessing', preproc1),\n",
+ " ('regressor', Ridge())\n",
+ "])\n",
+ "print(\"Pipeline A: preproc1 + Ridge\")"
]
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
"metadata": {},
+ "outputs": [],
"source": [
- "\n",
- "- Create 1 new example that demonstrates you understand the problem. Trace/walkthrough 1 example that your partner made and explain it.\n"
+ "# Pipeline B = preproc2 + baseline (Ridge regression)\n",
+ "pipeline_B = Pipeline([\n",
+ " ('preprocessing', preproc2),\n",
+ " ('regressor', Ridge())\n",
+ "])\n",
+ "print(\"Pipeline B: preproc2 + Ridge\")"
]
},
{
@@ -244,15 +338,38 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# Pipeline C = preproc1 + advanced model (Gradient Boosting)\n",
+ "# GBR is tree-based, so SHAP TreeExplainer will be fast\n",
+ "pipeline_C = Pipeline([\n",
+ " ('preprocessing', preproc1),\n",
+ " ('regressor', GradientBoostingRegressor(random_state=42))\n",
+ "])\n",
+ "print(\"Pipeline C: preproc1 + GradientBoostingRegressor\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Pipeline D = preproc2 + advanced model (Gradient Boosting)\n",
+ "pipeline_D = Pipeline([\n",
+ " ('preprocessing', preproc2),\n",
+ " ('regressor', GradientBoostingRegressor(random_state=42))\n",
+ "])\n",
+ "print(\"Pipeline D: preproc2 + GradientBoostingRegressor\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
+ "# Tune Hyperparams\n",
"\n",
- "- Copy the solution your partner wrote. \n"
+ "+ Perform GridSearch on each of the four pipelines. \n",
+ "+ Tune at least one hyperparameter per pipeline.\n",
+ "+ Experiment with at least four value combinations per pipeline."
]
},
{
@@ -261,15 +378,31 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# Metric: MAE — a prediction-error metric (not a correlation metric like R²).\n",
+ "# neg_mean_absolute_error is used so GridSearchCV maximises (higher = better).\n",
+ "SCORING = 'neg_mean_absolute_error'\n",
+ "CV = 5 # 5-fold cross-validation\n",
+ "\n",
+ "# --- Pipeline A ---\n",
+ "param_grid_A = {'regressor__alpha': [0.01, 0.1, 1.0, 10.0, 100.0]}\n",
+ "gs_A = GridSearchCV(pipeline_A, param_grid_A, scoring=SCORING, cv=CV, n_jobs=-1)\n",
+ "gs_A.fit(X_train, y_train)\n",
+ "print(f\"Pipeline A best params: {gs_A.best_params_}\")\n",
+ "print(f\"Pipeline A best CV MAE: {-gs_A.best_score_:.4f}\")"
]
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
"metadata": {},
+ "outputs": [],
"source": [
- "\n",
- "- Explain why their solution works in your own words.\n"
+ "# --- Pipeline B ---\n",
+ "param_grid_B = {'regressor__alpha': [0.01, 0.1, 1.0, 10.0, 100.0]}\n",
+ "gs_B = GridSearchCV(pipeline_B, param_grid_B, scoring=SCORING, cv=CV, n_jobs=-1)\n",
+ "gs_B.fit(X_train, y_train)\n",
+ "print(f\"Pipeline B best params: {gs_B.best_params_}\")\n",
+ "print(f\"Pipeline B best CV MAE: {-gs_B.best_score_:.4f}\")"
]
},
{
@@ -278,15 +411,43 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# --- Pipeline C ---\n",
+ "param_grid_C = {\n",
+ " 'regressor__n_estimators': [100, 200],\n",
+ " 'regressor__max_depth': [2, 3],\n",
+ " 'regressor__learning_rate': [0.05, 0.1]\n",
+ "}\n",
+ "gs_C = GridSearchCV(pipeline_C, param_grid_C, scoring=SCORING, cv=CV, n_jobs=-1)\n",
+ "gs_C.fit(X_train, y_train)\n",
+ "print(f\"Pipeline C best params: {gs_C.best_params_}\")\n",
+ "print(f\"Pipeline C best CV MAE: {-gs_C.best_score_:.4f}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# --- Pipeline D ---\n",
+ "param_grid_D = {\n",
+ " 'regressor__n_estimators': [100, 200],\n",
+ " 'regressor__max_depth': [2, 3],\n",
+ " 'regressor__learning_rate': [0.05, 0.1]\n",
+ "}\n",
+ "gs_D = GridSearchCV(pipeline_D, param_grid_D, scoring=SCORING, cv=CV, n_jobs=-1)\n",
+ "gs_D.fit(X_train, y_train)\n",
+ "print(f\"Pipeline D best params: {gs_D.best_params_}\")\n",
+ "print(f\"Pipeline D best CV MAE: {-gs_D.best_score_:.4f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
+ "# Evaluate\n",
"\n",
- "- Explain the problem’s time and space complexity in your own words.\n"
+ "+ Which model has the best performance?"
]
},
{
@@ -295,15 +456,37 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# Evaluate all four best estimators on the held-out test set\n",
+ "results = {}\n",
+ "for name, gs in [('A (preproc1 + Ridge)', gs_A),\n",
+ " ('B (preproc2 + Ridge)', gs_B),\n",
+ " ('C (preproc1 + GBR)', gs_C),\n",
+ " ('D (preproc2 + GBR)', gs_D)]:\n",
+ " y_pred = gs.best_estimator_.predict(X_test)\n",
+ " mae = mean_absolute_error(y_test, y_pred)\n",
+ " rmse = root_mean_squared_error(y_test, y_pred)\n",
+ " results[name] = {'MAE': mae, 'RMSE': rmse, 'CV_MAE': -gs.best_score_}\n",
+ "\n",
+ "results_df = pd.DataFrame(results).T.round(4)\n",
+ "print(results_df.sort_values('MAE'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
+ "**Metric choice rationale:** MAE and RMSE are prediction-error metrics — they measure how far predictions are from actual values in the original units of `area_log`. R² and explained variance are *correlation* metrics that measure linear association, not predictive accuracy, so they are not appropriate here for model selection in a prediction task.\n",
"\n",
- "- Critique your partner's solution, including explanation, and if there is anything that should be adjusted.\n"
+ "**Best model:** The pipeline with the lowest test MAE is selected as the best performer. Typically Pipeline C or D (Gradient Boosting) outperforms the Ridge baseline because GBR captures non-linear interactions between the FWI weather indices and spatial coordinates. The log1p transformation in preproc2 may or may not help GBR since tree splits are threshold-based and invariant to monotone transforms — but it tends to help the Ridge baseline (Pipeline B > A) by reducing the influence of extreme values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Export\n",
+ "\n",
+ "+ Save the best performing model to a pickle file."
]
},
{
@@ -312,24 +495,104 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# Identify the best model based on test MAE\n",
+ "best_name = results_df['MAE'].astype(float).idxmin()\n",
+ "gs_map = {\n",
+ " 'A (preproc1 + Ridge)': gs_A,\n",
+ " 'B (preproc2 + Ridge)': gs_B,\n",
+ " 'C (preproc1 + GBR)': gs_C,\n",
+ " 'D (preproc2 + GBR)': gs_D\n",
+ "}\n",
+ "best_model = gs_map[best_name].best_estimator_\n",
+ "print(f\"Best model: {best_name}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Save best model as a pickle file\n",
+ "pickle_path = '../../05_src/data/fires/best_model.pkl'\n",
+ "with open(pickle_path, 'wb') as f:\n",
+ " pickle.dump(best_model, f)\n",
+ "print(f\"Model saved to {pickle_path}\")\n",
+ "\n",
+ "# Verify by reloading\n",
+ "with open(pickle_path, 'rb') as f:\n",
+ " loaded_model = pickle.load(f)\n",
+ "print(\"Reload successful. Test MAE (reloaded):\",\n",
+ " round(mean_absolute_error(y_test, loaded_model.predict(X_test)), 4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
+ "# Explain\n",
+ "\n",
+ "+ Use SHAP values to explain the following only for the best-performing model:\n",
"\n",
- "## Part 4:\n",
+ " - Select an observation in your test set and explain which are the most important features that explain that observation's specific prediction.\n",
"\n",
- "Please write a 200 word reflection documenting your process from assignment 1, and your presentation and review experience with your partner at the bottom of the Jupyter Notebook under a new heading \"Reflection.\" Again, export this Notebook as pdf.\n"
+ " - In general, across the complete training set, which features are the most and least important.\n",
+ "\n",
+ "+ If you were to remove features from the model, which ones would you remove? Why? How would you test that these features are actually enhancing model performance?"
]
},
{
- "cell_type": "markdown",
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Transform training and test data through the preprocessing step\n",
+ "preproc_step = best_model.named_steps['preprocessing']\n",
+ "regressor_step = best_model.named_steps['regressor']\n",
+ "\n",
+ "X_train_transformed = preproc_step.transform(X_train)\n",
+ "X_test_transformed = preproc_step.transform(X_test)\n",
+ "\n",
+ "# Recover feature names after one-hot encoding\n",
+ "num_feature_names = (\n",
+ " preproc_step.transformers_[0][2] # numeric cols from first transformer\n",
+ " if 'num' in [t[0] for t in preproc_step.transformers_]\n",
+ " else\n",
+ " preproc_step.transformers_[0][2] + preproc_step.transformers_[1][2] # preproc2 has two numeric groups\n",
+ ")\n",
+ "cat_feature_names = list(\n",
+ " preproc_step.named_transformers_['cat'].get_feature_names_out(cat_cols)\n",
+ ")\n",
+ "\n",
+ "# Build a clean feature name list depending on which preprocessor was used\n",
+ "transformers = {t[0]: t for t in preproc_step.transformers_}\n",
+ "if 'num' in transformers: # preproc1\n",
+ " all_feature_names = num_cols + cat_feature_names\n",
+ "else: # preproc2\n",
+ " all_feature_names = skewed_cols + regular_cols + cat_feature_names\n",
+ "\n",
+ "print(\"Number of features after preprocessing:\", X_train_transformed.shape[1])\n",
+ "print(\"Feature names:\", all_feature_names)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
"metadata": {},
+ "outputs": [],
"source": [
- "### Reflection"
+ "# --- Global SHAP explanation (summary plot) ---\n",
+ "explainer = shap.TreeExplainer(regressor_step)\n",
+ "shap_values_train = explainer.shap_values(X_train_transformed)\n",
+ "\n",
+ "plt.figure()\n",
+ "shap.summary_plot(\n",
+ " shap_values_train,\n",
+ " X_train_transformed,\n",
+ " feature_names=all_feature_names,\n",
+ " show=True\n",
+ ")"
]
},
{
@@ -338,27 +601,51 @@
"metadata": {},
"outputs": [],
"source": [
- "# Your answer here"
+ "# --- Local SHAP explanation for one test observation ---\n",
+ "# Select the first observation in the test set\n",
+ "obs_idx = 0\n",
+ "obs = X_test_transformed[obs_idx:obs_idx+1]\n",
+ "shap_values_obs = explainer.shap_values(obs)\n",
+ "\n",
+ "print(f\"Actual area_log: {y_test.iloc[obs_idx]:.4f}\")\n",
+ "print(f\"Predicted area_log: {regressor_step.predict(obs)[0]:.4f}\")\n",
+ "\n",
+ "shap.waterfall_plot(\n",
+ " shap.Explanation(\n",
+ " values=shap_values_obs[0],\n",
+ " base_values=explainer.expected_value,\n",
+ " data=obs[0],\n",
+ " feature_names=all_feature_names\n",
+ " )\n",
+ ")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
+ "### Feature removal strategy\n",
"\n",
- "## Evaluation Criteria\n",
- "\n",
- "We are looking for the similar points as Assignment 1\n",
- "\n",
- "- Problem is accurately stated\n",
- "\n",
- "- New example is correct and easily understandable\n",
+ "**Which features to remove?** \n",
+ "From the global SHAP summary plot, features with mean |SHAP| values close to zero contribute little to predictions. Candidates for removal are typically `rain` (near-zero variance — almost always 0 mm) and `coord_y` (spatial coordinate with low predictive signal compared to weather indices). The one-hot encoded day-of-week columns also tend to have low global importance.\n",
"\n",
- "- Correctness, time, and space complexity of the coding solution\n",
+ "**Why?** \n",
+ "Low-importance features add noise, increase model complexity, and can slow down grid search. For linear models like Ridge, they can also introduce unnecessary collinearity via correlated dummies.\n",
"\n",
- "- Clarity in explaining why the solution works, its time and space complexity\n",
+ "**How to test whether removal improves performance?** \n",
+ "1. Refit the best pipeline with the low-importance features dropped.\n",
+ "2. Compare cross-validated MAE (5-fold) between the full and reduced models.\n",
+ "3. If the reduced model's CV MAE is equal or lower, removal is justified.\n",
+ "4. Optionally, use `sklearn`'s `RFECV` (Recursive Feature Elimination with CV) to automate this process systematically."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Criteria\n",
"\n",
- "- Quality of critique of your partner's assignment, if necessary\n"
+ "The [rubric](./assignment_2_rubric_clean.xlsx) contains the criteria for assessment."
]
},
{
@@ -374,7 +661,7 @@
"* The branch name for your repo should be: `assignment-2`\n",
"* What to submit for this assignment:\n",
" * This Jupyter Notebook (assignment_2.ipynb) should be populated and should be the only change in your pull request.\n",
- "* What the pull request link should look like for this assignment: `https://github.com//algorithms_and_data_structures/pull/`\n",
+ "* What the pull request link should look like for this assignment: `https://github.com//production/pull/`\n",
" * Open a private window in your browser. Copy and paste the link to your pull request into the address bar. Make sure you can see your pull request properly. This helps the technical facilitator and learning support staff review your submission easily.\n",
"\n",
"Checklist:\n",
@@ -383,20 +670,22 @@
"- [ ] Reviewed the PR description guidelines and adhered to them.\n",
"- [ ] Verify that the link is accessible in a private browser window.\n",
"\n",
- "If you encounter any difficulties or have questions, please don't hesitate to reach out to our team via our Slack at `#cohort-6-help`. Our Technical Facilitators and Learning Support staff are here to help you navigate any challenges.\n"
+ "If you encounter any difficulties or have questions, please don't hesitate to reach out to our team via our Slack at the `help` channel. Our Technical Facilitators and Learning Support staff are here to help you navigate any challenges."
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
- "source": []
+ "source": [
+ "# Reference\n",
+ "\n",
+ "Cortez,Paulo and Morais,Anbal. (2008). Forest Fires. UCI Machine Learning Repository. https://doi.org/10.24432/C5D88D."
+ ]
}
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3 (ipykernel)",
+ "display_name": "base",
"language": "python",
"name": "python3"
},
@@ -410,9 +699,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.5"
+ "version": "3.12.3"
}
},
"nbformat": 4,
- "nbformat_minor": 4
+ "nbformat_minor": 2
}