Homework 5

Homework 5#

Coverage: Lectures 16–18
Due: Sunday, October 18, 2026, 11:59 p.m. ET
Total: 100 points

Instructions#

  • Complete this notebook in Google Colab.

  • All code is provided. Run the cells in order, unchanged. Do not delete supplied setup or helper cells; you are not expected to study the setup or plotting implementation.

  • Answer the seven short parts in the marked cells. One to three sentences or short bullets per part are enough. Refer to displayed tables instead of copying them. No derivations, model tuning, or separate report are required.

  • Your completed notebook must run from beginning to end in a fresh Colab runtime without Google Drive, absolute paths, or additional package installation. A CPU runtime is sufficient.

  • Submit a PDF of your completed notebook, including your name, answers, tables, and figure.

Allow about 90–120 minutes, including reading and running the notebook.

The course material needed for this assignment is collected here:

Student details#

  • First name:

  • Last name:

  • Purdue email:

Problem 1 — A probability is not a decision (20 points)#

A camera system records a fault on a steel plate. Let \(y=1\) denote a K-scratch fault and \(y=0\) denote another fault type. Action \(a=1\) routes the record to a K-scratch specialist; action \(a=0\) routes it to the other-fault queue. Neither action accepts or rejects the plate.

For this exercise, a missed K-scratch costs 5 units and an unnecessary K-scratch referral costs 1 unit. Correct routing costs zero. These are assumed relative costs, not measured industrial losses.

Let \(p\) be the model’s estimated probability that \(y=1\) for a particular record. The expected costs of the two actions are

\[ R(a=0)=5p, \qquad R(a=1)=1(1-p). \]

Credit: This decision exercise applies the expected-loss rule in the course’s decision-making example. The routing scenario and cost values are specified for this assignment.

1(a) Which action? (10 points)#

For \(p=0.20\), evaluate both expected costs and choose an action. Why can this action differ from choosing the more likely fault category?

Your answer (1–3 sentences):

1(b) A more expensive miss (10 points)#

For positive false-positive cost \(C_{\mathrm{FP}}\) (an unnecessary referral) and false-negative cost \(C_{\mathrm{FN}}\) (a missed K-scratch), the minimum-expected-cost rule chooses \(a=1\) when \(p\) is at least the threshold

\[ t=\frac{C_{\mathrm{FP}}}{C_{\mathrm{FP}}+C_{\mathrm{FN}}}. \]

Here \(t=1/6\). If the cost of a missed K-scratch increases while the referral cost stays fixed, should the threshold increase or decrease? Explain what this does to routing. No derivation is required.

Your answer (1–3 sentences):

Problem 2 — How much information can we discard? (50 points)#

The UCI Steel Plates Faults dataset contains 1,941 recorded faults, 27 numerical features, and seven fault types. A feature is an input measurement or recorded attribute, such as pixel area, luminosity, or steel type. We predict K-scratch versus the other six types. The original label is spelled K_Scatch; the code keeps that spelling.

Every record is a fault. There are no fault-free examples. This distinction will matter when we interpret the results.

Credit: Data: M. Buscema, S. Terzi, and W. Tastle (2010), Steel Plates Faults, UCI Machine Learning Repository, licensed under CC BY 4.0. We use all rows and predictors and combine the seven labels into the binary target described above. Methods: course examples on logistic regression with many features, classification diagnostics, and PCA. The code uses scikit-learn’s StandardScaler, LogisticRegression, and PCA.

# Setup — run unchanged. No package installation is needed in Colab.
from io import BytesIO
from pathlib import Path
from urllib.request import urlopen
from zipfile import ZipFile
import hashlib

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

SEED = 539
pd.set_option("display.precision", 4)
plt.rcParams.update({"font.size": 11, "figure.dpi": 120})

Run the next cell unchanged. It checks the downloaded file before using it. If the UCI download is temporarily unavailable, download the ZIP using the dataset page’s Download button. In Colab, upload it through the Files sidebar under the name steel-plates-faults.zip; locally, put it in the notebook’s working folder. Then rerun the cell.

# Download and verify the original data — run unchanged.
DATA_URL = "https://archive.ics.uci.edu/static/public/198/steel+plates+faults.zip"
local_zip = Path("steel-plates-faults.zip")
if local_zip.is_file():
    archive_bytes = local_zip.read_bytes()
else:
    with urlopen(DATA_URL, timeout=45) as response:
        archive_bytes = response.read()
with ZipFile(BytesIO(archive_bytes)) as archive:
    raw = archive.read("Faults.NNA")
    names = archive.read("Faults27x7_var").decode().splitlines()
assert hashlib.sha256(raw).hexdigest() == (
    "08994f5b0185a8e6cc22afe9910fd36834990fb1ca30852ed79012a832fbff55"
), "Dataset changed. Please contact the course staff."
values = np.loadtxt(BytesIO(raw))
assert values.shape == (1941, 34) and len(names) == 34
assert np.isfinite(values).all()
assert np.isin(values[:, 27:], [0, 1]).all()
assert np.all(values[:, 27:].sum(axis=1) == 1)
X = values[:, :27]
fault_names = np.array(names[27:])[values[:, 27:].argmax(axis=1)]
y = (fault_names == "K_Scatch").astype(int)
print(f"{len(y)} faults; {X.shape[1]} features; {y.sum()} K-scratch records.")

2(a) Which data may we learn from? (10 points)#

We set aside 30% of the records before fitting anything. This held-out set is used only to evaluate the fixed models below. The stratified split keeps approximately the same proportions of the two labels in each set.

Standardization subtracts each feature’s training mean and divides by its training standard deviation. This puts features with different scales on a comparable footing. fit_transform learns these quantities and applies the transformation; transform applies quantities already learned.

Run the cell. Why should both sets use the same transformation, with its means and standard deviations learned from the training set?

idx_train, idx_test = train_test_split(
    np.arange(len(y)), test_size=0.30, random_state=SEED, stratify=y
)
X_train_raw, X_test_raw = X[idx_train], X[idx_test]
y_train, y_test = y[idx_train], y[idx_test]
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train_raw)
X_test = scaler.transform(X_test_raw)
display(pd.DataFrame({
    "Records": [len(y_train), len(y_test)],
    "K-scratch": [int(y_train.sum()), int(y_test.sum())],
}, index=["Training", "Held out"]))

Your answer (1–3 sentences):

2(b) Count the errors that matter (15 points)#

Logistic regression estimates the probability of \(y=1\) from the features. Its coefficients are fitted on training data. All model settings are fixed for this comparison; do not tune them. The default coefficient penalty discourages very large coefficients. We do not estimate uncertainty in the fitted coefficients in this exercise.

A false positive (FP) routes another fault to the K-scratch specialist. A false negative (FN) sends a K-scratch to the other-fault queue. For \(n\) held-out records, the observed mean routing cost is

\[ \text{mean cost}=\frac{\mathrm{FP}+5\mathrm{FN}}{n}. \]

Run the two cells below. When the threshold changes from \(0.5\) to \(1/6\), which error count increases, which decreases, and does the mean cost improve on this held-out set? This is an observed comparison on one sample, not a guarantee of future performance.

# Two supplied helpers: fit a model, then count its held-out errors.
def fit_probabilities(train_features, test_features):
    model = LogisticRegression(C=1.0, solver="lbfgs", max_iter=2000)
    model.fit(train_features, y_train)
    return model.predict_proba(test_features)[:, 1]

def routing_errors(probabilities, threshold):
    prediction = (probabilities >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, prediction, labels=[0, 1]).ravel()
    return {"Threshold": threshold, "FP": int(fp), "FN": int(fn),
            "Mean cost": (fp + 5 * fn) / len(y_test)}

p_full = fit_probabilities(X_train, X_test)
full_results = pd.DataFrame([
    routing_errors(p_full, 0.5),
    routing_errors(p_full, 1 / 6),
])
display(full_results)

Your answer (1–3 sentences):

2(c) Preserve variation or preserve useful information? (25 points)#

Principal component analysis (PCA) finds directions of feature variation, without using fault labels. A PCA score is a record’s coordinate along one of these directions. The explained-variance fraction is the fraction of total training-feature variance retained in the selected directions. Here it refers to the standardized features. We keep the different variances of the PCA scores; we do not rescale each score to unit variance. Computing these scores still uses all 27 original features: PCA reduces the number of coordinates passed to the classifier, not the measurements needed to obtain them.

The code fits two PCA representations on training features: two components, and the smallest number whose combined explained-variance fraction reaches at least 90%. It then fits logistic regression on each representation.

Run the cell. Report the fraction retained by two components and the number needed for 90%. At threshold \(1/6\), which of the three representations has the lowest held-out cost? Does keeping 90% of feature variance guarantee keeping 90% of the information useful for classification? Explain briefly. These three comparisons were chosen in advance: do not use this held-out set to search for additional models.

pca_two = PCA(n_components=2, svd_solver="full", whiten=False)
pca_ninety = PCA(n_components=0.90, svd_solver="full", whiten=False)
Z_train_two = pca_two.fit_transform(X_train)
Z_test_two = pca_two.transform(X_test)
Z_train_ninety = pca_ninety.fit_transform(X_train)
Z_test_ninety = pca_ninety.transform(X_test)

comparison = [{"Representation": "All features", "Dimensions": X_train.shape[1],
               "Variance retained": 1.0, **routing_errors(p_full, 1 / 6)}]
for label, pca, train_scores, test_scores in [
    ("Two PCs", pca_two, Z_train_two, Z_test_two),
    ("90% variance", pca_ninety, Z_train_ninety, Z_test_ninety),
]:
    probabilities = fit_probabilities(train_scores, test_scores)
    comparison.append({"Representation": label, "Dimensions": train_scores.shape[1],
                       "Variance retained": pca.explained_variance_ratio_.sum(),
                       **routing_errors(probabilities, 1 / 6)})
comparison = pd.DataFrame(comparison).set_index("Representation")
display(comparison)

Your answer (1–3 sentences):

Problem 3 — Can geometry find a fault type? (30 points)#

K-means groups records around cluster centers using distance in feature space. We fit two clusters using the 27 standardized training features, without labels. A held-out record is assigned to its nearest fitted center. We use two clusters for comparison with the binary labels, not because we know that two natural groups must exist.

The two-component PCA representation is used only for plotting. A projection can hide separation in other directions. Cluster numbers 0 and 1 are arbitrary identifiers; they are not predictions of labels 0 and 1.

Credit: The method follows the course examples on k-means and visualizing high-dimensional clusters with PCA, using scikit-learn’s KMeans. Data: Buscema, Terzi, and Tastle (2010), UCI Steel Plates Faults, CC BY 4.0; the same binary relabeling as in Problem 2 is used only to interpret clusters.

3(a) Clusters versus labels (15 points)#

A contingency table counts records for each combination of two categories: here cluster ID and known fault label. Run the next two cells. Does either cluster contain only one label? Identify the cluster with the larger fraction of K-scratches and cite its two counts. Explain why we can see substantial agreement with the fault labels even though k-means did not use them during fitting. Use the table for counts; you do not need to inspect individual points or reproduce the supplied plot.

kmeans = KMeans(n_clusters=2, n_init=20, random_state=SEED)
kmeans.fit(X_train)  # No fault labels are passed to k-means.
cluster_test = kmeans.predict(X_test)
counts = pd.crosstab(
    pd.Series(cluster_test, name="Cluster"),
    pd.Series(y_test, name="Fault label"),
).reindex(index=[0, 1], columns=[0, 1], fill_value=0)
counts.columns = ["Other faults", "K-scratch"]
display(counts)

# Supplied plot: all records above, a labeled zoom below. No data are discarded.
inside = ((Z_test_two[:, 0] >= -5) & (Z_test_two[:, 0] <= 15)
          & (Z_test_two[:, 1] >= -5) & (Z_test_two[:, 1] <= 6))
outside_count = int((~inside).sum())
fig, axes = plt.subplots(2, 2, figsize=(10, 7), constrained_layout=True)
panels = [(y_test, ["Other faults", "K-scratch"], ["#2866a4", "#d65f00"], "Known labels"),
          (cluster_test, ["Cluster 0", "Cluster 1"], ["#16887b", "#985ba1"], "K-means groups")]
for column, (groups, labels, colors, title) in enumerate(panels):
    for row in range(2):
        ax = axes[row, column]
        for group, (label, color, marker) in enumerate(zip(labels, colors, ["o", "^"])):
            mask = groups == group
            ax.scatter(Z_test_two[mask, 0], Z_test_two[mask, 1],
                       s=15, alpha=0.6, c=color, marker=marker, label=label)
        ax.set(xlabel="PC 1 score", ylabel="PC 2 score")
        ax.grid(alpha=0.15)
        if row == 0:
            ax.set_title(title + " — full range")
            ax.legend(fontsize=9)
        else:
            ax.set(xlim=(-5, 15), ylim=(-5, 6))
            noun = "record" if outside_count == 1 else "records"
            ax.set_title(f"Zoom — {outside_count} {noun} outside view")
plt.show()
print("All 583 held-out records are included in the table and in the full-range panels.")

Your answer (1–3 sentences):

3(b) What has this study established? (15 points)#

A colleague concludes: “The overlap in the PCA plot proves that the clusters are not separated in the original features. Also, our small routing cost shows that this system can reliably separate faulty plates from fault-free plates.” Give one reason why each claim is unsupported. Two sentences are enough.

Your answer (1–3 sentences):

Before submitting, run all cells from the beginning. Check that your PDF includes your name, all seven answers, and the generated tables and figure. No additional analysis is required.

Contents