ML Classification Study
Neural Networks · K-Means Clustering · Decision Trees
MNIST · UCI Iris · Backpropagation from Scratch
ENEE436 is the Machine Learning course at the University of Maryland, requiring both mathematical derivation from first principles and full implementation from scratch. Project 2 was a comparative study of three fundamentally different classification paradigms — gradient-based neural network optimization, unsupervised geometric clustering adapted for classification, and recursive feature partitioning via decision trees — applied across two standard benchmarks: the MNIST handwritten digit dataset (60,000 training / 10,000 test images) and the UCI Iris dataset (150 samples, 3 species classes). The project covered the full arc from mathematical derivation through implementation, evaluation, failure-mode analysis, and mitigation.
The neural network derivations grounded every implementation decision: working out the backpropagation update rule for both sigmoid and tanh activations from the chain rule, understanding why the sigmoid derivative simplifies to f(z)(1−f(z)), and expressing the weight update in matrix form before writing a single line of code. This derivation-first approach shaped the implementation — the per-sample SGD update, the one-hot target encoding, the augmented input vector with a prepended bias term — all following directly from the math rather than from a library abstraction.
Across three experiments, the project produced concrete results with distinct takeaways: the neural network reached 92.17% test accuracy on MNIST and converged within five epochs; K-Means over-clustering (K=50) improved classification accuracy by 25.1 percentage points over a K=10 baseline by decoupling visual modes from semantic classes; and two decision trees trained on the same Iris data with different random seeds achieved identical training accuracy but disagreed on approximately 42% of feature space — illustrating the structural variance that ensemble methods like Random Forests were designed to address.
Three Experiments
Three classification paradigms — each with different assumptions, failure modes, and results.
Neural Network on MNIST
Gradient descent · Sigmoid · SGD
A single-layer sigmoid network mapping 785-dimensional augmented inputs to 10 digit-class outputs, trained with per-sample stochastic gradient descent. Derivations for sigmoid and tanh backpropagation were worked out analytically before implementation. Converges within roughly five epochs and plateaus near the theoretical ceiling for a linear classifier on raw pixels.
K-Means on MNIST
Unsupervised clustering · Majority vote
K-Means with majority-vote label assignment. K=10 left digits 5 and 9 without any centroid, achieving only 54.96% test accuracy. Over-clustering to K=50 covered all digit classes and produced a +25.1 percentage point improvement — demonstrating that the failure was structural, not incidental to algorithm performance.
Decision Trees on Iris
Recursive partitioning · Variance analysis
Two decision trees trained on identical Iris data with different random seeds and row orderings. Both achieve 100% training accuracy but select different root features and internal splits — disagreeing on approximately 42% of the full feature space and predicting opposite classes for the same divergent example.
Technical Breakdown
Implementation details and analytical findings across all four project problems.
01
Backpropagation Derivations
Worked out the backpropagation update rule analytically for both sigmoid and tanh activations before any implementation. For the sigmoid case, the chain rule yields ∂L/∂W_kj = (ŷ_k − t_k) · ŷ_k(1 − ŷ_k) · X_j, collapsing to the matrix update W ← W − η·δ·X^T where δ = (ŷ − t) ⊙ ŷ ⊙ (1 − ŷ). The tanh derivative f'(x) = 1 − f(x)² was derived via the quotient rule — a closed form depending only on the activation value, mirroring the sigmoid structure. Both derivations were verified before a single line of implementation code was written.
02
Single-Layer Sigmoid Network
Implemented entirely in NumPy. Each MNIST image is flattened to 784 pixels and prepended with a bias term (X₀ = 1), producing a 785-dimensional augmented input. Weights initialized from N(0, 0.01²) in float32. Training runs per-sample SGD with η=0.1 and random epoch shuffling for 20 epochs with checkpointing after each epoch for resumable training. The sigmoid is numerically stabilized via clipping. Final performance: 92.25% train / 92.17% test — near the known ceiling for a linear classifier on raw MNIST pixels without learned feature representations.
03
K-Means Classification & Over-Clustering
MiniBatchKMeans (batch size 4096, n_init=5) was fit on the full 60,000×784 MNIST array. With K=10, majority-vote labeling left digits 5 and 9 without centroids — the algorithm placed two centroids on visual sub-modes of '0' and '1' instead, achieving only 54.96% test accuracy with 8 of 10 classes representable. The over-clustering fix: K=50 decouples visual modes from semantic class labels, allowing each digit to occupy multiple centroids covering different writing styles. Result: 80.02% test accuracy, all 10 classes covered — a structural fix, not an optimization improvement.
04
Decision Tree Variance Analysis
A DecisionTreeClassifier with max_features=2 was fit on UCI Iris twice — original and randomly permuted row ordering, different random seeds. Both trees achieve 100% training accuracy. A 40⁴ = 2,560,000-point grid spanning per-feature min/max found disagreement on 1,076,158 points (~42%). The most illustrative divergent example x* = (6.146, 2.185, 6.749, 0.654) has contradictory features: long petal (virginica-characteristic) but narrow petal width (setosa-characteristic). Tree B routes via petal_width ≤ 0.8 → setosa. Tree C routes via sepal_length > 5.45 → virginica. The disagreement is the variance that ensemble methods — Random Forests, Extra-Trees — are designed to average out.
Results
92.17%
Neural Network Test Accuracy
MNIST · 20 epochs · per-sample SGD
+25.1pp
K-Means Accuracy Gain
K=10 → K=50 over-clustering
~42%
Decision Tree Disagreement
Feature space where identical-accuracy trees diverge
Key Findings
What each experiment reveals beyond its accuracy number.
01
Gradient Descent vs. Nearest-Centroid
The 37-point accuracy gap between the neural network (92.17%) and K-Means K=10 (54.96%) on the same MNIST data illustrates the fundamental difference between learned representations and geometric partitioning. The neural network optimizes directly against the classification objective; K-Means optimizes cluster compactness — only indirectly related to class separability.
02
Structural Failure vs. Hyperparameter Failure
K-Means K=10 failed not because the algorithm performed poorly, but because K=10 is structurally insufficient for a 10-class problem with visual sub-modes. The fix — over-clustering to K=50 — doesn't improve the optimization. It changes the problem structure so the same optimization can succeed. Diagnosing failure as structural rather than incidental is what leads to the correct solution.
03
Perfect Accuracy Is Not Enough
Both Iris decision trees achieve 100% training accuracy yet disagree on 42% of feature space. A model can memorize training data perfectly while generalizing to fundamentally different decision regions. This is variance — and it's why ensemble methods average over many trees, not to improve any single tree, but to cancel out their structural disagreements.
Stack
Figures
Exported directly from the project report — learning curves, cluster centroids, and decision tree structures.
Figures from ENEE436 Project 2 Report · University of Maryland · Spring 2026
My Documents
Full project report including mathematical derivations, implementation details, and analysis.