NNFS Extreme — The Arthist

A multi-phase engineering progression — from raw C++ linear algebra all the way to gesture-controlled real-time digit classification and, eventually, an inference engine running on a microcontroller. Built with zero external dependencies in both the math library and the neural network engine.

The Progression

Phase 1Completed

Spalten — Linear Algebra Library

A foundational C++ matrix engine built from scratch. Provides the Matrix<T> class template that powers all neural network math downstream. Written with zero external dependencies.

⬡

atharvesting/spalten-linalg-library

Custom C++ linear algebra library — provides Matrix<T>, optimised GEMM, and the full math backbone for NNFS-Extreme.

→

Phase 2Completed

NNFS-Extreme — Neural Network Engine

Built by starting with a dot product and working upward to a fully performant neural network classifying MNIST digit images. Inspired by Michael Nielsen's book — re-implemented in C++. After achieving 95% accuracy, an experiment swapped inputs and outputs: the network (1 hidden layer, 30 neurons) then generated recognisable digit images.

Weights & Biases: vector<Matrix<float>> per layer
Mini-batches: stacked as columns — X [784×m], Y [10×m] for GEMM
Concurrency: thread pool across sub-ranges, accumulating ∇W and ∇b under a mutex
Activation: sigmoid everywhere; cost derivative: output − actual
Binary format: num_layers → sizes[] → weights → biases

Network nn({ 784, 30, 10 });
nn.SGD(training_data, 20, 32, 2.0F, test_data);

rendering…

Phase 3Completed

MNIST-CV — Live Gesture Classification

Classifies digits drawn live in the air. A webcam feed is processed through OpenCV and MediaPipe hand landmark tracking; detected strokes are cropped, processed into MNIST-compatible images, and sent through the NNFS-Extreme engine via pybind11 Python bindings.

File	Role
`MNIST_CV.py`	Entry point — calls `pipe()`
`looping.py`	`@loop` decorator (webcam lifecycle) + per-frame logic
`hand_tracking.py`	MediaPipe landmarks, air-canvas sketch, bounding boxes
`image.py`	MNIST-compatible image processing pipeline
`nnfs_extreme.pyd`	pybind11 C++ extension — exposes `Network` and `feedforward`

rendering…

Phase 4Upcoming

ESP32-MNIST — Edge AI on a Microcontroller

The final phase: deploy the full pipeline — live video stream → gesture drawing → real-time classification — onto the ESP32-S3 Sense board with 8MB PSRAM and 8MB Flash. The primary obstacle is the MediaPipe hand landmark model, which alone exceeds 8MB.

Project Structure

NNFS_Extreme/ ├── NNFS_Extreme/ │ ├── CMakeLists.txt # Build instructions │ ├── NN.hpp / NN.cpp # Neural Network from scratch │ ├── activation_functions.* # Common activation functions + derivatives │ ├── data_loaders.* # MNIST binary loading utilities │ ├── utils.* # ThreadPool and extras │ ├── python_bindings.cpp # pybind11 Python interop │ ├── NNFS_Extreme.cpp # Standalone entry point │ └── examples/ # Ready-to-use training examples ├── MNIST_CV/ │ ├── MNIST_CV.py # Main entry point │ ├── looping.py # Camera loop decorator + main body │ ├── hand_tracking.py # Landmarks, bounding boxes, air-canvas │ ├── image.py # Image processing pipeline │ ├── nnfs_extreme.pyd # pybind11-generated C++ bindings │ └── nnfs_extreme.pyi # Typing stubs for Network class ├── third_party/ │ └── Spalten/ # Git submodule — foundational matrix engine ├── data/ # Model binaries and program outputs └── CMakeLists.txt # Main build instructions

Started: 2026-07-29 · Languages: C++, Python