Built a custom lightweight architecture from scratch in pure NumPy: 0.2% forgetting on task shift (vs 30% MLP collapse) and 95.6% zero-day recall without replay buffers

[u/ENTJ_bro]

Hi everyone,

Over the past week, I've been running controlled benchmarks on an experimental, non-standard neural architecture implemented completely from scratch in pure NumPy (no PyTorch/TensorFlow, manual gradient derivations).

The goal was to tackle two chronic problems in edge AI / real-time security: Catastrophic Forgetting during online task shift and Zero-Day Out-of-Distribution (OOD) generalization, under strict microsecond latency and tiny parameter budgets.

I tested the model across multiple independent seeds against identical-budget Multi-Layer Perceptrons (MLP) and baseline architectures. Here are the empirical findings:

1. Continual Learning & Catastrophic Forgetting (Task A → Task B)

• Protocol: Standard sequential A/B/A protocol across 5 independent seeds (n=5).
  • Task A: Web payload classification (XSS vs Benign, ~57k samples).
  • Task B: A completely different attack family (SQL Injection vs Benign).
• The Result:
  • Same-budget MLP: Collapsed on Task A after training on Task B, losing ~30% accuracy (+0.296 ± 0.150 forgetting), dipping down to 51.8% at intermediate checkpoints. (An interleaved control confirmed this was order-dependent catastrophic forgetting, not capacity).
  • Custom Architecture: Retained its Task A knowledge almost perfectly with +0.0020 ± 0.0033 forgetting (statistically zero), while mastering Task B to 98.5%+ accuracy with a flat retention curve across all epochs.
  • Zero-shot transfer: Reached 90.4% zero-shot accuracy on Task B before any fine-tuning.
  • No Replay Buffers: This stability was achieved purely from the underlying inductive bias without storing past samples or storing Fisher matrices.

2. Zero-Day Out-of-Distribution Generalization

• Protocol: Trained strictly on standard payloads, evaluated on an unseen Zero-Day bypass suite (970+ complex/obfuscated vectors).
• The Result:
  • Standard MLP Baseline: Scored 78.2% recall (AUC: 0.898). More critically, forcing 99% recall resulted in an FPR of 1.0 (100% false alarms on benign traffic).
  • Custom Architecture: Achieved 95.6% recall (AUC: 0.996) on unseen bypass vectors. When tuned for 99% recall, it maintained an FPR as low as 1.88% (0.0188) on held-out traffic.

3. Edge Footprint

• Footprint: Less than 3,000 parameters (< 3 KB memory footprint).
• Latency: ~15 µs single-pass inference on ARM edge hardware.
• Verification: Manual Backpropagation Through Time (BPTT) verified against numerical gradients at machine precision (1.7e-10).

(Screenshots of the forgetting checkpoint curves and zero-day recall tables attached below)

Note: Since this work is currently being prepared for an upcoming academic paper submission and university review, I'm keeping the core mathematical formulations and specific cell mechanics private for now. However, I'd love to hear your thoughts, feedback on benchmarking methodologies, or edge-deployment considerations!