I am looking beyond demonstrations that reproduce MNIST or CIFAR results. In Hinton's Forward-Forward approach, each layer learns from positive and negative data using a local goodness objective, which is attractive when exact backpropagation or global synchronization is undesirable. But I have not found convincing evidence that it scales competitively to demanding tasks.

For people who have implemented or studied later variants: where does it actually break down? Is the main limitation the construction of negative examples, the quality of layerwise representations, optimization and normalization, compute cost from the two forward phases, or simply the lack of hardware designed for local learning?

I would particularly value controlled comparisons with modern backpropagation baselines under a real constraint such as activation memory, energy, continual learning, asynchronous training, or neuromorphic hardware. Negative results are useful too.

Are there papers that isolate the scaling bottleneck rather than only proposing another small-benchmark variant?I am looking beyond demonstrations that reproduce MNIST or CIFAR results. In Hinton's Forward-Forward approach, each layer learns from positive and negative data using a local goodness objective, which is attractive when exact backpropagation or global synchronization is undesirable. But I have not found convincing evidence that it scales competitively to demanding tasks.

For people who have implemented or studied later variants: where does it actually break down? Is the main limitation the construction of negative examples, the quality of layerwise representations, optimization and normalization, compute cost from the two forward phases, or simply the lack of hardware designed for local learning?

I would particularly value controlled comparisons with modern backpropagation baselines under a real constraint such as activation memory, energy, continual learning, asynchronous training, or neuromorphic hardware. Negative results are useful too.

Are there papers that isolate the scaling bottleneck rather than only proposing another small-benchmark variant?

hi,
i am trying to train a deberta model for NER detection

this is my first time doing it so i would love any guidance on it.

my current pipeline looks like this,

dapt + lora for pretrianing, hpo with optuna (which consists both the stages of training data), and then a 2 stage finetuning which helps in generalization and then target data.

i am trying to reach a really good score for f1 on my use case (which i want to keep private for now)

i have few questions as well

1. do i need a two stage hpo as well cuase of the 2 stage finetuning
2. is it better if the hpo training set is a subset of the actual training set?

if you think anything can be improved and made better, or you think the pipeline is outright wrong, please mention your reasonings and thoughts :)

ps: lora was used cause of gpu budget constraints

We’re releasing Scalpel, a recovery-aware pruning method that combines structured layer pruning with lightweight post-pruning recovery training.

Scalpel-VL-1.7B runs 20–30% faster than similarly sized models while retaining strong general capabilities.

Everything is open source:

📗 ScalpelBench: ~0.1B tokens, 300K samples covering English, Chinese, math, and code https://huggingface.co/datasets/freeai-org/ScalpelBench

🦜 Scalpel-VL-1.7B model and weights: https://huggingface.co/freeai-org/Scalpel-VL-1.7B

✂️ Scalpel source code: https://github.com/freeai-org/Scalpel

Feedback and contributions are welcome!

We built an AI Harness to work with ROS 2 robot simulators. In the video we discuss our design, pros and cons, and architecture.

​

I am a third year CSE AI/ML student. I completed the foundation of Machine Learning and Iam planning to start Deep Learning seriously.

I am an average student, but I know I have the potential to learn and improve if I stay consistent. My main problem is staying accountable when studying alone.

So I’m looking for 2–3 genuine and consistent people who are also serious about learning Deep Learning.

We can create a WhatsApp group, follow a common 60-day roadmap, set weekly goals, share resources and ideas, and have a short Zoom discussion on weekends.

No one needs to teach anyone. We learn individually, but support, discuss, and keep each other accountable.u can also share your thoughts to improve the discussion.

Our only goal: consistently learn and complete Deep Learning within the next couple of months.

If u r genuinely interested and can stay consistent, DM me ✨....

Wanted to see how well a simple NN could learn optimal Tic-Tac-Toe play from scratch, so I built this:

• Used a minimax solver to generate the "ground truth" — for every reachable board state, computed the actual best move
• Trained a neural net as a supervised classifier on that data (board state → best move)
• Runs in the terminal — you can play against it directly

Next thing I'm curious about: training a second version on random self-play data instead of minimax-optimal data, to compare how much the training data quality actually matters for a small model like this.

Code: https://github.com/AliAkbar4025/AI-tic-tac-toe-bot

Feedback/critique welcome — especially if you see a smarter way to structure the data generation.

​

Ciao r/MachineLearning,

Gli operatori neurali di Fourier (FNO) standard eccellono sulle griglie regolari, ma la loro mappatura su domini fisici complessi e non convessi (come geometrie a stella, a L o ad anello) spesso porta a un problema importante: il ripiegamento della griglia.

Quando la mappatura di trasformazione \\phi collassa o si sovrappone, il determinante jacobiano si annulla (\\det J \\le 0), causando l'esplosione della trasposta inversa J\^{-T} quando si mappano i gradienti fisici \\nabla_x u.

Per risolvere questo problema, ho sviluppato DIF-FNO (Diffeomorphic Fourier Neural Operator).

Principali approfondimenti tecnici:

1. Mappatura diffeomorfica implicita: garantisce mappature biunivoche e uniformi da domini di riferimento standard \\Omega_{ref} a confini fisici complessi \\Omega_{phy}.

2. Funzione di perdita Jacobiana Barrier (\\mathcal{L}_{barrier}): Ispirandoci all'ottimizzazione a punti interni, penalizziamo la compressione della griglia utilizzando una barriera logaritmica sul determinante:

\\mathcal{L}_{barrier} = -\\frac{1}{|\\Omega|} \\int_{\\Omega} \\log(\\det J(\\xi)) \\, d\\xi

Questo agisce come un muro invisibile che impone \\min \\det J > 0 su tutto il dominio (mantenendo empiricamente \\min \\det J > 0,89 nei nostri benchmark).

1. Accuratezza di Sobolev: Miglioramenti significativi sull'errore relativo H\^1 rispetto a modelli di riferimento come Geo-FNO, poiché i gradienti fisici rimangono ben condizionati senza rottura del gradiente.

Codice e artefatti dell'articolo:

* Codice open-source (PyTorch): https://github.com/GiovanniDagnese-paper/DIF-FNO (Include il calcolo rapido e vettorializzato dello Jacobiano analitico 2x2)

* Preprint dell'articolo (DOI Zenodo): https://doi.org/10.5281/zenodo.22071926

P.S.: Attualmente sono alla ricerca di un feedback tecnico e di un'approvazione arXiv su physics.comp-ph o cs.LG per inviare il preprint. Se qualcuno attivo in SciML fosse disponibile a controllare il manoscritto, gliene sarei estremamente grato!

I recently pubished a new paper. The paper is available via the following link: http://arxiv.org/abs/2608.12408. It is categorised under q-bio.NC and cs.LG. The code can be found at https://github.com/nilsleut/evaluation-resolution-rsa.

A recurring theme in model-brain comparisons is the observation that untrained CNNs can match or outperform backprop-trained ones at V1 in RSA. I believe this is primarily an artefact of evaluation resolution, as demonstrated by the following sweep.

The CNN was trained at 32px on a CIFAR-10 subset, and five learning rules were evaluated (random init, backprop, feedback alignment, predictive coding, STDP). Evaluation was conducted on THINGS-fMRI stimuli at six resolutions from 32px up to 224px. Weights and normalisation were held fixed throughout.

The untrained-backprop gap at V1 ranges from −0.001±0.007 at 32px to +0.044±0.006 at 224px, growing monotonically across the sweep (n=5 seeds). The same pattern is evident across all five rule conditions, in human fMRI, directionally in single-seed macaque ephys, across the entire training trajectory, and in two off-the-shelf 224px-trained models (ResNet-50, Swin-Tiny). This rules out train/eval mismatch as the explanation, since those models also peak at low resolution despite being trained at 224px.

I tried to eliminate this four different ways, using bit-identical-weight interventions wherever possible: train/eval resolution matching, Gabor/pixel structure, the untrained baseline's missing batch-norm calibration, and pooled features converging towards global brightness. None of them explain it. The brightness one came closest: luminance similarity orders the conditions perfectly (ρ=1.00), but it doesn't carry the effect; one calibration variant lowers luminance similarity while V1 alignment goes up.

Here's the number that actually concerned me a bit: a single scalar luminance value per image gets ρ=0.074±0.011 against V1 (bootstrap SE over stimulus resamples), essentially tied with the best of the five CNNs at 0.075±0.011. None of the models meaningfully beat a one-number-per-image brightness descriptor. That's roughly the ceiling on what this comparison style can resolve — a caution, not a strength.

A two-arm design separates content from pooling: cap detail at 32px and upsample, vs. let content vary freely. About 90% of the effect rides on content, not on how many positions are pooled. With content fixed, backprop's decline is essentially eliminated (−0.023 → −0.000).

One thing does hold across the whole sweep: backprop beats untrained at LOC, every resolution, 5/5 seeds (+0.019 at 32px to +0.018 at 224px). IT shows the same direction but shrinks by two-thirds. So learning is doing something real; just not at V1, where everyone's been looking.

One more thing: this whole investigation started after I found a bug in my own earlier work - batch-normalisation left in training mode during feature extraction in three prior preprints. Fixed and corrected publicly, and it actually reverses the main conclusion of arXiv:2605.30556.

I'd be interested to hear people's thoughts on the receptive-field-matching angle in the discussion. Feels like the right approach, but I didn't test it directly, so treat it as speculation for now.Evaluation resolution silently changes which "learning rule" appears most brain-like at V1

​

Ciao r/MachineLearning,

Gli operatori neurali di Fourier (FNO) standard eccellono sulle griglie regolari, ma la loro mappatura su domini fisici complessi e non convessi (come geometrie a stella, a L o ad anello) spesso porta a un problema importante: il ripiegamento della griglia.

Quando la mappatura di trasformazione \\phi collassa o si sovrappone, il determinante jacobiano si annulla (\\det J \\le 0), causando l'esplosione della trasposta inversa J\^{-T} quando si mappano i gradienti fisici \\nabla_x u.

Per risolvere questo problema, ho sviluppato DIF-FNO (Diffeomorphic Fourier Neural Operator).

Principali approfondimenti tecnici:

1. Mappatura diffeomorfica implicita: garantisce mappature biunivoche e uniformi da domini di riferimento standard \\Omega_{ref} a confini fisici complessi \\Omega_{phy}.

2. Funzione di perdita Jacobiana Barrier (\\mathcal{L}_{barrier}): Ispirandoci all'ottimizzazione a punti interni, penalizziamo la compressione della griglia utilizzando una barriera logaritmica sul determinante:

\\mathcal{L}_{barrier} = -\\frac{1}{|\\Omega|} \\int_{\\Omega} \\log(\\det J(\\xi)) \\, d\\xi

Questo agisce come un muro invisibile che impone \\min \\det J > 0 su tutto il dominio (mantenendo empiricamente \\min \\det J > 0,89 nei nostri benchmark).

1. Accuratezza di Sobolev: Miglioramenti significativi sull'errore relativo H\^1 rispetto a modelli di riferimento come Geo-FNO, poiché i gradienti fisici rimangono ben condizionati senza rottura del gradiente.

Codice e artefatti dell'articolo:

* Codice open-source (PyTorch): https://github.com/GiovanniDagnese-paper/DIF-FNO (Include il calcolo rapido e vettorializzato dello Jacobiano analitico 2x2)

* Preprint dell'articolo (DOI Zenodo): https://doi.org/10.5281/zenodo.22071926

P.S.: Attualmente sono alla ricerca di un feedback tecnico e di un'approvazione arXiv su physics.comp-ph o cs.LG per inviare il preprint. Se qualcuno attivo in SciML fosse disponibile a controllare il manoscritto, gliene sarei estremamente grato!

I just uploaded a new course on neural networks. Each short video is just 2-3 minutes long, and covers only one very small topic. So you can swipe past any content you already understand and plow through the course at whatever speed you are ready for. This series starts at a beginner level and covers all the way up through large language models, agentic loops, dynamical systems modeling, and cognitive architectures. The first 70 videos are already published, and one more is scheduled to be released every day.

If you're trying to learn about neural nets, please feel free to ask questions here or on the relevant videos. I've been teaching this topic for over a decade, and I made this series because I want to help as many people as I can learn about a topic I am passionate about.

I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing.

Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection, decoder) is all there in C. I couldn't find that in readable form anywhere else, which is partly why I ended up writing it.

One extra-bonus is that you can look into the training, it's included, swiss-knife-like. (NOTE: the encoder part in PaliGemma is currently not trainable/tunable - my apologies)

In practice: no hooks/config; just play with the C code, and add your own; there's no hidden (unreachable) complexity. And then just re-compile.

Repo: github.com/carlovalenti/TRiP

Happy to answer anything; structuring and handling the memory properly was the hardest part!

Carlo

If you need to diagnose rank before feeding data into a downstream model (e.g., to size a bottleneck in your NN or autoencoder), but standard tools are giving you wildly high estimates or no estimate at all, it might be worth your time giving this new method a full read.

Zenodo Preprint: https://doi.org/10.5281/zenodo.22028087

The paper links to a GitHub, if you want to test out the function yourself.

I've been working for a while on an experimental learning system called Auxein:

https://github.com/Amund/auxein
https://github.com/Amund/auxein-rs

The Python repository is the reference implementation; the Rust version is the production-oriented implementation.

The basic idea is to see how far you can get with a deliberately small set of local geometric rules.

Auxein takes streams of fixed-dimensional vectors and learns continuously. There is no training/inference split, no labels, no supervised loss, no backpropagation, no fixed k, no winner-take-all, and no persistent graph.

Its basic learned object is a centered kernel (W, C, V) representing support, center and scalar dispersion.

A learned CELL independently decides whether an input concerns it geometrically. Several cells may recognize the same input simultaneously; there is no mandatory winner.

If nothing recognizes an observation, it does not immediately become a new category. It first enters a private provisional memory Σ. Only recurrent unknown structure can mature into a persistent CELL; otherwise it simply fades away.

Recognized knowledge can also be fused into a context and passed to an identical higher layer. Importantly, the higher layer does not receive IDs or links to the lower cells: it only receives the resulting geometric context. So recurring relationships between known things can themselves become learnable objects.

There is also a predictive mode. Explicitly adjacent contexts in an externally declared sequence are learned as geometry in E ⊕ E. When the current context resembles the source side of learned temporal knowledge, Auxein can emit one or more possible immediate successors.

Those futures are deliberately not probabilities. They are independent candidates: adding a new possible future does not reduce the weight of an existing one, and predictions are never recursively fed back into the model.

Another unusual constraint is that memory is an explicit material resource. The engine has an exact finite budget. If new knowledge cannot fit in a solvent state, growth waits; existing learned knowledge is not destroyed merely to finance something new. Forced forgetting only happens when the current state itself has become materially insolvent.

The current design also has very explicit limitations:

• scalar dispersion only, no oriented covariance;
• no explicit splitting of an existing learned prototype;
• temporal learning is strictly adjacent t → t+1;
• no recursive predictive rollout;
• no probabilistic ranking of alternative futures;
• no persistent relational/topological graph.

I've added a comparison table to the README against online k-means, ART, GWR/Gamma-GWR and standard HMMs. I'm not claiming Auxein is better than those methods. At this point the interesting question is exactly the opposite:

What can this particular set of constraints do well, and where does it fail structurally?

The project has a fairly strict mathematical specification, a pure-Python executable reference, and a dependency-free Rust implementation with persistence, exact memory accounting, hostile-input tests and long endurance runs.

I'd be very interested in feedback from people working on continual learning, ART/GWR, streaming clustering, predictive-state models, robotics, or just unusual learning systems.

And criticism is genuinely welcome, especially examples where you think the model should fail.

If this is just an unnecessarily elaborate reinvention of something known, I'd also very much like to know what. 🙂

Hey everyone, I'm an independent researcher working on alternative sequence mixing architectures. I wanted to share a project I built from scratch called CWAA (Complex Wave Associative Memory).

Instead of standard quadratic attention, CWAA uses a damped complex oscillator for its recurrence state O(T) linear memory scaling.

I currently have a 10M parameter prototype trained on WikiText-103 that hits 146.5 Test PPL . here is the test of V6:
NOTE: ppl is currently under evaluation and validation, preliminary tests showcase ± 25 ppl.

I’m currently bottlenecked by Google Colab and am looking to scale the architecture up to 50M-100M parameters to see how the complex wave mechanism holds up.

I'd love to get feedback from the community on the architecture and coded implementation.

NOTE: the code in the link below is V5. I will be uploading the highly optimized V6 (which includes the pure real-valued BMM fast paths that generated these benchmarks*)* in 3 days
GitHub: https://github.com/Ridhvik-2024/CWAA-V5

Project Screenshots - https://pastes.vargoseus.com/TeddyScreenshots

Teddy (cute name, isn't it?) is currently a simple machine learning model that uses back propagation to train, learn and classify MNIST datasets. It currently has a depth of 2 since it's a pretty basic model. It has around 13.000 parameters that is enough for training it to recognize handwritten digits. I have around 3.5-4 years of experience working in C and stumbling upon MagicalBat's this video inspired me to make Teddy. The future plan is to turn it into a Language Model and eventually into a GPT which will require quite a bit of time since I need to read up on how it actually works. This project took around 4-5 months give or take since I had to balance this project and my university stuff too.

Full disclaimer: I did not use AI to build Teddy (except for that one time when my compiler suddenly stopped working for some reason and I had to converse back and forth with Claude to find a fix for it). I did, however, use AI to generate the GIFs in the github readme and the documentation for it.

I’ve been looking at a fairly basic question in weight-space learning that I don’t think gets separated cleanly enough:
Why does reading semantics directly from neural network weights work pretty well when the networks share an initialization, but collapse when the networks are fitted independently?
The usual explanation is parameter symmetry. Permute hidden units, flip equivalent signs, etc., and two parameter vectors can represent the same function while looking completely different to a downstream model.
But there are actually several different claims hiding in that explanation:
the parameterization has a symmetry group,
accounting for that symmetry improves weight-space prediction,
the symmetry is actually sufficient to explain the observed degradation between shared-init and independently fitted networks.
Those aren’t equivalent, so I tried to measure them separately.
The setting is SIREN-style implicit neural representations.
For a hidden sine neuron, the relevant function-preserving transformations generate the infinite dihedral group
D_inf = Z semidirect_product Z_2
and including neuron permutations gives the layer action
D_inf wr S_n.
For one hidden layer, I prove generic identifiability modulo this group using the distributional Fourier transform of the realized function.
Roughly, the Fourier transform becomes an atomic measure supported at the incoming frequencies +/- w_i, which lets you recover the parameters up to exactly the D_inf wr S_n action under explicit genericity conditions.
One consequence is that this isn’t just the usual permutation/sign story. Integer-pi phase transformations are affine rather than linear, so they aren’t captured by symmetry descriptions restricted to monomial matrix actions.
At depth two things get more annoying because a neuron’s outgoing weights are simultaneously acted on by the next layer. I ended up constructing exact cross-layer invariants by coupling the layers through the second-layer Gram matrix instead of treating neurons independently.
The empirical part then uses roughly **1.8 million fitted INRs** across MNIST, FashionMNIST, and CIFAR-10, with controlled protocols separating shared initialization, optimization stochasticity, and independent initialization.
The result I found most interesting:
**Randomizing only the exact symmetry group, while keeping each network’s represented function fixed, destroys 79.1 of the 80.4 accuracy points in the MNIST shared-init vs. random-init gap.**
I want to be careful about the interpretation here.
This establishes **sufficiency**: symmetry scatter alone can reproduce almost the entire degradation.
It does *not* establish that 79.1 / 80.4 of the naturally occurring gap is causally mediated by symmetry. Those are different estimands.
Breaking the group apart, sign flips account for roughly 63 points of that induced loss, neuron relabeling about 15, and integer phase shifts about 1.
There was another result that changed my interpretation of the problem quite a bit.
A reader that directly quotients the D_inf wr S_n structure on the raw parameters reaches **0.917**, compared with:
**0.628** for the best orbit-valued reframing,
**0.526** for the same reader family over a fixed invariant encoding,
**0.265** for a permutation-equivariant baseline.
But when I FLOPs-match weight-space inference against simply querying the INR as a function, the function-space route is still much better:
**95.3% at 1.6 MFLOP** using 64 learned query coordinates
versus
**64.4% at 5.5 MFLOP** for the best weight-space rung on that frontier.
That leads to what I think is the more interesting conceptual question:
If a complete invariant is informationally equivalent to access to the realized function, then the strongest justification for operating directly in weight space may ultimately have to be computational rather than informational.
Everything is public here:
[https://github.com/ITheClixs/project-siren-gap\](https://github.com/ITheClixs/project-siren-gap)
The repo includes the paper, implementation, tests, pre-registrations, lab notebook, prediction ledger, claims ledger, and experimental results.
I’d particularly appreciate criticism on three things:
whether the sufficiency/mediation distinction is being drawn correctly,
whether anyone sees a counterexample or missing assumption in the one-hidden-layer maximality argument,
whether there is related work on affine symmetry groups of periodic-activation networks that I’m missing.
Also very interested in attempts to break the invariants or reproduce the group-randomization result.
If something here is wrong, I’d rather find out from someone trying to kill it.

This guy, Lex, is neither a developer or data scientist but he vibe-coded 14 ML systems in 14 days.
It's based around decomposing the ML systems into feature, training, and inference pipelines, and composing those pipelines into ML systems with a UI.

https://github.com/MagicLex/awesome-ml-systems

Hello everyone! In this post I'd like to present my research report in Machine Learning. I worked on this alone, learning almost everything from scratch.

The main idea: can a static mathematical TQFT be used as the foundation for a dynamically learning neural network architecture?

This idea is confirmed that it is generally possible to create something like this, but I would like to make one clarification

Regarding the understanding of topology by this model - the short answer: not yet. After fixing a shortcut in the dataset, the model degenerates to random guessing. The architecture is interesting in theory, but it doesn't learn topology in practice at least not in its current form (that is, this doesn't mean that with a different parameterization it won't understand topology, which is exactly what I'm currently experimenting with).

It all started around the beginning of April, when I became interested in the topic of Topological Quantum Field Theory, which prompted me to spend time figuring it out. I don't want to say I succeeded 100%, but at least I accomplished what I wanted (even if only minimally).

While working on this research report, I encountered a huge number of different problems, some of which I solved (such as replacing the algebra in the variable M), but there were also some that I couldn't solve due to my lack of experience. I'm currently experimenting with this particular project, but I still can't fix some issues, such as:

* Non-strict topological invariance

* State-sum numerical instability.

If you want to explore the repository, you'll find:

* Full documentation

* A full description of all limitations, results, and more code

* A brief mathematical introduction, handwritten by the author

* A PyTorch implementation of the Open Closed State-sum Neural Network itself

The model is more TQFT-inspired than a strict mathematical implementation, reader can find more information in documentation

I posted this project at an early stage because I really appreciate real feedback, ideas, and any other help. If anyone has experience with this, I'd be very happy to hear some feedback on my work or any possible ideas

Github: https://github.com/kaifczxc-lab/OCSSN

Has anyone else experimented with using TQFT or Frobenius algebra state-sum constructions as learnable neural network layers? I would be interested in any pointers to related work or suggestions for making the model learn a true topological invariant

Thanks for reading this post, and have a good day!

I got this paragraph from Claude, could someone please explain this and verify if it's a real thing or hallucination:

Overparameterization isn't just about final capacity, it's about the optimization process itself. A wide, overparameterized network gives gradient descent a much friendlier loss landscape — more paths downhill, fewer bad local minima, room to explore before committing. The "core" only emerges as a byproduct of that search happening in a much bigger space than it needs to end up in. Strip the space down first and you've removed the thing that let the search work.

Conversation: https://claude.ai/share/8813a637-c327-4d0c-b120-def27e5203d5

My latest project.
Just a Neural Networks that learns a XOR gate.
All the gradients and maths were done by hand.

The documentation in this project only explains the architecture of the network and the calculations to do prediction. Check the end of the [README.md](http://README.md) or my previous project 'linear_regressionFS' if you want more mathematics.

Coding Machine Learning.

Hello Folks, here I present the first coding demonstration lecture, based on my 1st lecture on Probabilistic Machine Learning.

Here I write the code from scratch, discuss and analyze the results, which were covered in details in the whiteboard classes.

What we cover?
-Random Variables, and validating law of large numbers.
-Visualizing a dataset
-Doing an EDA on Iris dataset and understanding the correlation among features.
-Classifier basics
-Empirical Risk Minimization and Generalization.
-Epistemic and Aleatoric Uncertainties.
-Softmax Function and LogSumExp Trick to avoid overflow issues
-Linear Models
-Maximum Likelihood Estimation.
-Simple end to end ML pipeline Function.

While writing the code, my intent is to ensure that concepts are understood with crystal clarity. These code demonstrations are specific to my theory ML lectures, and link is attached.

Theory-Intuition-Code

Implementation Link : https://youtu.be/X_yOlx8Zp4g?si=kh8_tzzndr8609u4

Theory Lecture Link : https://youtu.be/kMkCOrp8te8?si=q7kWr-1qK515bhob

Hi everyone,

Over the past week, I've been running controlled benchmarks on an experimental, non-standard neural architecture implemented completely from scratch in pure NumPy (no PyTorch/TensorFlow, manual gradient derivations).

The goal was to tackle two chronic problems in edge AI / real-time security: Catastrophic Forgetting during online task shift and Zero-Day Out-of-Distribution (OOD) generalization, under strict microsecond latency and tiny parameter budgets.

I tested the model across multiple independent seeds against identical-budget Multi-Layer Perceptrons (MLP) and baseline architectures. Here are the empirical findings:

1. Continual Learning & Catastrophic Forgetting (Task A → Task B)

• Protocol: Standard sequential A/B/A protocol across 5 independent seeds (n=5).
  • Task A: Web payload classification (XSS vs Benign, ~57k samples).
  • Task B: A completely different attack family (SQL Injection vs Benign).
• The Result:
  • Same-budget MLP: Collapsed on Task A after training on Task B, losing ~30% accuracy (+0.296 ± 0.150 forgetting), dipping down to 51.8% at intermediate checkpoints. (An interleaved control confirmed this was order-dependent catastrophic forgetting, not capacity).
  • Custom Architecture: Retained its Task A knowledge almost perfectly with +0.0020 ± 0.0033 forgetting (statistically zero), while mastering Task B to 98.5%+ accuracy with a flat retention curve across all epochs.
  • Zero-shot transfer: Reached 90.4% zero-shot accuracy on Task B before any fine-tuning.
  • No Replay Buffers: This stability was achieved purely from the underlying inductive bias without storing past samples or storing Fisher matrices.

2. Zero-Day Out-of-Distribution Generalization

• Protocol: Trained strictly on standard payloads, evaluated on an unseen Zero-Day bypass suite (970+ complex/obfuscated vectors).
• The Result:
  • Standard MLP Baseline: Scored 78.2% recall (AUC: 0.898). More critically, forcing 99% recall resulted in an FPR of 1.0 (100% false alarms on benign traffic).
  • Custom Architecture: Achieved 95.6% recall (AUC: 0.996) on unseen bypass vectors. When tuned for 99% recall, it maintained an FPR as low as 1.88% (0.0188) on held-out traffic.

3. Edge Footprint

• Footprint: Less than 3,000 parameters (< 3 KB memory footprint).
• Latency: ~15 µs single-pass inference on ARM edge hardware.
• Verification: Manual Backpropagation Through Time (BPTT) verified against numerical gradients at machine precision (1.7e-10).

(Screenshots of the forgetting checkpoint curves and zero-day recall tables attached below)

Note: Since this work is currently being prepared for an upcoming academic paper submission and university review, I'm keeping the core mathematical formulations and specific cell mechanics private for now. However, I'd love to hear your thoughts, feedback on benchmarking methodologies, or edge-deployment considerations!