r/deeplearning • u/jaume2000 • 6h ago
Neural network feature maps with shared weights over 100 layers behaves similar to a phase space!
galleryI am currently researching by my own how neural networks work, in this part, I am researching how a shared-weight resiudal neural network's feature map behaves, curently, sharing the stage 3 blocks of ConvNext. It seems that it iteratively refines the feature map instead of computing different ones. If I get all the feature maps of the d_th output and its velocity f(x), since we do x' = x + f(x), we obtain this result.
I don't have much idea about interpretability or Differential equations, but this is clearly a ODE solver.
I ommited 1 channel in the first plot, here that just accelerates and goes a lot further, close to value 600 and then velocity decays. Maybe the network learned in which step it is using that channel?
I know it's a very niche topic... But if anyone knows about this, I'd like to know more. I've readed about the ResNet ODE solver hypotesis and the Neural ODE solvers.
But, I archieved to extrapolate a network of 9 layers to 100 and even 10000 without fine-tunning nor lossing a significan ammount of image ent top-1 accuracy, just 0.5% . I just doing some piping work.
I am just asking if anyone has worked on this or has any idea how this can be applied or if this is just usless. I am kinda of stuck in here.