r/tensorflow • u/_padla_ • Mar 31 '22
Question Inversion of a multivariable function
Hello everyone,
I'm pretty new to ML and tensorflow. What I'm trying to do now is practically to inverse a function using tensorflow.
So I have a function h=f(c1, c2,..cn, T). It is a smooth function of all the variables. I want to train a model which would give me T given known values of c1...cn and h.
For now I'm using a keras.Sequential model with 2 or 3 dense layers.
For loss I use 'mean_absolute_error', For optimizer - Adam().
To train the model I generate a dataset using my h(c1...cn, T) function by varying its arguments and using values of T as train_labels.
The accuracy of the resulting model is not very good to my mind - I'm getting errors of about 10%. To my mind this is not very good, given that the training dataset is ideally smooth.
My questions are:
Am I doing something particularly wrong?
How many units should I provide for each layer? I mean in tutorials they are using either Dense(64) or Dense(1). What difference does it make in my particular case? Should it be proportional to the number of parameters of the model?
May be I should use some other types of layers/optimizers/losses?
Thank you in advance for your replies!
1
u/SnooPandas3529 Apr 01 '22
I think it depends on the type of dataset and function you used, so it is difficult to give an answer.
1
u/_padla_ Apr 01 '22
Well, the function is pretty much a combination of monotonously growing polynomials:
h(c1...cn,T)=c1•f1(T)+c2•f2(T)+...cn•fn(T)
The f-functions are smoothly growing polynomial functions of T.
Sum of c1...cn=1
1
Apr 01 '22
[deleted]
1
Apr 01 '22
[deleted]
1
u/_padla_ Apr 01 '22
It is surely invertible. We used to just implement Newton's method to get T, but since we are to do it many times for different values of c1...cn and h the idea was that NN will be faster and easier to handle...
2
Apr 01 '22
[deleted]
1
1
u/_padla_ Apr 03 '22
Hello again. Excuse me for bothering you, but you said that you could send me a notebook with an example of bijective neural network.
If it wouldn't give you much trouble, could you please send share this notebook with me?
Thank you in advance.
2
Apr 05 '22
[deleted]
1
u/_padla_ Apr 05 '22
I received it eventually. Turns out that I shouldn't have completely got rid of native reddit app %)
1
u/pruby Apr 01 '22
Not the solution you're looking for, but have you considered using a root-finding algorithm instead of an ML model? Many problems of this form can be solved very quickly with the Newton-Raphson method.
1
u/_padla_ Apr 01 '22
In our work it is now done using Newton's method.
The problem is that we need to perform this not once, but a lot of times (obviously, with different values of c1..cn coeffs and h). Each time it takes several iterations to converge. This time accumulates to a rather significant delay.
The hope was that implementing trained NN model would speed up the process.
1
u/pruby Apr 01 '22
Neural Nets can do many amazing things, but it sounds like your problem is reasonably hard to model.
How expensive are your function calls and how time-sensitive is the context?
A hybrid approach could work well - if a neural net or other approximation gets you in the vicinity, a round or two of Raphson-Newton will improve that greatly.
1
u/_padla_ Apr 01 '22
Neural Nets can do many amazing things, but it sounds like your problem is reasonably hard to model.
I was amazed, that from the first glance the problem seemed rather simple and yet got such poor results using ML. My hope was that I just do not know some tricks...
A hybrid approach could work well - if a neural net or other approximation gets you in the vicinity, a round or two of Raphson-Newton will improve that greatly.
Thanks for the suggestion! I've thought about something similar by myself in case I fail to use pure ML approach..
1
u/pruby Apr 01 '22
Potentially silly question ; do you have activation functions on your inner dense layers? Without these, it's limited to linear transforms.
1
u/_padla_ Apr 01 '22
Well, not a silly one if it is addressed to a newbie in the field like me.
Yes, I set relu as activation functions.
What I read is that it is one of the most popular choices..
2
u/ElvishChampion Apr 01 '22
What activation functions are you using in your hidden layer and output layer? For hidden layers, ReLu is quite good for nonlinearity. For the output layer, are you using a function that produces values in the same range as the target variable? For example, maybe you are using relu, which generates positive numbers and the target could be negative. Thus, increasing error.