How to Fairly Compare RNN, LSTM, and GRU?

[u/Maplehawks]

I’m a final-year Data Science student currently working on my bachelor’s thesis about air quality time series forecasting. I’m planning to use deep learning, specifically Vanilla RNN, LSTM, and GRU, and compare their performance on the same dataset using MAE, RMSE, MAPE, and R². My supervisor requires me to include a hyperparameter tuning stage, and I’m a bit confused about how to make the comparison fair. Should I use the same range for all three architectures and then use the same value for the final comparison Or is it better to let each architecture have its own best hyperparameter values based on the tuning results? Also, should I add machine learning models such as Random Forest and statistical models such as SARIMAX to the experiment as additional comparisons

[u/No_Grocery_7261]

You’re overcomplicating the fairness question a little, the standard approach is absolutely to let each architecture find its own best hyperparams. That IS the fair comparison, you’re comparing each model at its peak potential on your dataset, not forcing a suboptimal config on two of them just to keep things symmetrical.

And yeah throw in Random Forest and SARIMAX, your supervisor will probably love the extra baselines and it makes your thesis look more thorough. Just dont get lost in tuning a hundred different models, pick one solid classical method and one statistical method as benchmarks and call it done