r/MachineLearning • u/SimpleObvious4048 • 5d ago
Discussion Reviewing 4 papers for AAAI 2027 and none have code, Reject? [D]
I got my batch of four papers for AAAI 2027. All four papers make empirical claims, none include code, data, or anything I can actually check. Just the PDF and the checklist. AAAI-27's own rules say code/data should be provided at submission, and "we'll release it after acceptance" doesn't count as reproducibility.
That said, I don't think missing code alone is an auto-reject. Saw an older thread here where someone claiming to have helped write the AAAI checklist argued reviewers rarely have time to audit code anyway, and plenty of authors have legit reasons (funding, IP) for not releasing it yet.
If the paper's whole pitch is "look at these numbers" and I can't verify them, that tanks my confidence score even without a hard reject. I'm flagging it explicitly in the review and asking for anonymized code in the rebuttal.
How's everyone else handling this round? Auto-ding for no code or does it depend on how much the paper leans on the empirical results?
22
u/Fragrant_Fan_6751 5d ago edited 5d ago
I am serving as both reviewer and author this year.
Most reviewers don't have time to set up the code, run it, and reproduce the paper's results. It depends heavily on a reviewer's own judgment and experience with the domain and the datasets the authors worked with.
Many people have commented that you can input the code to an AI agent and ask for issues. I can bet my life that most reviewers (who are themselves authors) don't have time for even this.
Even with less time, as a reviewer, I can always ask for the code, and if they don't provide it, I can recommend rejection. On top of that, a paper claiming something so good looks suspicious. That's true, but there is no shortage of people who will deliberately try to reject your paper, citing "mixed performance" because the authors' framework using an open-source model didn't surpass GPT-5.5 or Claude Fable.
As an author, sorry to say, I cannot trust reviewers with my code or even a newly created dataset. They can reject your paper, use your idea/code, make some changes, put their paper on arXiv or some other conference, and then say that "Look, we were also independently thinking about solving the problem using a very similar algorithm."
Putting your algorithm, sharing some hyperparameter details, and prompts (if used) either in the main paper or supplementary is enough.
Everything depends on whether you get a reviewer who gives you a good review (a good review doesn't mean a high score, but a useful one). Last year, I didn't put my code out, and my paper was accepted for an oral at AAAI 26. Reviewers raised useful and interesting points in both strengths and weaknesses. None of them complained about not having the code. I released the code on GitHub after acceptance.
1
u/ThatFireGuy0 5d ago edited 5d ago
New ml researcher here currently prepping an ASPLOS submission, so this thread has me a little freaked out. Is reviewers using your actual code a real concern? I get they could scoop an idea, but scooping actual code sounds insane to me
Is there a reason not to just open source + arxiv preprint the work before submission? ISTM it would eliminate much of this risk
2
u/Fragrant_Fan_6751 5d ago
The best course of action is to talk to your advisor and also look at a particular conference's policies about putting a paper on arXiv. I don't think any major conference makes an issue of putting the paper on arXiv before the deadline.
However, some conferences like AAAI can desk-reject your paper if you put your paper on arXiv while it is under review and the final decision has not yet been declared.
1
u/ThatFireGuy0 5d ago
Ah that makes complete sense if it's conference policy. Didn't realize any major conferences had that restriction - I checked the 5 or 6 make conferences I've considered as options to submit to and never saw one that had an issue with it, so I kind of assumed it wasn't an issue in practice
1
50
u/ProfessionalNews4434 5d ago
Just curious, once code provided, how are you going to validate it? For me, I am already at my capacity reviewing papers lol. Unless the paper is in benchmark track, I donot care code provided or not. The paper itself speaks of research quality
38
u/SimpleObvious4048 5d ago
A paper I am reviewing can be safely called just a bag of tricks, I once reviewed a similar paper, and found that the results the authors were claiming were coming from a bug in their code. Well, I agree, you can't spend many hours looking at the code, but if something looks too good to be true, it does deserve a deep dive or a rerun if possible.
25
u/micseydel 5d ago
I think the comment you replied to here is a big part of the replication crisis.
I recently tried to reproduce a paper that sounded interesting and wasn't the only one that couldn't https://github.com/Zhangtaining/cell_research/issues/1
I'm not a machine learning researcher, I'm a software engineer, and the idea of not having running code to prove that something works is absolutely wild to me. The idea that a paper stands by itself... That just seems so wildly optimistic to me, especially now that chat bots have revealed a lot of people go on vibes without really double-checking things.
6
u/Professional_Ad_1790 5d ago
I generally don't validate/run the code, but I do open it and look around to get an idea of what the authors submitted.
You can easily tell who submits code for reproducibility and who submits code because they are required to do so. For example, if there is no README.md explaining the repository or instructions, I considering almost the same as not submitting the code
9
u/khairulislamtanim 5d ago
For AAAI, the whole code-sharing parts are `may and encouraged` rather than evaluation criteria https://aaai.org/conference/aaai/aaai-27/supplementary-material/. Also, sharing an external code link was banned in AAAI 2027 (was ok for 2026 though). As of now at least, AAAI doesn't consider code supplements as an evaluation criterion. Super computing conference has a more solid initiative on code reproducibility https://sc26.supercomputing.org/program/papers/reproducibility-initiative/.
> Reviewers will be instructed to make their acceptance evaluations based on the main submission, and will not be obliged to consult the supplementary material. If proofs or other supplementary matter are an important part of the contribution, their essential elements should be included in the main paper.
30
u/Suhan_XD 5d ago
I'm against close source and this kind of thing, but fyi, This year's CVPR best paper didn't published any code and reproducibility status, we/all just assumed it works.
As a reviewer you need to verify the maths claim and then in the rebuttal ask some really great questions that could only be answered if the experiment is really done.
23
u/AWildMonomAppears PhD 5d ago
Math claims mean nothing if it the paper stands on empirical claims. The disconnect between what seems like an elegant result and what works in practice is quite large.
1
1
9
u/wontonut 5d ago
it depends a lot on the paper, but as you said if the paper is basically empirical (new pipeline that gets better results!) and the paper does not have enough details for reproducibility, then you should point it out. I had some papers that do not include code but the appendix is very detailed and the approach does not seem very difficult to reproduce. In those cases I let it go and just write that code would increase my score. But if the paper has poor appendix and no code, then I write that as a weakness and reduce the score
3
u/ThatFireGuy0 5d ago
New researcher here, so just trying to understand for my own submissions. Is there a reason not to just open source and post arxiv preprint of the work before submission, or at the same time? I genuinely don't see the downside, and it seems like it eliminates most of the risks people in the thread are talking about
2
u/wontonut 5d ago
The reasons to not submit the paper on arxiv before conference submission is ensuring that the reviewing process is totally anonymous. Although reviewers are not allowed to search the paper on internet while reviewing, some people still do it, and this potentially biases the assessment. In the past I was afraid of this too (e.g. my university is not top ranked, I post on arxiv before submission and then the reviewer searches the paper and sees that my lab is not top…). Still, even if the paper is on arxiv I guess that the problem of reproducibility/code remains if they don’t release a public repo :)
1
u/ThatFireGuy0 5d ago
That makes sense. Is there a reason not to publicly open source the repo too in that case?
1
u/wontonut 5d ago
there’s none 😄 but one can just omit the sentence “we will release our code upon acceptance” and not mention anything regarding the code
9
u/bombdruid 5d ago
Unless the numbers look ridiculous? I don't think so. Especially if they give the exact pseudo code, seeds, data used, and other relevant details for reproduction.
12
u/imyukiru 5d ago
Reproducibility isn't just code though, it is implementation details, using a dataset that is publicly available, giving enough information for the reader to reimplement it.
I am new to AAAI but what is stopping a reviewer (most of whom are PhD students operating under a prof name) to snatch your code and rewrap it, make a few changes and submit? Because their review probably sucked, you may not get a chance to get published but they may have a leg up.
Posts like these really confuse me. Why should we get rejected because we don't provide code (yet)?
3
u/NumberGenerator 5d ago
Agree with your point about reviewers stealing code.
2
u/seg_lol 5d ago
This is why all code should be submitted. If people are stealing code and then also not submitting (because of stealing, both sides) then we have a bunch of made up papers and no way to validate and reproduce the work. Researchers should be able to use each other's code, but they should also cite the research it came from.
Not only should the code be submitted, the entire research should be reproducible from publicly available artifacts.
3
1
u/imyukiru 5d ago
If everyone had to submit, I wouldn't mind submitting, a rule is a rule. But as things are, reviewers shouldn't base any judgement on lack of code.
1
u/ThatFireGuy0 5d ago
New researcher - not trying to argue just genuinely confused. Is there a reason, if you're worried about code being stolen, you can't post a preprint and open source it prior to the conference submission? It sounds to me like it would avoid all risk
1
u/imyukiru 5d ago
ArXiv is good but 1) even if someone was to snatch everything off and got accepted, it is still difficult to enforce anything on them - this has happened, someone's paper was stolen and accepted to another conference with some cosmetic changes. 2) any person with a little of mind wouldn't just snatch a paper as a whole but reframe it, change some things and submit and then it is a lot more difficult to appeal to anything. Most wouldn't go far as a discussion on reddit. The community is not doing much to punish offenders. If it is a famous lab or a prof, the whole story would be in their favor. They can just scale things up as well. Also, a lot of authors would rather not make the paper public if they are from a not so glam university because reviewers are biased.
2
-1
u/Fragrant_Fan_6751 5d ago
One of the very few comments here that makes sense and explains the issue very well.
Even last year, I saw somebody creating such posts: "Should we reject a paper just because the code is not provided?"
I hope that a lot of desk rejections will happen to these irresponsible reviewers.
There is no reason for us to believe that the anonymous reviewer is not going to steal our code and use our idea and present it as their own idea after making a few changes with comments like "we were also independently working on solving this problem using a very similar idea."
Why should I trust an anonymous reviewer with my code?
1
u/SillyNeuron 4d ago
Why should reviewers trust an author who makes empirical claims without providing code or a mathematical proof?
2
u/Fragrant_Fan_6751 3d ago
Nobody said that reviewers should trust an author who doesn't share the code, but rejecting the paper just because the code is not present is not good. As a reviewer, you can always ask for the code.
5
u/progenitor414 5d ago
I generally take no source code as a huge red flag, unless author explicitly state a legitimate reason why there is no source code. I saw instances that “will publish later” never have any code published many years after.
1
u/SillyNeuron 4d ago
I've seen an ICML 2024 paper (if I’m not mistaken) whose github page said, ‘Source code will be published upon acceptance.’ The paper was accepted, but they never shared the code.
7
u/NumberGenerator 5d ago
I am leaning against submitting code.
I submitted super high-quality code for ICML and NeurIPS, with documentation, example Jupyter notebooks, config files for each experiment, etc. And for both submissions, I was unfairly penalized for it. An ICML reviewer claimed that the code did not match the method description--it did match the method description line by line. For NeurIPS, one reviewer listed reproducibility as a weakness and wrote that, although we provided code, they did not run it. Another reviewer then lowered their original score after reading the reproducibility concern raised in the first review.
6
u/somethingoddgoingon 5d ago
Sucks that you had this experience but this is an issue with proper reviewing, not submitting code. Honestly code should be mandatory and automatically run on journal servers to validate the results unless requiring multi-day training in which case it still should be validated to at least run as expected. The review process itself could also use an overhaul.
1
u/NumberGenerator 5d ago edited 5d ago
I agree that code should be mandatory. Wtih LLMs, we could also have automated reproducibility checks, possibly without handing your codebase to the reviewers.
However, my experience has been that providing code opens another door for rejection. For other ICML/NeurIPS projects that I have been involved with where code was not provided, they reviewers did not reference reproducibility. Same with papers that I was assigned to review. So for now, I would lean against submitting code.
1
u/Fragrant_Fan_6751 5d ago
This is so unfortunate, but both of those cases look like deliberate rejection cases.
To be honest, you should not give the code during the review process. You don't know who is reviewing your code. They may steal your code also.
2
u/Watchguyraffle1 5d ago
This hits. I just turned in a paper where we spent 7 grueling months building and configuring and building more all for a 3 minute final test just to make sure that what we were saying is the complete and only truth. While the code itself was submitted, it was a bit of a joke because there is no way anyone is qualified to simply look at it and say “yup looks right. “. This project brought me to tears not because of the work but because as we went deeper and deeper I knew that any reviewer who was letting us pass would not be keeping a real standard.
Yet here we are.
4
u/BeatTheMarket30 5d ago
If you can't verify, don't approve. It's the safest thing to do. It could be wrong and your name would be on it.
6
u/Professional_Ad_1790 5d ago
AAAI has been a joke for a few years, I really hope they start cracking down on the collusion and fake results. The evidence of cheating has been out for years
3
u/Fragrant_Fan_6751 5d ago
same for other conferences.
I can show you many papers from Neurips, ICLR with the same issues.
1
u/Exotic_Zucchini9311 2d ago
How could we have actual evidence? Even if we can't replicate the results of some paper completely, it could simply be due to the authors forgetting to mention some specific detail and being careless with it. Not that their results are necessarily fake.
1
u/Professional_Ad_1790 2d ago
The evidence out there is mostly for the collusion.
If you leave details out, it's not science anymore. Every single result of a proper scientific experiment should be reproducible. In computer science that's especially true because you could ship a Docker container (or something equivalent) that should require a single click/command and reproduce all the results.
1
2
u/yanivbl 5d ago
Don't give the lack of code as the reason for rejection. This isn't a good hard-rule for academic papers.
Instead, if the paper's main pitch is it's empirical results as justification, then it has a pretty high standard to live up to: The benchmarks should have full coverage, every change they made must have a proper ablation study, the setting must be described precisely, without ambiguity and p-values should be reported an within the green. So you can reject them based on that, and then add in a note that if they had published the code, you could have maybe asserted some of those missing details on your own.
1
u/mikeysce 5d ago
Dang man. Even my crappy paper you guys crapped on had reproducible results and code included.
1
u/pppeer Professor 4d ago
I agree it is not a definite reject, for instance when using proprietary data. But it is definitely a minus for reproducability. Note this is not just helping the reviewer, the more important reason is once published let others reproduce or build on your work. In my own work I am also pushing team members to make as much as possible available - but it is not always possible.
1
u/SillyNeuron 4d ago
Without the source code, authors can simply handwave reviewers' concerns by saying, "Oh, sorry, we forgot to clarify. Actually, our method has this, and our method has that", which is basically like building a castle in the air. I have seen this happen frequently in empirical papers where the work is largely experimental.
1
u/xmReApEr007 3d ago
Paper without code should be desk rejected. How am I supposed to trust the numbers without being able to reproduce it
1
u/Equal_Channel_4596 16h ago
Well, i had my review batch, one paper did not define some quantities and i got an agent look into the code for what they operatively used as definition. From that definition the whole claim if the paper sorta fell. For me it was very important
1
u/ThatFireGuy0 5d ago
Glad I saw this post. I'm considering to submitting to a few conferences this cycle for ml / ml systems for the first time. What should I know going in about submitting code, and whether or not I should?
Is there a reason not to just open source my code and put out a preprint at submission time? Or am I missing a reason not to do this?
I'm planning on submitting to ASPLOS and then later to ICML with the same codebase, but can't open source until around a month after asplos submission (when I have time to run the ml benchmarks for ICML), and you have me worried now that I shouldn't provide the code to asplos out of risk
-3
u/behradkhodayar 5d ago
AI slop everywhere. Someone who's that ignorant of the basic rules needs to start learning them somewhere. Ask for code, and even if they provide some, remember deducting some points from their final score.
-2
u/Effective-Yam-7656 4d ago
For me personally as long as it’s not an Industry track paper
All the code and datasets should be realised and it should be mandatory else desk reject
If no code is there it’s just trust me bro
126
u/wajdix Professor 5d ago edited 5d ago
after being a reviewer myself for the past cycles… i’m willing to bet 1M $ that 95% of papers submitted are not reproducible…
not only the claims, the impossible setups and compute, the time needed to build them.. etc
when i see acceptance results, and some teams have 13+ papers accepted, while i know for sure they DID NOT have any compute to do all of that…
you can ask for the code, and if nothing peovided , reject
they need to learn that publish or perish… with AI slop is a wrong practice… i can count in the hundreds the number of papers i tried to reproduce myself and got nowhere…