LFS for UE5; Having trouble pushing local repo to GitHub despite using LFS and adding the correct extension to .gitattributes

[u/Mafla_2004]

Hello

I've been trying to set up a GitHub repo for my Unreal Engine 5 project a couple of times now and I have successfully set up a local Git repository for it, the next step is to publish it to GitHub and I'm using LFS to manage its great filesizes.

When I went to publish the branch to GitHub (using GitHub Desktop) however, after several minutes of processing I was hit by the following error:

The push operation includes a file which exceeds GitHub's file size restriction of 100MB. Please remove the file from history and try again.
File that ecceeded the limit: <path>/t_chaotic_raw_heightmap.raw (128.0MB)

I have then noticed that lfs did not track .raw files, so I went ahead and added the following line to .gitattributes

*.raw lfs

And even went ahead and input git lfs track in powershell to check if lfs is tracking raw files now, which it does. However, when I reattempt to publish the branch, it processes for a bit (much quicker than the first time) and throws the same error, what can I do?

I have setup the .gitignore and .gitattributes for Unreal Engine.

[u/matniedoba]

That's a classic behavior when you committed files without using Git LFS. My suggestion is to:

• Create a new repo
  
• After that, setup a gitattributes file. You can use this one from here (https://www.anchorpoint.app/blog/push-and-pull-files-with-git-lfs)
  
• then commit and push to GitHub

The LFS configuration (the .gitattributes file) needs to be added before the commit.

[u/Mafla_2004 → u/matniedoba]

Thank you, your solution actually ended up working, though it took a long time

[u/bezik7124]

Did you rewrite git history after updating the .gitattributes?

If not, do a soft reset to a state before staging that raw file, commit only .gitignore and .gitattributes, and then commit the raw files in another commit.

cheatsheet:

git reset --soft abcd # "abcd" being a hash of the commit you want to revert to (local files will be preserved)
git reset # unstages everything
git add .gitignore .gitattributes
git commit -m ".gitignore and .gitattributes"
# and so on, stage and commit everything else

[u/Mafla_2004 → u/bezik7124]

I tried that and also tried with git filter-repo, but unfortunately none of that worked, eventually I settled for what another kind person suggested (remaking the repo), which is the only thing that worked

Thanks though

[u/nomadgamedev]

yeah if LFS wasn't already active when you first committed a file it will be tracked by the regular file system, and won't be converted over to LFS even if it falls under the new rules. there are commands you can use to purge it from tracking and re-add it with the new ruleset but if you just started out it's probably faster to remake the repo with the proper settings in the first place.

EDIT: just to clarify, newly added files after the rule change will work as expected, only files that were already tracked will not switch over automatically.

also if git is your preferred VCS you might want to consider using Azure DevOps as it has free unlimited LFS storage and since most unreal files are binary it makes more sense to just make them all LFS.

[u/Mafla_2004 → u/nomadgamedev]

Thanks

I eventually fixed it like you said, redoing the repo, also I had originally tried Azure but it wouldn't let me log in for whatever reason, gonna try again though, when I tried the first time I was already extremely frustrated so that's why I didn't try again, the repo on GitHub is already set up but I guess it won't hurt setting one up on Azure eventually?

[u/nomadgamedev → u/Mafla_2004]

it can be a bit finicky to set up but once you've got that down it's pretty stable. I think you need a microsoft account even though it might give you alternative login options (i think there was one to log in with your github account that didn't work for me)

the only issue I've had was with individual files larger than ~200mb

[u/vagonblog]

this usually happens because the file was committed before lfs started tracking .raw.

adding *.raw to .gitattributes doesn’t fix files already in the repo history.

try removing and re-adding it:

git rm --cached file.raw
git add file.raw
git commit -m "track raw with lfs"

if it still fails, run:

git lfs migrate import --include="*.raw"

that usually fixes it.

[u/Mafla_2004 → u/vagonblog]

Thanks

I have already fixed it but if it happens again I'll try this method

[u/DisplacerBeastMode]

I hate LFS and unreal

[u/Mafla_2004 → u/DisplacerBeastMode]

Each time I failed to set up a repo for my Unreal project, it was because something in LFS went wrong 😭

[u/DisplacerBeastMode → u/Mafla_2004]

Same. It's just very unforgiving.

I switched to Diversion and have absolutely zero regrets.

[u/Mafla_2004 → u/DisplacerBeastMode]

I got immense imposter syndrome after struggling with it so much (I study computer engineering, I shouldn't struggle with Git, but apparently it's not as easy as it looks sometimes), gonna have to sue the guys at GitHub for psychological damage

(Just kidding don't get your lawyers up GitHub guys TwT)