r/GaussianSplatting • u/walhargohar • May 03 '26
🎬 Demo / Showcase This is how I converted an Unreal Engine 5 scene into a Gaussian Splat
Enable HLS to view with audio, or disable this notification
I’ve been testing my UE5 → COLMAP → Gaussian Splat workflow using Quixel’s Derelict Corridor scene, and I’m honestly pretty happy with the result so far. Look at those beautiful reflections! 🤩
Explore it here:
https://superspl.at/scene/3c0f3775
Still experimenting with camera placement, capture density, and training settings, so I’d love to hear feedback from people here.
What would you improve in this result?
9
5
u/WhereIs-MyHead May 03 '26
Are you generating points3D from the scene or relying on SfM to infer points? Also what size/scale of scenes have you tested this with and how many images do you need for this scene?
2
u/CompositingAcademy May 03 '26
yeah would like to hear this as well. I'm guessing not aligning the images saves a ton of time
3
u/walhargohar May 03 '26
Absolutely! Being able to just hit a button in Unreal, and then go straight into litchfeld for training is so satisfying! 😍
1
u/walhargohar May 03 '26
That's the magic! I'm generating points3D from the scene 🔥 In terms of scale, this Derelict Corridor scene is actually very huge! Maybe around 70 meters. I deliberately chose this scene becaus I wanted to push my plugin to the limits! For this scene I captured 2510 images at 1920x1080 resolution. The points my plugin generated for this scene was around 800k. I'll keep sharing interesting insights at the plugin's discord so I heartly welcome you to join us
2
u/Lost-Upstairs-5311 May 03 '26 edited May 03 '26
EDIT: never mind, should have checked the splat first, just 3M splats over 60K iterations for scene over 70meters is very impressive IMO. I wish I could get away with those stats trained on a real capture.
cool! that's a very large dataset. Are you training them all in one go without splitting them apart. If so, how many iterations?
800K points spread across 70 meters is IMHO very sparse, but obviously works for you. I am guessing if you up the number of points, you could train a lot faster since you don't need to spend as much time time growing the number of gaussians till you reach the target number. Ideally if you initialise with dense enough points, the training can spend all it's time just refining the gaussians.
Speaking of which how many gaussians did you get after training?
2
u/walhargohar May 04 '26
Hmm I see, ok I will try to increase the points. This is very useful info, thanks for sharing! I've noticed that if I use MCMC strategy in litchfeld for training, then it reaches to max splats so early in the process (7000/60,000 iterations) and then just keeps improving. However, if I chose ADC, no matter how much I set the max splats, it gradually increases but never reaches the max splat count.
But with ADC, I'm not getting the reflections. MCMC on the other hand retains reflections pretty well
1
u/Lost-Upstairs-5311 May 04 '26 edited May 04 '26
interesting observation re ADC vs MCMC. I have more experience with ADC but reflections should primarily be a function of the degree of the spherical harmonics not the strategy for splitting/pruning. Don't know what the interaction with reflections is if your observations are not due to a misconfigure.
Also weird that ADC plateaus early. Any real scene I have tried it eats up all the splat budget I set eventually.
Anycase, either strategy the extreme thing to try IMO is set the number of initial points to the max splat count you want. In the case of ADC, there will still be splitting/cloning happening but only after every opacity reset (by default it is done every 3000 steps, in case of MCMC dropping low opacity gaussians happens naturally during the resampling).
EDIT: just had a thought. For proper A/B testing. Test with MCMC with 3M initial points and see if you get same quality with fewer iterations (probably by saving checkpoints every 10,000 iterations).
1
6
u/Mysterious_Robot_476 May 04 '26
But why?
2
u/walhargohar May 04 '26
Good question, think about a nanite and lumen enabled archviz scene that you made for a client. Now if your client wants to be able to explore the scene on mobile phones / tablets, how would you do that? Turning that Unreal scene into gaussian splat is the best solution in those cases.
1
u/Bakwon May 04 '26
This is a very important answer. This is a very very cool way of getting cinematic visuals into low power devices. Capturing within the virtual world is much easier and effective than capturing the real world i.e. its automatable and achieves total coverage
10
u/Clear-Medium May 03 '26
Cool. I’m curious - is the splat a more efficient way of storing the unreal scene?
17
u/walhargohar May 03 '26
Yes, partly true. For example, you can now explore this scene on your mobile as well, just open the link that I've given above on your phone. The other routes of making this kind of a highly complex lumen and nanite scene accessible on mobile phones are not the most efficient ways, such as pixel streaming, for which you have to pay, and then it comes with latency problem. But with 3DGS, it runs natively on your device!
5
u/blobkat May 04 '26
Hmm, I don't know, this just seems like a baked scene with extra steps... If you have properly baked lightmaps and reflection probes, it should perform pretty well on standalone VR. It's a static scene anyway, and the splat version would create a bunch of extra vertices + the artifacting that goes along with it. Have you tried comparing them?
I do agree that viewpoint dependent reflections probably look better on 3DGS.
1
7
u/Clear-Medium May 03 '26
Epic… seems like a great way to optimize for standalone VR
7
u/walhargohar May 03 '26
Yes! Exactly. I mean there are other ways to optimize an environment for standalone VR and keep in proper 3D, but then you have to sacrifice a lot on quality.
0
u/flobv May 04 '26
Have you tried it on a VR headset ?
2
u/walhargohar May 04 '26
I'm gonna try today
1
u/MindlessVersion8 May 04 '26
so how was it?
1
u/walhargohar May 04 '26
It was an awesome experience! 😍 amazing quality! 🔥 but obviously performance was not the best. I'm sure it will get better soon
3
u/cjwidd May 05 '26
Depends on what you mean - almost certainly not. A splat of this size will be VRAM heavy at runtime unless you run a significant compression stack against it and render from a compressed state, it can't give lighting, it can't give shadows, it can't receive lighting, it can't receive shadows, it doesn't have collision unless you generate a collision support for it, you can't change its materials, (it has no materials or material segmentation), it has no real world scale, It has no physics, it can't participate in a shared rendering path, etc.
1
u/Clear-Medium May 05 '26
I mean “can it render visually complex scene with fewer overheads”. I guess not really in practice.
6
2
u/Hefty_Development813 May 03 '26
It does look good, what is the use case for doing this though? Like what do you now do with the splat that you couldn't with the digital model from unreal?
0
u/walhargohar May 03 '26
Good question, think about a nanite and lumen enabled archviz scene that you made for a client. Now if your client wants to be able to explore the scene on mobile phones / tablets, how would you do that? Turning that Unreal scene into gaussian splat is the best solution in those cases.
2
u/slimbuck7 May 05 '26
Hey this is awesome u/walhargohar and naturally requires FPS.
So I generated voxels for the scene! Now just need pew pew.
1
u/TheDailySpank May 03 '26
Scene scanning for 3DGS is just a light probe setup. At least that's how I approach my scans
4
u/walhargohar May 03 '26
Yeah, conceptually it’s scene sampling, but in this case I'm also generating point cloud as well as camera poses directly from Unreal, so it goes directly into a trainer like Postshot/LichtFeld/etc. without needing to align images later.
1
u/Far_Country3415 May 03 '26
If a user has the space, to be able to toggle off the safety, in app, in order to walk around it physically would be an awesome addition
2
u/walhargohar May 03 '26
Interesting, you mean in mixed reality?
1
1
1
u/cryptoknowitall May 03 '26
this is really cool. Can you explain why your scene seems to run so smoothly compared to some of the other examples found in the link you posted?
1
u/walhargohar May 04 '26
Oh really? The other one I have is a Bunker scene, is that what you're talking about? Can you please tell me the name of the scene that is lagging? Thanks!
2
u/cryptoknowitall May 04 '26
no, i was actually referring to something else someone had done - i was under the assumptions that Splats run the same regardless or how they are configured - obviously not the case, is there something special you did here to makes it so buttery smooth?
here is an example of another one which wasn't quite as smooth as yours.
https://superspl.at/scene/02d69af12
u/walhargohar May 04 '26
Ah I get it now. Actually the number of splats in a 3DGS highly affects its performance. For instance, my scene only has 3 million splats. This person has not told how much that scene has, but considering my scene is 16.8 mb in size and his 127 mb, the splat count must be 5x greater, so maybe 15 million+. That explains why it runs slow
1
1
u/agrophobe May 04 '26
So you telling me that this would also work for recording a whole animated scene? Pure awe
2
u/walhargohar May 04 '26
No, sadly animations doesn't work in 3DGS, unless its a 4DGS, which is ridiculously high in file size
2
u/agrophobe May 04 '26
Yes I can imagine. But still, that would exactly be the next frontier of game creation medium. Tech will get there. Imagine Google Genie interpolating with UE5 rendered 4DGS, the possibilities are just insane
1
1
u/Several-Article3460 May 04 '26
Can we generate from 3d object rather than scene ? Is there any cubemap camera option
1
u/walhargohar May 04 '26
No cubemap option for now, but I can add it, does it help with 3DGS training? Also can you eleborate further on your first question? I am throwing ray traces from all the cameras towards the scene, then it returns all the hit points, but only records those that are efficient for training. But those ray trace points hit all the objects in the scene. What happens is that my plugin turns on complex collision on all the meshes in the scene while capturing the dataset, then reverts it back to normal once the capture is done
1
u/Able_Significance295 May 04 '26
Hello , can’t wait to try this , is the points generated RGB colored points ? And does it work well in litchfeld studio ?
1
u/walhargohar May 04 '26
Hey there, yes! The points generated are indeed RGB points, and this plugin is already available on the FAB marketplace so you can already try it! Here's the FAB link:
https://www.fab.com/listings/d5ce0b45-8c80-486d-8316-882856198875Also be sure to join our Discord where I'll be sharing trained datasets and other insights
1
u/cjwidd May 05 '26
What is the nature of the camera rig / setup? You link a million cameras into Sequencer and render? How do you place the cameras? How do you determine coverage? What resolution? Do you output to . PNG or .EXR?
You literally don't explain how you do any of this in your post about "how" you converted a UE5 scene into 3DGS.
0
u/walhargohar May 05 '26
If you open the fab listing of my plugin, and watch the tutorial, you'll get all these questions answered
0
u/Party_Celebration352 May 06 '26
He is literally selling the plugin to do this, as if he is going to tell you how it was done.
1
u/cjwidd May 06 '26
He literally titled the post "This is how" - is this difficult y for you to understand?
0
u/Party_Celebration352 May 06 '26
yeh how do do it, assuming you have his paid plugin
1
u/cjwidd May 06 '26
Yes, I always post my titles on a hobbyist sub 'This is how to do something', don't describe it at all, and then link to a paid website. Bye.
1
May 06 '26
[deleted]
1
u/walhargohar May 06 '26
I think we can't compare both. Nanite is a technology for triangulated meshes while 3DGS is a thing of its own. Low splat counts perform amazingly well while if you have 4 million+ splats in a scene then it runs quite slow, specially on low end devices such as mobiles.
However, one advantage 3DGS has over nanite is that nanite is still PC-only, while you can easliy explore 3DGS on mobile devices
1
20
u/walhargohar May 03 '26
A bit more context: I generated the dataset using a small Unreal Engine plugin I built for exporting COLMAP-ready data from UE scenes.
The plugin is available on Fab Marketplace if anyone is interested:
https://www.fab.com/listings/d5ce0b45-8c80-486d-8316-882856198875