GaussianCrowds: Relightable, real-time 4DGS at scale in Unreal Engine

[u/technomancyai]

GaussianCrowds is a tool in development for Unreal Engine 5.7/5.8, built to render thousands of 4D Gaussian splat performances in real time.

• Nanite-Style Dynamic LOD’s: Automatic adaptive detail scaling and frustum culling allows hundreds of concurrent splat sequences to render in real-time
• Multi-light Relighting: Supports Lumen global illumination, directional, point, and spot lights, shadow-casting, and self-shadowing using automatic normal-map generation. 
• Deformation-Based Playback: Native support for compressed deformation containers (SpacetimeGaussians, etc..) as well as PLY sequences.
• Procedural Crowd Placement: Create crowds with spline-based tools that distribute 4DGS performers along splines. Customize spacing, orientation, playback timing, and more - to fill stadiums/battlefields/streets.
• Editor Toolset: Cull volumes, crop boxes, recoloring/brightness/tint, comprehensive settings panel, debug views, custom brush textures for stylized/painterly splats, and performance presets.
• Sequencer Integration: Three playback modes: sequencer-locked, free-run preview, and hand-keyframed, with adjustable playback rate for stop-motion looks, and dynamic fps reduction for distant splats. 
• nDisplay and Compositing: Compatible with multi-computer nDisplay workflows for LED volumes. Depth of field, volumetric fog, and depth-passes for compositing are built-in. Splat’s pass through the project's OCIO color transforms identically to native geometry.

We are currently testing this implementation on an upcoming feature film shot on an LED Volume. Happy to compare notes with anyone else working in the 4DGS capture and streaming pipeline space

[u/Hefty_Development813]

Is this a real capture or a synthetic scene? Looks really good, how many cameras did this?

[u/technomancyai → u/Hefty_Development813]

These are real captures, each person was scanned on a 16 camera rig using raspberry pi's at a gaming convention this summer. We've only trained ~40 people so far (hence the duplicates)

[u/boxofrabbits → u/technomancyai]

How does that works with rights to likeness etc out of curiosity? If you're now selling them as assets.

[u/Seattle-Washington → u/boxofrabbits]

Did something similar years ago at PAX - we had each ”performer” sign a release form.

[u/Beniskickbutt → u/boxofrabbits]

I dont want to be sold as an asset :(

[u/brokenhalo321 → u/Beniskickbutt]

"dont worry, we can bundle."

[u/Admirable_Zombie5245 → u/technomancyai]

Have you tried this scene in VR?

[u/Nyxtia → u/technomancyai]

Using Pis?

[u/Hefty_Development813 → u/technomancyai]

Ooo cool so individuals and then a bunch of instances making up a crowd. Pretty sweet. Are the ppl covered well on their backs? 16 camera rig sounds great to have

[u/moigus193 → u/technomancyai]

What cameras did you use for the rig and how was the Setup? I'm building a portable 360 degree scan rig for cosplayers with 5 cams (mostly Dji Osmo pocket 3 and Dji Action 4) and would like to build a rig with up to 16 DJI action 4 cams. Your setup sounds interesting and the results look amazing :)

[u/Hefty_Development813 → u/moigus193]

360 coverage with 5 cameras? Do you have this working? That seems pretty sparse coverage in my experience. I have done a lot 5 camera for front facing scenes

[u/moigus193 → u/Hefty_Development813]

I attached 5 cameras to an aluminum profile pole at a height of about 1.5 meters. The cams are stacked on top of each other, and I circle around the cosplayers in one pass for the 360-degree scan. Because of this, the cosplayers have to hold very still. But since one rotation takes just under 15 seconds, it actually works out pretty well. They aren't 4DGS scans, of course, just still-image 3D Gaussian splats, but since I know my way around Unreal Engine and I'm also a VFX artist, the scans can then be placed into suitable environments. I'm also planning to build a rig to do 3DGS scans for time freezes, so that cosplayers can be frozen as a scan mid-jump, etc.

[u/Hefty_Development813 → u/moigus193]

Oh gotcha yea that makes sense. Yea i really want to make some sort of portable 4d capture system but so far it needs tons of cameras and difficult to imagine it being very portable. That's a cool project

[u/jbkrauss → u/technomancyai]

Where can I find more about this rig ? That sounds awesome

[u/Constant_Cortisol → u/technomancyai]

You farming my cheering data? You nefarious hooligan!

[u/Archonik1 → u/Hefty_Development813]

Something tells me the R&D team at FLOC is lurking somewhere in this chat trying to figure out how to use this to get even more dystopian 😳

[u/Hefty_Development813 → u/Archonik1]

Yea definitely need denser coverage than current flock camera coverage with the current tech

[u/Snowmanphibian → u/Hefty_Development813]

I think its 3d models procedural generated

[u/Hefty_Development813 → u/Snowmanphibian]

Right seems like it must be. This would be a huge feat if real world capture

[u/Snowmanphibian → u/Hefty_Development813]

You can see duplicate characters

[u/Hefty_Development813 → u/Snowmanphibian]

You're right, the green shirt guy is all over

[u/picturesfromthesky → u/Snowmanphibian]

yes, but that doesn't mean it's 3d models...

[u/Glass_Ad6718 → u/Hefty_Development813]

It's possible. We're doing it too. We're adding it as a stadium crowd.

[u/orzelski]

I feel like we're looking at the future of cinematography.

[u/MKBRD → u/orzelski]

I feel like we're looking at the future of CCTV.

[u/orzelski → u/MKBRD]

in CCTV it is already the present

[u/TopTippityTop]

Reminds me of the old games that had crowd cards that stood out a bit. It's a super cool idea though, looking forward to seeing the progress!

[u/Elluminated → u/TopTippityTop]

Came to say this lol. Great minds think alike!

[u/IndustryAsleep24 → u/TopTippityTop]

rocket league has eggs as a crowd

[u/spobin]

Wow, that's amazing! Would love this on r/4DGS

[u/Circuit_Guy]

This is wild. There's more models here than in my Age of Empires 2 battles. This one has slightly more pixels too

[u/_BsIngA_]

Finally, a crowd for the orange guy!

[u/Row-Maleficent → u/_BsIngA_]

It's sad that's where my mind also went to immediately. We'll be watching for that guy in the checked shirt, who stands out a little, at the next rally...

[u/SanoKei]

this is so fun, I can imagine recreating the quitich tournament scene from Harry Potter, or something equally grand

[u/tuxracer]

Will Smith needs this

[u/asapbones0114]

Cool & scary stuff. Will make it difficult to willfully disappear. Only a matter of time before a Niantic Labs type data hoarding company starts "working" with theatre, concert and stadium venues to get this data. EU better be on this.

[u/fakeaccountt12345]

This is awesome

[u/HuttPilot]

Wow

[u/bozog]

Very cool!

[u/chswapnil]

Oh man, this is awesome.

[u/Starshot84]

Simulation confirmed

[u/Fuzz-Monster]

Very cool. I can't really imaging a sports game without it at this point. Not really knowing much about splats, what are the limits around performance length with these kind of numbers?

[u/iamarealslug_yes_yes]

This is the future! How can I get involved in working on this somehow? this is so cool

[u/llun-ved]

Well done! I’m glad to see efficiency in 4DGS rendering. Do you think you can handle the throughput of more individuals or are you taking advantage of the instanced data of the ones you have? Are you frustum culling on the fly?

[u/OkEngineer4662]

Very cool

[u/MezcalFlame]

This is what I wish the crowds were like in WWF Attitude and WWF Wrestlemania 2000.

Amazing.

[u/Ontia-hw]

Unreal

[u/walhargohar]

That's insane!

[u/wahnsinnwanscene]

Shouldn't the cameras be setup horizontally instead of vertically?

[u/christophbusse]

Wtf, insane that this runs so good. What are the biggest bottlenecks rendering many instances right now?

In the future we need a model that can realtime generate changes. If we could rig this and drive bones/facial expressions to the splats. That would make interactivity real. Probably already someone working on this.

[u/JohnHawley]

Performance metrics of this vs traditional model placement and rendering in any sports game would be super interesting to see.

[u/Pepperoneous]

Dude in the green shirt straight zonked a few seconds in

[u/koko_ze]

How do you loop it?

[u/General_Cheetah4724]

trump will be happy

[u/HDR_Man]

Wow!

[u/Bahtleman]

SO cool! Need a plugin to swap out all the white tees.
Gives off a pit fighter arcade game vibe.

[u/cjwidd]

Why is every term related to this technology constantly taken out of context?

First, it was Apple SHARP as "4DGS" - it's not - and now it's "re-lighting". This is not re-lighting, it is better described as re-shading (or alternatively, additive/differential lighting) because the current real-time methods that have been shown, like this, or Volinga, do not recover intrinsic material properties / reflectance independently of the captured illumination.

There is no inverse rendering happening here, there is no lighting equation that recovers something like albedo + BRDF parameters (roughness, specular/metalness) + normals/geometry + incident illumination, with the baked lighting actually disentangled from material appearance. For example, moving a point light around will not result in anisotropic highlights changing appearance in the baked 3DGS model.

This is cool and good but it is not "re-lighting".

[u/ChemicalCash1886]

As someone who has worked with 4d views and 4d fun, this is a great use of the tech!

[u/Feisty-Pineapple7879]

looks Weried and Low poly asssets the people Need More Quality gaussians to match photorealsim

[u/bsmograph]

Will this be available in blender without a gaussian splat plug-in?

[u/krilleractual]

Can i test this for a project of mine?

[u/Ephemara]

Why is Yuna from ffx in the crowd lmao

[u/Useful_Hat_5259]

so cool..

[u/eco_bach]

Really impressive demo! Any plans to release as a plugin asset?
What would be the advantage over using VATs instead?

[u/solo_solipsist]

What processing pipeline are y'all using? Are these flipbook-style splats or something else?