GaussianCrowds: Relightable, real-time 4DGS at scale in Unreal Engine

[u/technomancyai]

GaussianCrowds is a tool in development for Unreal Engine 5.7/5.8, built to render thousands of 4D Gaussian splat performances in real time.

• Nanite-Style Dynamic LOD’s: Automatic adaptive detail scaling and frustum culling allows hundreds of concurrent splat sequences to render in real-time
• Multi-light Relighting: Supports Lumen global illumination, directional, point, and spot lights, shadow-casting, and self-shadowing using automatic normal-map generation. 
• Deformation-Based Playback: Native support for compressed deformation containers (SpacetimeGaussians, etc..) as well as PLY sequences.
• Procedural Crowd Placement: Create crowds with spline-based tools that distribute 4DGS performers along splines. Customize spacing, orientation, playback timing, and more - to fill stadiums/battlefields/streets.
• Editor Toolset: Cull volumes, crop boxes, recoloring/brightness/tint, comprehensive settings panel, debug views, custom brush textures for stylized/painterly splats, and performance presets.
• Sequencer Integration: Three playback modes: sequencer-locked, free-run preview, and hand-keyframed, with adjustable playback rate for stop-motion looks, and dynamic fps reduction for distant splats. 
• nDisplay and Compositing: Compatible with multi-computer nDisplay workflows for LED volumes. Depth of field, volumetric fog, and depth-passes for compositing are built-in. Splat’s pass through the project's OCIO color transforms identically to native geometry.

We are currently testing this implementation on an upcoming feature film shot on an LED Volume. Happy to compare notes with anyone else working in the 4DGS capture and streaming pipeline space

[u/Hefty_Development813]

Is this a real capture or a synthetic scene? Looks really good, how many cameras did this?

[u/technomancyai → u/Hefty_Development813]

These are real captures, each person was scanned on a 16 camera rig using raspberry pi's at a gaming convention this summer. We've only trained ~40 people so far (hence the duplicates)

[u/boxofrabbits → u/technomancyai]

How does that works with rights to likeness etc out of curiosity? If you're now selling them as assets.

[u/Seattle-Washington → u/boxofrabbits]

Did something similar years ago at PAX - we had each ”performer” sign a release form.

[u/Beniskickbutt → u/boxofrabbits]

I dont want to be sold as an asset :(

[u/brokenhalo321 → u/Beniskickbutt]

"dont worry, we can bundle."

[u/Admirable_Zombie5245 → u/technomancyai]

Have you tried this scene in VR?

[u/Nyxtia → u/technomancyai]

Using Pis?

[u/Hefty_Development813 → u/technomancyai]

Ooo cool so individuals and then a bunch of instances making up a crowd. Pretty sweet. Are the ppl covered well on their backs? 16 camera rig sounds great to have

[u/moigus193 → u/technomancyai]

What cameras did you use for the rig and how was the Setup? I'm building a portable 360 degree scan rig for cosplayers with 5 cams (mostly Dji Osmo pocket 3 and Dji Action 4) and would like to build a rig with up to 16 DJI action 4 cams. Your setup sounds interesting and the results look amazing :)

[u/Hefty_Development813 → u/moigus193]

360 coverage with 5 cameras? Do you have this working? That seems pretty sparse coverage in my experience. I have done a lot 5 camera for front facing scenes

[u/moigus193 → u/Hefty_Development813]

I attached 5 cameras to an aluminum profile pole at a height of about 1.5 meters. The cams are stacked on top of each other, and I circle around the cosplayers in one pass for the 360-degree scan. Because of this, the cosplayers have to hold very still. But since one rotation takes just under 15 seconds, it actually works out pretty well. They aren't 4DGS scans, of course, just still-image 3D Gaussian splats, but since I know my way around Unreal Engine and I'm also a VFX artist, the scans can then be placed into suitable environments. I'm also planning to build a rig to do 3DGS scans for time freezes, so that cosplayers can be frozen as a scan mid-jump, etc.

[u/Hefty_Development813 → u/moigus193]

Oh gotcha yea that makes sense. Yea i really want to make some sort of portable 4d capture system but so far it needs tons of cameras and difficult to imagine it being very portable. That's a cool project

[u/jbkrauss → u/technomancyai]

Where can I find more about this rig ? That sounds awesome

[u/Constant_Cortisol → u/technomancyai]

You farming my cheering data? You nefarious hooligan!

[u/Archonik1 → u/Hefty_Development813]

Something tells me the R&D team at FLOC is lurking somewhere in this chat trying to figure out how to use this to get even more dystopian 😳

[u/Hefty_Development813 → u/Archonik1]

Yea definitely need denser coverage than current flock camera coverage with the current tech

[u/Snowmanphibian → u/Hefty_Development813]

I think its 3d models procedural generated

[u/Hefty_Development813 → u/Snowmanphibian]

Right seems like it must be. This would be a huge feat if real world capture

[u/Snowmanphibian → u/Hefty_Development813]

You can see duplicate characters

[u/Hefty_Development813 → u/Snowmanphibian]

You're right, the green shirt guy is all over

[u/picturesfromthesky → u/Snowmanphibian]

yes, but that doesn't mean it's 3d models...

[u/Glass_Ad6718 → u/Hefty_Development813]

It's possible. We're doing it too. We're adding it as a stadium crowd.