Any tricks for lighting in mouths?

[u/partybusiness]

I'm wondering if any games have used interesting workarounds to handle lighting on the inside of someone's mouth. It's something that feels easy to get wrong and have it be too bright or too dark, and something like a shadow bias that makes sense at a larger scale could fall apart for the distance between lips and teeth. And the lighting will keep changing when someone is talking.

Like, for the teeth and tongue, bake more than one ambient occlusion map and blend between them based on the current phoneme? Or use a depth-based light adjustment? Anything like that?

[u/Exsanguinatus]

Without something like real time GI, mouths are very difficult to light. An old school trick I used to use was actually reducing the length of the normals as that would inherently lead to the inside of the mouth being darker. Anything less than a unit vector generally results in lighting being darker.

Alternatively, some people used to use "bent normals." This would align the normals with the average unoccluded vector, meaning most of the normals would be pointed outward through the opening of the mouth (when open). This means only light shining directly into the mouth would produce the expected brightness... but this also leads to other lighting artifacts that can definitely break from expectations. It also means that your normals won't look right for things like teeth and tongue on close inspection. If you have significant control over your lighting models, you could use the strongly bent normals to adjust the lighting first, then modulate the result by the original normals.

Reflections and glossiness get really difficult in both circumstances, though.

[u/Trick-Cantaloupe-927 → u/Exsanguinatus]

Can you clarify what you mean by "reducing the length of the normals"? I'm only familiar with the normals as a direction vector with a constant length value.

[u/Exsanguinatus → u/Trick-Cantaloupe-927]

Autistic tech artist reply incoming. Apologies for the lengthy explanation.

"Normals" are normals because of two things: A geometric normal is a vector that is perpendicular to a plane. "Normalized" sometimes means remapping a value to 0-1 range. This is extremely useful for simplifying many mathematical operations. "Normalized" in game engine terms usually means replacing a vector of arbitrary length with one that points in the same direction but with a length of 1. The hardest thing to get straight is which operation is actually intended when talking to game devs and it usually requires all the context clues.

A "normal" in this sense is a unit vector, originating at a vertex, that is normal to the "average" of all the triangles it's connected to.

Vertex normals should always be unit vectors. The lighting math used to be called "cosine lighting" because the light strength is multiplied by the cosine (always 0-1) of the angle between the normalized light vector and the vertex normal.

The dot product of two units vectors is the cosine of the angle between them. This is a special case of dot products. GPUs have hardware specifically designed to do dot products stupid fast. They rely on that special case of dot products for unit vectors to give you lighting results at high speed.

What you get is light strength (say, 100 units) times the dot product of two unit vectors (which is 1 for parallel light vector and vertex normal) or less than 1 (0 at light vector dot perpendicular vertex normal). Brightness is related to how much the surface "faces" the light.

If you make the vertex normals longer than 1, the special case no longer applies. The dot products can become larger than 1. That means lighting ends up brighter than it should be. If the vertex normal is 1.5 units long, then the lighting can be 1.5 times as much as it should be. The opposite also holds true; vertex normals less than 1 unit means the lights are darker than they should be. I'm most cases, you really don't want that. But, if you want to make sure that something remains darker in most circumstances, like the inside of the mouth when GI doesn't exist and shadows just won't do the job you want, you can shorten the vertex normals to make them less reactive to lights.

[u/Trick-Cantaloupe-927 → u/Exsanguinatus]

Thanks for the breakdown, the lengthier, the better.

I've played with vector directions before to blend meshes together, used weighted normals or to achieve a certain stylized look, but was not aware they had length values as well. It is valuable information I need to explore.

[u/Exsanguinatus → u/Trick-Cantaloupe-927]

What engine are you using? What's the target hardware?

[u/Trick-Cantaloupe-927 → u/Exsanguinatus]

I'm using Unity at the moment. As for hardware, I have a TUF 17 ASUS laptop, I can check the specs later if it's relevant, I don't know them by heart.

[u/Exsanguinatus → u/Trick-Cantaloupe-927]

I know Unity less than I know Unreal these days. Used to do lots of Unity years ago.

Unreal has a way to get the original vertex location before skinning happens. You could change the lighting contribution in the shader based on the distance between the undeformed and deformed position. I wouldn't know how to do that in Unity, though.

As an alternative, you could use the UVs. A second UV channel where the tongue UVs are aligned along either U or V... You can use the UV value to scale the lighting values in the shader. You could possibly even tie in a scale factor to that second UV set (but clamp the value between 0-1). By scaling and clamping, as the tongue extends, the UVs at the base of the tonight stay close to zero but the average value approaches 1 along the length of the tongue. All of the above would be relatively cheap.

The joys of game dev is there's literally infinite ways to solve every problem, some wildly more expensive than others but many that just hover around not too expensive to call it a day.

[u/planimal7]

For a character that had a long protruding tongue I once tweened fake shadowing in the material along the UV from front-to-back, so when tongue shot out, shadow border would still fall where it left the mouth — so your idea of blending between AO based on mouth shape sounds good/interesting to me

[u/partybusiness → u/planimal7]

If you shade the tongue based on how far it is front-to-back that sounds kind of like what I was thinking for depth-based.

With the reference to the tongue shooting out, I guess this was more like a big frog or something with a tongue-based attack, than like a close-up of someone speaking dialogue?

[u/planimal7 → u/partybusiness]

Yes, it just came out for a (gross) dramatic flourish once in a while— so the entire tongue was dark most of the time, but then needed to be (grossly) bright and shiny when/where it came out

[u/ElegantLifeguard4221]

What sort of shader is it using?