Robotica
GameDev

RoboEngine DevBlog#10

Seeing Through Things:
Alpha Cutouts, Glass, and Deferred Rendering

Up until now, every single object in RoboEngine’s world was 100% solid. Great for marble vases and animated foxes, not so great if you want leafy trees, chain-link fences, stained glass, or wispy smoke.

After landing our massive Engine/Editor split, I pushed to teach RoboEngine’s materials how to see through things. Now, foliage can have crisp holes (that even show up in cast shadows!), glass panes can catch PBR reflections while revealing what sits behind them, and everything fits cleanly into both our Forward and Deferred renderers.

The trickiest puzzle? Fitting transparency into a Deferred pipeline, which by design can only store one surface per screen pixel.

Three Alpha Modes
(Straight From glTF)

MaterialComponent gained glTF 2.0’s three-tier alpha model: adding an AlphaMode, an opacity multiplier, and an alphaCutoff threshold:

// How a material's alpha (opacity × Base Color map's alpha) is used; matches glTF's alphaMode
enum class AlphaMode
{
    Opaque, // Alpha is ignored completely
    Mask,   // Cut-out: pixels below alphaCutoff are discarded; the rest are solid (leaves, fences)
    Blend,  // See-through: alpha-blended over what's behind, drawn after all opaque meshes (glass, smoke)
};

AlphaMode alphaMode   = AlphaMode::Opaque;
float     opacity     = 1.0f; // Mask / Blend: multiplied with the Base Color map's alpha
float     alphaCutoff = 0.5f; // Mask: pixels with lower alpha get clipped out

A pixel’s final alpha is always opacity × baseColorMap.a. All three fields save to .scene.json files, show up as live controls in the Inspector, and import directly from glTF files (where base_color_factor[3] now maps cleanly to opacity instead of being ignored):

switch (source->alpha_mode)
{
    case cgltf_alpha_mode_mask:  material.alphaMode = AlphaMode::Mask;  break;
    case cgltf_alpha_mode_blend: material.alphaMode = AlphaMode::Blend; break;
    default:                     material.alphaMode = AlphaMode::Opaque; break;
}
material.alphaCutoff = source->alpha_cutoff;

Each mode flows through the frame’s render passes differently:

Alpha ModeOpaque / G-Buffer PassShadow PassPost-Opaque Pass
OpaqueDrawn normallyFast solid depth—
MaskDrawn; low-alpha pixels discardedCutout depth (preserves holes!)—
BlendSkipped entirelySolid depthForward-lit, alpha-blended, back-to-front

Alpha Cutouts (Mask):
Sharp Holes & Leafy Shadows

On Screen: Zero Extra Pipelines

Mask is the easy win: a pixel is either 100% solid or completely gone, meaning it needs no depth sorting and no alpha blending. Both our forward shader (mesh.frag) and G-buffer shader (gbuffer.frag) simply call one helper at the top of main():

// opacity × Base Color map's alpha
float sampleAlpha(vec2 uv)
{
    return meshOpacity() * texture(baseColorMap, uv).a;
}

// Mask materials: drop pixels below the cutoff (cutoff is 0.0 for Opaque, so nothing discards)
void alphaCutout(vec2 uv)
{
    if (sampleAlpha(uv) < meshAlphaCutoff()) { discard; }
}

Because Opaque materials pass a cutoff of 0.0, both modes share the exact same graphics pipeline on screen.

Smuggling Alpha Into Push Constants

Our mesh push constants were already packed tight with a model matrix and a combined normal-plus-material matrix. Rather than expanding the push constant block, we smuggled both the opacity and alphaCutoff into the unused .w slots of the model matrix’s bottom row (which is always (0, 0, 0, 1) for standard affine transforms):

// Alpha packed into the model matrix's bottom row (always 0, 0, 0 for affine transforms)
const bool usesAlpha = material.alphaMode != AlphaMode::Opaque;
push.modelMatrix[0].w = usesAlpha ? glm::clamp(material.opacity, 0.0f, 1.0f) : 1.0f;
push.modelMatrix[1].w = material.alphaMode == AlphaMode::Mask ? glm::clamp(material.alphaCutoff, 0.0f, 1.0f) : 0.0f;

Our GLSL shaders unpack those floats and scrub the matrix back to clean (0, 0, 0, 1) inside meshModelMatrix().

In Shadows: Punching Holes in Light

Our standard shadow pass only reads vertex positions which would turn a leafy branch into a solid rectangular shadow. To fix that, each mesh system gained a dedicated cutout shadow pipeline that reads UVs, binds the base color texture, and runs a tiny shadow_cutout.frag shader:

// shadow_cutout.frag
void main()
{
    if (push.alpha.x * texture(baseColorMap, fragUV).a < push.alpha.y) { discard; }
}

During renderShadowDepth(), RoboEngine blasts through all solid shadow casters first using the ultra-fast position-only pipeline. Only if it spots a Mask material along the way does it bind the cutout pipeline for a second pass, and because skinned meshes got skeletal_mesh_shadow_cutout.vert, an animated character’s tattered cloak or hair casts accurate holes too!

See-Through Surfaces (Blend):
Hybrid Forward Rendering

Why Deferred Can’t Hold Glass

A G-buffer stores a single depth and material per pixel, as the updated ReSceneRenderer header puts it, it “can’t hold glass and what’s behind it.”

The classic solution is a hybrid pass: Blend materials skip the G-buffer completely. Once the opaque scene finishes (whether via Forward or Deferred lighting), a dedicated renderTransparentMeshes() pass draws all translucent surfaces forward-style right on top of the lit image:

// Deferred path
deferredLightingSystem->render(frameInfo, gbuffer->getDescriptorSet());
renderTransparentMeshes(frameInfo, {staticMeshSystem.get(), skeletalMeshSystem.get()}); // Forward-lit over the result
lightBillboardSystem->render(frameInfo);

// Forward path
skeletalMeshSystem->render(frameInfo);
renderTransparentMeshes(frameInfo, {staticMeshSystem.get(), skeletalMeshSystem.get()}); // Back-to-front
lightBillboardSystem->render(frameInfo);

Sorting Across Static and Skinned Meshes

Alpha blending is strictly order-dependent: a translucent window in front of a ghostly skinned character has to draw after the character. To sort across both static and skeletal mesh systems in one unified list, both implement a TransparentMeshDrawer interface:

class TransparentMeshDrawer
{
public:
    virtual void collectTransparent(const FrameInfo &frameInfo, std::vector<TransparentDraw> &out) = 0;
    virtual void beginTransparent(FrameInfo &frameInfo) = 0; // Bind pipeline + frame descriptor sets
    virtual void drawTransparent(FrameInfo &frameInfo, const TransparentDraw &draw) = 0;
};

renderTransparentMeshes() collects every Blend draw across all drawers, sorts them farthest-first using the squared distance from the camera to each mesh’s bounding-box center, and swaps pipelines only when the drawer type changes:

std::stable_sort(draws.begin(), draws.end(),
                 [](const TransparentDraw &a, const TransparentDraw &b) { return a.distanceSq > b.distanceSq; });

TransparentMeshDrawer *bound = nullptr;
for (const TransparentDraw &draw : draws)
{
    if (draw.drawer != bound) { draw.drawer->beginTransparent(frameInfo); bound = draw.drawer; }
    draw.drawer->drawTransparent(frameInfo, draw);
}

Each mesh system compiles a third lit pipeline with alpha blending enabled, depth testing on, and depth writes off (so opaque walls occlude glass, while panes of glass don’t occlude each other). Inside mesh_transparent.frag, it calls the exact same shadeSurface() function as every opaque object—meaning glass still catches PBR specular highlights, HDR environment reflections, and cast shadows!

Traps & Mobs Along the Way

Blend materials still cast solid shadows.

  • Right now, the shadow pass skips Mask materials in its solid loop, but Blend meshes still run through the solid depth pass, meaning a clear glass window casts a shadow as dark as a brick wall. Passable for heavily tinted glass, though true colored/attenuated translucent shadows are a future quest.

Sorting happens per mesh, not per triangle.

  • Sorting by bounding-box center works well for separate panes of glass, but intersecting transparent meshes, or a concave translucent object, can still exhibit order-dependent artifacts from certain camera angles.

Depth writes off means billboards draw on top.

  • Because transparent surfaces don’t write to the depth buffer, anything drawn afterward (like editor light billboards) renders cleanly over them rather than getting clipped.

Always read the model matrix via meshModelMatrix().

  • With opacity and cutoff riding in .w of the first two columns, any shader reading raw push.modelMatrix directly will get garbage transform math.

Descriptor sets shift in skinned cutout shadows.

  • Because skeletal_mesh_shadow_cutout.vert needs set 0 for the material’s baseColorMap texture, the skinning storage buffer slides over to set 1 (SKINNING_SET 1). Any future shader combining skinning and material textures has to follow the same convention.

Zero overhead when unused.

  • The solid shadow pass tracks an anyCutout flag; if a scene has zero Mask materials, the second cutout shadow loop and pipeline switch are skipped completely.

Next Up on the Quest

With transparency unlocked, RoboEngine can now render just about anything a glTF 2.0 file throws at it! Next, we’re shifting gears from graphics programming to repository architecture: in my next DevBlog, I’ll tackle extracting the game content out into its own standalone repository and turning RoboEngine into a reusable Git submodule ready to power real game projects!