[Audit][High] Mat4.lookAt crashes with degenerate vectors when camera looks straight up/down

[MichaelFisher1997]

🔍 Module Scanned

libs/zig-math/ (engine-math module, re-exported via engine-math)

📝 Summary

The Mat4.lookAt function does not handle degenerate cases where the forward direction is parallel or anti-parallel to the world up vector. When a camera looks straight up (+Y) or down (-Y), the computed right vector becomes zero, leading to an invalid view matrix with zero-length axes. This causes incorrect vertex transformation and could crash GPU operations or produce invisible geometry.

📍 Location

• File: libs/zig-math/mat4.zig:79-103
• Function/Scope: Mat4.lookAt function

🔴 Severity: High

• Critical: Crashes, data corruption, security vulnerabilities, GPU device loss
• High: Memory leaks, race conditions, incorrect rendering, broken features
• Medium: Performance degradation, missing error handling, suboptimal patterns
• Low: Code style, dead code, minor improvements

💥 Impact

When the camera direction is aligned with the world up vector (e.g., player looking straight up or down), lookAt produces an invalid view matrix. The right vector r = f.cross(world_up) becomes zero when f is parallel to world_up. When normalize() is called on a zero vector in libs/zig-math/vec3.zig:65-68, it returns Vec3.zero, which is not a valid direction. This causes:

• Incorrect vertex transformation in shaders
• Potential invisible geometry (if all directions collapse)
• Could cause undefined behavior in frustum culling calculations that rely on the matrix

User-visible symptoms: Camera jumps or rendering glitches when looking straight up or down.

🔎 Evidence

In libs/zig-math/mat4.zig:79-103:

pub fn lookAt(eye: Vec3, target: Vec3, world_up: Vec3) Mat4 {
    const f = target.sub(eye).normalize();
    const r = f.cross(world_up).normalize();  // <-- r becomes zero when f || world_up
    const u = r.cross(f);
    // ... fill matrix rows with r, u, -f vectors
}

The normalize() function in libs/zig-math/vec3.zig:65-68:

pub fn normalize(self: Vec3) Vec3 {
    const len = self.length();
    if (len == 0) return Vec3.zero;  // <-- Returns zero vector on degenerate input
    return self.scale(1.0 / len);
}

When f is parallel to world_up, f.cross(world_up) produces zero vector. Normalizing zero returns zero, so r = Vec3.zero. Then u = r.cross(f) also produces zero. The resulting matrix has degenerate rows, producing incorrect transformations.

🛠️ Proposed Fix

Add validation and fallback behavior in Mat4.lookAt:

1. Detect near-parallel case by checking if f.cross(world_up) magnitude is small
2. Use an alternative up vector when this occurs (e.g., use X-axis as fallback when looking along Y)
3. Or use the existing lookAt in libs/zig-math/math.zig which may already handle this edge case

pub fn lookAt(eye: Vec3, target: Vec3, world_up: Vec3) Mat4 {
    var f = target.sub(eye).normalize();
    var r = f.cross(world_up).normalize();
    
    // Handle degenerate case: f is parallel to world_up
    if (r.lengthSquared() < 0.0001) {
        // Use a different reference vector to construct orthonormal basis
        const alternate_up = if (@abs(world_up.y) > 0.9) Vec3.right else Vec3.up;
        r = f.cross(alternate_up).normalize();
    }
    
    const u = r.cross(f);
    // ... rest of implementation
}

Also consider adding a compile-time assertion or debug-only check that validates the output matrix has orthonormal rows (or close to it).

✅ Acceptance Criteria

• All existing unit tests pass after the fix
• Camera looking straight up (+Y) produces a valid view matrix
• Camera looking straight down (-Y) produces a valid view matrix
• New test added: lookAt with degenerate input vectors produces identity-like fallback matrix
• Visual verification: camera can smoothly orbit to point straight up/down without glitches

📚 References

• Standard lookAt implementation pattern: Gram-Schmidt orthonormalization handles degenerate cases
• OpenGL gluLookAt and DirectX XMMatrixLookAtLH both handle this edge case explicitly
• Related: Vec3.normalize returns zero vector for zero input (line 65-68 in vec3.zig)

[github-actions]

The audit accurately identifies the bug. The lookAt function in libs/zig-math/mat4.zig:79-103 has the degenerate case issue:

const f = target.sub(eye).normalize();
const r = f.cross(world_up).normalize();  // Zero when f ∥ world_up
const u = r.cross(f);

When the camera looks straight up/down and f is parallel to world_up, the cross product produces zero, which normalize() converts to Vec3.zero (line 67 in vec3.zig). The resulting matrix has degenerate axes.

Suggested fix location: libs/zig-math/mat4.zig:79 - add a degeneracy check before the cross products:

pub fn lookAt(eye: Vec3, target: Vec3, world_up: Vec3) Mat4 {
    const f = target.sub(eye).normalize();
    
    var up = world_up;
    if (@abs(f.dot(world_up)) > 0.999) {
        up = Vec3.init(1, 0, 0);  // Use X as fallback when looking along Y
    }
    const r = f.cross(up).normalize();
    const u = r.cross(f);
    ...

File locations:

• Implementation: libs/zig-math/mat4.zig:79
• Vec3.normalize (returns zero for zero input): libs/zig-math/vec3.zig:65-68
• Tests: modules/engine-math/src/mat4_tests.zig:24

opencode session  |  github run