🔬 Geometry-Supervised 3D Gaussian Splatting Rendering

💡 Code Implementation & Architecture Evolution

This project is deeply customized based on the Pointrix framework. The core objective is to introduce Geometry Supervision and Temporal Consistency (Optical Flow) constraints to improve reconstruction quality under sparse views.

1. Renderer Modifications

Extended the traditional Splatting rasterization pipeline in MsplatNormalRender. In addition to RGB color, real-time rendering capability for Optical Flow has been added:

• Dual-Projection Mechanism: When rendering the current frame t, 3D Gaussians are simultaneously projected to the camera view of the next frame t+1.
• Flow Calculation: The instantaneous motion field of each Gaussian sphere is obtained by calculating the difference between the two projection coordinates Δuv = uv_t+1 - uv_t.
• Rasterization: This motion field is Alpha Blended as an attribute to obtain a differentiable rendered optical flow map for subsequent supervision.

2. Loss Function Design

To leverage geometric priors, multiple new loss functions were constructed in NormalModel:

• Depth Loss: Introduced a Scale-Shift Invariant alignment strategy. Since monocular depth estimation (e.g., MoGe) has scale uncertainty, the code aligns rendered depth with prior depth via least squares and introduces Edge-aware Weighting to reduce the impact of alignment errors at object boundaries.
• Flow Loss: Constrains rendered optical flow against GT optical flow using Charbonnier Loss. Robustness handling is specifically added to filter out minute motion noise, and weights are dynamically adjusted based on RGB image gradients to preserve texture edges.

3. Pipeline & Validation

Resolved the temporal discontinuity issue caused by training/validation split. Refactored camera management logic in examples/supervise/model.py, introducing all_camera_model to maintain global Original Frame IDs. This ensures that even in a randomly sampled validation set, the physical "next frame" can still be correctly indexed for optical flow validation.