Research training · 2025 · NJU-3DV (now PIL)

This was an introductory research project under Prof. Yao Yao. Built on the lab’s Pointrix / msplat framework, it explored pretrained geometric priors, depth and cross-view optical-flow supervision, and their effects on rendering.

Project code (my changes) Change history

Framework and attribution

Pointrix provides the training framework, msplat CUDA rendering backend, depth rendering, and a normal-supervision example. My work extended and adapted these existing components through integration, debugging, and experiments. Normal supervision was adapted from the framework’s example.

My changes are on my-change-branch; the default main branch retains the original framework. Optical-flow and MoGe visualization utilities also retain their respective upstream implementations.

My contributions

  • Prior integration and alignment: ran RAFT and monocular geometry models, integrated depth, normal, and flow priors, and handled file formats, normal-coordinate conventions, validity masks, and flow scaling during image resizing.
  • Rendering and supervision extensions: integrated geometric features into the existing Python rendering pipeline. Cross-view flow uses the difference between projected Gaussian centers in adjacent cameras, followed by the existing alpha blending. Added depth and flow supervision with depth scale-and-shift alignment, robust residuals, gradient constraints, and edge weighting.
  • Camera-index fixes in validation: separated dataset-local indices from original frame IDs, constructed a global camera table, and reindexed cameras after sorting to fix adjacent-frame pairing across nonconsecutive validation samples.
  • Experiments and documentation: added geometric visualizations and logging, and compared priors, supervision weights, and training-view counts.

Experiments and observations

Scene Comparisons Recorded evidence
LLFF fern 6 / 12 / 20 training views; normal weights 0 / 0.1 / 0.3 Training curves and qualitative novel-view comparisons
LLFF flower Separate normal, depth, and flow supervision Geometry renders, RGB L1 curves, and comparisons before and after camera-index fixes

Some geometry renders moved closer to the pretrained priors. Depth supervision produced smoother surfaces, and normal supervision improved parts of the normal maps. RGB reconstruction gains were inconsistent, and the flow prior’s ability to preserve detail remained an open question.

Depth-supervision comparison and RGB L1 curves on flower, with a pretrained depth prior in the right column
flower depth example: no supervision, depth weight 0.1, and the predicted prior, with RGB L1 curves below. The original “gt” label refers to a pretrained model prediction, not measured depth ground truth.

These were exploratory training experiments. The records do not provide reliable PSNR / SSIM / LPIPS comparisons, and per-view training exposure was not matched across view counts. They therefore do not establish a general improvement in novel-view synthesis. The public branch is an implementation snapshot; the reports provide context for the experimental settings and process.

Resources

← Back to projects