Detailed Description of CorbeauSplat: An All-in-One Gaussian Splatting Automation Tool for macOS Silicon

Introduction

CorbeauSplat is a revolutionary all-in-one automation tool designed specifically for macOS users equipped with Apple Silicon processors. Developed as an amalgamation of existing open-source projects, this application streamlines the complex workflow of converting raw video or image data into high-fidelity 3D scenes using Gaussian Splatting—a cutting-edge technique in computer vision and graphics rendering. Originally conceived to facilitate the technical demands of a documentary film titled Le Corbeau, CorbeauSplat was created by an individual who sought to automate repetitive tasks involving structure-from-motion (SfM), photogrammetry, and machine learning-based 3D reconstruction.

Unlike traditional workflows that require manual intervention across multiple applications—such as COLMAP for sparse reconstruction, Brush for Gaussian Splatting training, and SuperSplat for visualization—CorbeauSplat consolidates these processes into a single intuitive graphical user interface (GUI). This integration not only saves time but also reduces the risk of errors associated with managing dependencies between disparate tools. The tool is particularly well-suited for professionals in filmmaking, virtual production, game development, and augmented reality who require efficient 3D asset generation from real-world data.

---

Core Features and Functionality

1. Project Management

One of the primary strengths of CorbeauSplat lies in its ability to automate project organization. When users upload raw video or image sequences, the application automatically structures them into a hierarchical folder system that separates inputs (images, videos) from intermediate outputs (sparse reconstructions, checkpoints) and final results (trained Gaussian Splat models). This modular approach ensures that users can easily locate and revisit different stages of their workflow without manual file management.

Key Features:

• Automatic Folder Creation: Projects are saved in a standardized format within an output directory, making it simple to track progress across multiple sessions.
• Dependency Tracking: The tool checks for missing files or corrupted inputs during processing, providing clear feedback if any inconsistencies arise.
• Session Persistence: Intermediate results (e.g., sparse reconstructions from COLMAP) are saved as checkpoints, allowing users to resume training from the last saved state without redoing prior steps.

*Figure 1: The main interface of *CorbeauSplat, showcasing its modular tabs for project management and processing.

---

2. Sparse Reconstruction with COLMAP or Glomap

Gaussian Splatting relies on high-quality sparse reconstructions to generate detailed 3D scenes. CorbeauSplat integrates two popular tools for this purpose: COLMAP (a widely used Structure-from-Motion framework) and Glomap (a modern alternative with improved performance).

COLMAP Integration

• Feature Extraction: The tool automatically runs COLMAP’s feature detection and matching algorithms to identify keypoints and relationships between images.
• Mapping: It generates a sparse reconstruction, producing camera poses and point clouds that serve as the foundation for densification.
• Undistortion: To ensure optimal training quality, CorbeauSplat includes an undistortion step, correcting lens distortions in input images before further processing.

Glomap Integration

For users seeking faster or more efficient mapping, CorbeauSplat supports Glomap, a Rust-based alternative to COLMAP that offers better performance on Apple Silicon. The tool allows users to toggle between the two backends via an optional parameters tab, enabling them to choose based on their specific needs (e.g., speed vs. accuracy).

---

3. Image Upscaling and Preprocessing

Before training Gaussian Splats, high-resolution images are often required for better detail capture. CorbeauSplat includes a dedicated upscale module that leverages Real-ESRGAN—a state-of-the-art AI-based image restoration framework—to enhance resolution.

Key Features:

• Model Selection: Users can choose from pre-trained models like RealESRGAN_x4plus, which is designed to upscale images by factors of 2x or 4x.
• Automatic Dependency Installation: If the required dependencies (e.g., PyTorch, OpenCV) are missing, CorbeauSplat attempts to install them automatically via Homebrew.

---

4. Gaussian Splatting Training with Brush

The heart of CorbeauSplat is its integration with Brush, a dedicated Gaussian Splatting training application for macOS. Brush allows users to train high-quality 3D scenes directly on their Apple Silicon devices, leveraging the GPU acceleration available in modern Macs.

Training Workflow

1. Auto-Refinement: Users can resume training from the latest checkpoint, ensuring continuity across sessions.
2. Presets for Densification: Brush offers predefined densification strategies (e.g., "Aggressive Densification") to optimize the reconstruction process based on input data complexity.
3. Real-Time Feedback: The application provides visual feedback during training, allowing users to monitor progress and adjust parameters if necessary.

---

5. Visualization with SuperSplat

Once a Gaussian Splat model is trained, CorbeauSplat integrates SuperSplat, a web-based editor by PlayCanvas, for interactive visualization and editing of the resulting PLY files (a common format for 3D mesh data).

Key Features:

• Local Viewer: Users can load their trained .ply file directly into SuperSplat’s browser-based interface, enabling real-time adjustments to lighting, camera angles, and material properties.
• Editing Capabilities: The tool supports basic editing functions such as selecting, rotating, and scaling splats, making it easier to refine the final output before exporting.

---

Experimental Features

6. Single Image to 3D Conversion (Apple ML Sharp)

For users with limited data, CorbeauSplat includes a bonus feature that leverages Apple’s Machine Learning Sharp framework—a Swift-based tool for generating 3D models from single images.

Process:

1. Users select an input image.
2. The application applies Apple ML Sharp to estimate camera pose and reconstruct a basic mesh.
3. While this method is less detailed than Gaussian Splatting, it serves as a quick alternative for initial exploration or when working with constrained datasets.

---

7. 4D Gaussian Splatting (Experimental)

The term "4DGS" refers to the extension of Gaussian Splatting to handle multi-camera video sequences over time, enabling dynamic scenes that capture motion and depth variations across frames. CorbeauSplat includes a module for preparing such datasets in Nerfstudio’s format.

Key Features:

• Video-to-Nerf Conversion: Users upload synchronized camera videos, and the tool processes them into a structured dataset compatible with 4DGS training pipelines.
• Dependency Installation: The application checks for and installs required dependencies (e.g., Python packages, Rust tools) before proceeding.

---

8. 360° Video Extraction

For virtual reality (VR) or immersive media applications, CorbeauSplat supports the extraction of equirectangular 360° videos into optimized planar layouts such as Cube Maps or Rings. This is particularly useful for photogrammetry and VR content creation.

Key Features:

• AI Operator Masking: The tool automatically detects and masks out non-relevant elements (e.g., operators, reflections) to improve reconstruction quality.
• Layout Options: Users can choose between different layouts (Cube Map, Ring, Fibonacci) based on their specific use case.
• Dependency Installation: Required libraries like PySide6 (for GUI integration) and YOLOv8 (for object detection) are installed automatically if missing.

---

Technical Requirements and Installation

System Prerequisites

CorbeauSplat is designed exclusively for macOS Silicon, ensuring compatibility with Apple’s latest hardware. The following requirements must be met:

• macOS Version: Any recent version of macOS (preferably macOS Ventura or later).
• Python: Version 3.13+ (for optimal performance) or 3.11 (supported). Python is required for running COLMAP, Brush, and other dependencies.
• Xcode Command Line Tools: Essential for compiling custom engines like Glomap or Brush.
• Homebrew: A package manager for installing system dependencies such as COLMAP, FFmpeg, and Rust toolchains.
• Git: For cloning the repository.

Installation Steps

1. Clone the Repository: Users simply run a single command to download the entire project:

   git clone https://github.com/freddewitt/CorbeauSplat.git
   cd CorbeauSplat

2. Run the Launcher Script: The provided run.command script automatically detects missing dependencies and installs them, ensuring a seamless setup process.

---

User Interface Overview

The CorbeauSplat interface is organized into multiple tabs, each dedicated to a specific workflow stage:

1. Configuration Tab:

• Users select their input (video or image folder).
• Define a project name for organization.
• Initiate the COLMAP dataset creation process.

1. Params Tab:

• Allows users to tweak advanced COLMAP settings or switch between COLMAP and Glomap backends.

1. Upscale Tab:

• Enables image upscaling using Real-ESRGAN with customizable model selection and scale factors.

1. Brush Tab:

• Provides options for auto-refinement, densification presets, and starting Brush training.

1. SuperSplat Tab:

• Loads trained .ply files and launches the SuperSplat viewer for visualization and editing.

1. 4DGS Tab (Experimental):

• Prepares multi-camera video datasets for 4D Gaussian Splatting training.

1. 360° Extractor Tab (Experimental):

• Converts equirectangular videos into optimized layouts with AI masking.

1. Apple ML Sharp Tab:

• Offers a quick single-image-to-3D conversion using Apple’s machine learning tools.

---

Command Line Interface (CLI)

In addition to its GUI, CorbeauSplat exposes all features via a command-line interface (CLI), making it accessible for users who prefer scripting or automation. Users can refer to the CLI.md documentation for detailed instructions on running workflows programmatically.

---

Acknowledgments and Credits

The development of CorbeauSplat is deeply rooted in existing open-source projects, each contributing critical components to its functionality:

• COLMAP: The foundation for sparse reconstruction, providing structure-from-motion capabilities.
• Brush: A specialized Gaussian Splatting trainer optimized for macOS Silicon.
• SuperSplat: A web-based editor for visualizing and editing 3D splat scenes.
• 360Extractor: Specialized tools for processing equirectangular videos.
• Apple ML Sharp: Machine learning frameworks for Swift-based 3D reconstruction.
• Nerfstudio: The modular framework behind 4D Gaussian Splatting data preparation.

By leveraging these tools, CorbeauSplat consolidates their functionalities into a cohesive workflow, reducing the complexity of manual processing and enabling faster iteration in 3D content creation.

---

Conclusion

CorbeauSplat represents a significant advancement in automating the complex pipeline of converting real-world data into high-quality 3D scenes using Gaussian Splatting. By combining the strengths of COLMAP, Brush, SuperSplat, and other open-source tools, it provides a user-friendly yet powerful solution for professionals working on filmmaking, virtual production, or game development.

While CorbeauSplat is not without its limitations—particularly in experimental features like 4DGS and 360° extraction—its core functionality remains robust and efficient. The tool’s ability to handle dependencies automatically, support multiple workflows, and integrate with Apple Silicon hardware makes it an invaluable asset for anyone working in the intersection of computer vision, graphics, and multimedia production.

For those interested in exploring further or contributing to its development, the project is licensed under the MIT License, ensuring open access and collaboration. Whether used as a standalone application or integrated into larger pipelines, CorbeauSplat offers a compelling solution for streamlining 3D asset generation on macOS.