3D Gaussian Splatting for 3D reconstruction: progress and challenges

In the realm of 3D reconstruction, the concept of a radiance field has witnessed a surge in research activity in recent years, marked by the publication of numerous papers since the foundational NeRF paper in 2020. Several methods, including Instant-NGP, MeRF, Mip-NeRF, and Mobile-NeRF, have endeavored to enhance overall quality while also addressing training and rendering speed. However, achieving optimization across all these aspects simultaneously has remained a challenge. 

3D Gaussian Splatting (3DGS) is a game-changer, offering a unique combination of superior quality, real-time interactive rendering, and training speed that rivals the fastest state-of-the-art methods.  

In 3D Gaussian Splatting, you sample the volume with a 3D point cloud. During rendering, each sample is projected onto the 2D viewing plane, effectively deciding its position on the screen.  

When you splat the projected point onto the 2D viewing plane, instead of simply putting a dot there, you “spread” its value using a Gaussian curve. The center of the splat (where the original projected point was) will have the highest value (the peak of the Gaussian), and the values will taper off as you move away from the center, following the Gaussian curve.  

As you project more and more points from the 3D volume onto the 2D viewing plane, some of the Gaussian splats will overlap. When this happens, the splats are combined based on their weights (from the Gaussian function) and transparency. This amalgamation yields a smooth and faithful image.  

The 3D Gaussian Splat reconstruction (or training) process is therefore the process that generates the set of gaussians and their parameters (orientation, size, color, …). The process is iterative. It starts from an initialization point cloud in which each point in the point cloud is the center of a gaussian. During the iterative training step, the position of the gaussians is refined together with the other parameters to minimize a loss function which measures an error between rendered images and the input images. In this process a gaussian may be split into 2 gaussians, thereby adding details to the representation.  

The advantages of 3D Gaussian Splatting are so profound that this technique is, in essence, supplanting all prior NeRF-based reconstruction methods. However, numerous challenges persist and await resolution:  

  1. Precise Camera Poses: During the iterative reconstruction process, the accuracy of camera poses is critical. The loss function assesses the disparity between a rendered image and an input image. Misalignment of the poses between the two images results in incorrect loss calculations, which, in turn, propagate errors to the parameters of the Gaussians. 
  1. Simplified Representation: 3D Gaussian Splatting essentially encodes intricate scene details as geometry. This approach can lead to a substantial number of Gaussians, impacting file size and rendering speed. 
  1. Initialization and Iteration Count: The quality of the output is heavily reliant on the initial parameters and the number of iterations. Improved initialization parameters can lead to more efficient training and higher-quality results. 
  1. Edition and Post-processing: Much like in traditional photography, capturing the image is just the beginning of the process. Additional tasks such as cropping, denoising, recoloring, relighting, and more are required in the 3D context. 
  1. Motion and Dynamic Scenes: Dynamic scenes introduce a host of challenges, including the capture process, especially in monocular scenarios, the precise estimation of camera poses, maintaining temporal consistency in the representation, and managing the substantial data volume, to name a few. 

Stay tuned as progress is made on all these fronts. These challenges are the focal points of ongoing research and innovation, and as they are addressed, we can anticipate even more remarkable advancements in the realm of 3D reconstruction and visualization. 

Leading the Charge: Pioneering Advances in Radiance Fields

Today, I’m thrilled to share with you some truly remarkable advancements in the domain of Radiance Fields and the pivotal role our company is playing in shaping this exciting frontier of technology.

Radiance Fields represent a transformative leap in computer graphics and computer vision. It enables us to create stunningly realistic 3D scenes and objects, replete with intricate details and lifelike lighting. The implications of these advances extend far beyond the realm of technology; they are poised to redefine the way we experience and interact with the digital world.

Here are some of the latest developments we’ve been at the forefront of:

Hyper-Realistic Rendering: Our commitment to realism knows no bounds. We’ve honed our rendering techniques to deliver unparalleled visual fidelity, allowing us to recreate scenes that are almost indistinguishable from reality. This opens up exciting possibilities in fields like e-commerce, gaming, virtual reality, and architectural visualization.

A critical element to achieve this high quality fidelity has been to optimize the camera poses. The accuracy and appropriateness of camera positions significantly impact the fidelity and realism of the rendered 3D scenes. Precise camera poses ensure that captured data align accurately with the radiance field model, leading to a cohesive and realistic representation of the scene’s lighting and geometry. By leveraging advanced computer vision algorithms and machine learning techniques, our R&D team is able to intelligently determine optimal camera poses that minimize discrepancies and maximize coherence between captured imagery and the radiance field. This meticulous optimization ensures that the resulting radiance field faithfully represents the scene’s lighting and geometry, providing a high-fidelity foundation for immersive 3D rendering.

Faster Reconstruction: In our relentless pursuit of innovation, our R&D team has made significant strides in enhancing the reconstruction speed of Radiance Fields technology. Through meticulous research and experimentation, we’ve optimized the algorithms and fine-tuned the computational processes involved in reconstructing intricate 3D scenes.

By leveraging parallel computing and advanced GPU acceleration techniques, we’ve significantly reduced computation time without compromising the visual quality of the generated imagery. These breakthroughs not only enhance the efficiency of Radiance Fields but also pave the way to reach our ultimate goal of real-time or near real-time reconstruction and rendering of complex scenes.

Photo FX mobile application : The on-going deployment of our first iOS beta app to a broader audience represents a pivotal step in democratizing the exploration of radiance fields. This mobile app allows end users to delve into the captivating realm of radiance fields effortlessly, providing a user-friendly interface for an intuitive experience.

This initiative empowers a diverse audience, from enthusiasts to artists and developers, to experiment with radiance fields, fostering creativity and innovation. The app’s accessibility and ease of use open doors for enthusiasts to grasp the potential applications across various industries, including gaming, architecture, and design.

By gathering valuable feedback and insights from this beta phase, we will refine and optimize the app further, ensuring a seamless and enriching experience for a broader user base in the future. The goal is to encourage exploration, inspire new ideas, and catalyze the integration of radiance fields into mainstream technology, revolutionizing how we perceive and interact with virtual environments on mobile devices.

Collaborations and Partnerships: We’ve been actively collaborating with industry leaders, research institutions, and creative minds to push the boundaries of radiance fields even further. These partnerships have enabled us to pool resources, share knowledge, and accelerate the pace of innovation.

Moreover, our team continues to explore novel approaches to further improve reconstruction and rendering speed with no compromise on the visual quality. These efforts highlights our commitment to ensuring that our solutions are not only cutting-edge but also practical and accessible for a wide range of applications.


Embracing the Light : Stepping Out of Stealth Mode and into a Bold Future

Today marks an important moment in our company’s journey. After months of dedicate work, strategic planning, and R&D hard work, we have reached a crucial milestone : it’s time to step out of stealth mode and unveil our innovation to the community.

Throughout the stealth phase, we’ve operated quietly, diligently refining our vision, honing our technology, and assembling a team of exceptional individuals who share our passion for pushing boundaries. This period of introspection and focused effort has allowed us to incubate our ideas, validate (some of!) our assumptions, and develop a solid foundation upon which our future success will be built.

Our decision to emerge from stealth is a testament to our confidence in our product, technology, and the collective talent that propels us forward. We believe the time is right to share our innovation openly, inviting the community to witness the fruits of our labor and the strides we’ve made in our chosen field.

As we transition into this new phase, we’re excited about the opportunities that lie ahead. We will be unveiling our groundbreaking technology, showcasing its potential to disrupt our industry, and engaging with a broader audience that shares our passion for innovation and progress.

Transparency and collaboration will be at the heart of this new chapter. We eagerly look forward to establishing partnerships, welcoming feedback, and forging relationships with stakeholders who believe in our vision and are eager to join us on this exhilarating journey.

Our team has worked tirelessly to ensure that we’re not just stepping out of the shadows, but leaping into a bright and promising future. Together, we are committed to making a lasting impact and leaving an indelible mark on the industry we serve!