## What was done before NeRF?

Deep learning before this was essentially done a lot on 2D data, which is essentially what we call Euclidean data. Some common applications of deep learning on 2D data include classification, regression-based predictions, and segmentation.

Nowadays, however, research in deep learning has extended to 3D data, which is non-Euclidean in nature. The goal in this case would be to create a 3D model from Euclidean data after training for a finite amount of time for a given number of epochs. This brings us to NeRF!

## What is NeRF?

NeRF (Neural Radiance Field) is essentially a neural network that is trained to generate a set of new 3D images or a 3D representation of an object based on a set of mostly limited 2D images. It takes input scene images and interpolates between them to generate a complete scene.

An insight into the math behind it and how it works:

NeRF is a neural network that basically takes 5D coordinates with parameters such as spatial location (x,y,z) and viewing direction (θ, φ) and outputs the opacity and color, which is 4D. This process uses classic volumetric rendering techniques to bring out the densities and colors in an image.

Here I will explain this using code written in PyTorch 3D.

The following steps should be followed to create and train a simple neural radiation field:

• Initialize the implicit renderer
• Define the neural emission field model

Initialize the implicit renderer:

An implicit function φ can be defined as:

This basically maps 3D points to a scalar value.

An implicit surface, S can be defined as:

Thus, an implicit surface, S is the set of all 3D points ‘x’ whose value of f(x)=τ.

So for each point we evaluate f(x) to determine whether it lies on the surface or not. We then use a coloring function c(x) where

Thus,

which upon convergence we get

and here,

to assign colors to individual ray points. The Raymarching step is then performed, where all f(x) and c(x) are collapsed into a single pixel color on the given ray that is emitted by the camera.

Thus, the renderer consists of a Raysampler and a Raymarcher. Raysampler emits rays from the pixels of the image and samples points on them. Some common examples of Raysamplers used in PyTorch 3D include NDCGridRaySampler() and MonteCarloRaySampler(). Raymarcher then takes the densities and colors sampled from each ray and converts each ray to the color and opacity value of the ray’s output pixel. An example of a Raymarcher is EmissionAbsorptionRaymarcher().

Define the neural emission field model:

The Neural Radiance Field module defines a continuous field of colors and opacities over the 3D domain of the scene.