[Tutorial] From Bilateral Filtering to HDR Applications

We will introduce brief history and applications of Bilateral operation. At the end, we introduce the paper “Deep Bilateral Learning for Real-Time Image Enhancement”

Introduction of Bilateral Filtering: Bilateral filtering for gray and color images

• Background

When we apply Gaussian Filter for Denoising purpose, there is a well-known problem that it smoothen the edge information.

This problem is due to simple Gaussian distance between neighbors in euclidean space,

GB(I)_p = SUM_q(G(||p-q||)), where p and q are physical locations of pixels

From the problem above, we intuitively realized that we need an extra term to consider ‘edge’ information.

• Intensity term

To this end, we consider Intensity term, I(p), and I(q), along with the physical location term p and q.

Therefore, we now apply Gaussian to the Intensity term and secure Edge-preserving effect.

So the result looks like (c), preserving edges.

In practice, we can see visual difference between Bilateral Filter and Gaussian LPF like below

As you can see in the formula, there are two parameters, deltas, for Bilateral Filter. By adjusting them, you can adjust strength.

• Impact on research society

This paper shed light on future research and application. For next few years, there came out several papers suggesting various applications using Bilateral Filter operation, even HDR.

In summary, Bilateral Filter is a 3D dimensional operation that consider not only x, y position but also intensity I(x), I(y).

• Advance one more step

The benefits of this is edge-aware operation.

Now someone wondered “Can we perform each different operation on the regions separated by edges using the characteristics of ‘edge-awared operation’?”

However, the problem is that bilateral filtering operation is so slow that it cannot be used practically to apply several operations by regions.

Therefore, the following research suggested ‘Grid data structure’ to effectively store and apply regional operations for bilateral filtering.

Real-time Edge-Aware Image Processing with the Bilateral Grid

Recalling the bilateral filtering operation:

So, now lets make it faster.

Firstly, we start from generating ‘GRID’ matrix which we can use as an edge-aware computation.

For example, imagine there are brushes that can retouch the region inside each black circle. Then if we group contiguous circles and represent as a grids with represent values, it would be like right side graph.

Now we generalize this process using Gaussian.

There is a picture from the paper that can show this process well.

Grid creation: we can regard it as a resolution quantization.

Process : This is a process where we apply 3D operation to the grid. In other words, we apply bilateral filtering to 3D grid.

Slice : Grouping and averaging procedure.

Now let’s see how we can apply this technic practically.

This is the history of Edge-Awared Bilateral Grid Filtering.

Now let’s see the modern technic using Deep Learning.

Paper Review : Deep Bilateral Learning for Real-Time Image Enhancement

The paper begins with the questions below:

(1). Can we train the operation for splatting and bilateral grid using Deep Learning technic?

(2). Can we maintain full resolution with the operation above?

(3). Can we design operation to consider Global/Local semantic?

The author suggested new design to solve all the questions above

Now let’s closely take a look at.

In this procedure,

(1). 256x256 fixed size input for computational efficiency. This is possible because we are getting small resolution Bilateral grid of coefficients.

(2). Deep feature paths are later divided to Global feature path and Local feature path, then merged together.

(3) After all, we now get extra value to the x,y location value, x, y, and I(x,y), which we can interpret as a splatting.

This figure from the paper shows that learning the splatting increases expressive power.

So, now we ask; is global path helpful?

==> Yes.

Then we ask; How do we generate Bilateral Grid by doing feature fusion of global/local paths?

Consequently, grid cell is [16, 16, 8]-dimension and each cell has 12 numbers.

The 12 numbers can be interpreted as affine transformation coefficients

Next, there are few more technics to apply Grid operation.

Firstly, we need Guidance map where we apply grid operation. Then we perform operation on a full-resolution space.

We can use this novel structure for various tasks, as shown above.

Below are the sample images of the tasks above.

Author: Jonghwa Yim (임종화)