Enchantment on the Image Enhancement
Image enhancements can Fourier Transforms, White Balancing, Histogram Manipulation, and Contrast Stretching. I will discuss what I have learned here in this post.
1.Fourier Transforms
Since images can be transformed into numbers, it can also be thought that these numbers are a superposition of wave patterns. The resulting wave patterns, just like signals, can have frequencies that can be transformed using Fourier Transforms. As a result, Fourier Transforms can help data scientists remove artifacts from images.
Here, we use Marty from Back to the Future. On the left side is the grayscale image and to its right is its corresponding logarithmic Fourier transform. We shall be using the codes below to show how it’s done:
#importing libraries
import numpy as np
import matplotlib.pyplot as plt
from skimage.io import imread, imshow
from skimage.color import rgb2gray
marty = rgb2gray(imread('bttf.png'))
marty_fft = np.fft.fftshift(np.fft.fft2(marty))fig, ax = plt.subplots(1, 3, gridspec_kw=dict(width_ratios=[16,16,1]),figsize=(16,8))
img = ax[0].imshow(marty, cmap='gray')
ax[1].imshow(np.log(abs(marty_fft)), cmap='gray')
fig.colorbar(img, cax=ax[2]);
Upon careful inspection of the logarithmic Fourier transformed image, we can see that there are distinct white lines, where two of them are dominant vertical and horizontal lines at the middle of the Fourier transform. Removing the white lines on the image can be done by replacing the vertical line with a small value and then transforming the new values back into the image domain.
marty_fft2 = marty_fft.copy()
marty_fft2[:1250,marty_fft.shape[1]//2] = 1
marty_fft2[-750:,marty_fft.shape[1]//2] = 1
imshow(np.log(abs(marty_fft2)), cmap='gray');
imshow(abs(np.fft.ifft2(marty_fft2)), cmap='gray');
As you can see, the image had a lot more contrast after the modification, with the items on the dashboard gaining more detail after the processing. This is one of the potential uses of Fourier transforms in image processing.
2.White Balancing
White Balancing, on the other hand, corrects images from the color range, where whites or neutral colored images appear more white, depending on the editing done. There are different approaches to White Balancing:
White Patch Algorithm — The color white is present when RGB channels are at its maximum. We can correct images based on the color white by either normalizing or rescaling based on the maximum value for the corresponding channel.
Let’s try the algorithm out
import numpy as np
import skimage.io as skio
from skimage import img_as_ubyte, img_as_floatmarty = skio.imread('bttf.png')
skio.imshow(marty);
Since white is observed as a maximum value in the RGB channel, let’s try to divide each channel by it’s maximum value to bring about white balance
marty_wp = img_as_ubyte(marty*1.0 / marty.max(axis=(0,1)))
skio.imshow(marty_wp);
Oddly enough, the image looks like the earlier image displayed. Let’s check on the maximum values already contained within each channel
print('Max value for red channel is: '+str(np.max(marty[:,:,0])))
print('Max value for green channel is: '+str(np.max(marty[:,:,1])))
print('Max value for blue channel is: '+str(np.max(marty[:,:,2])))Max value for red channel is: 255
Max value for green channel is: 255
Max value for blue channel is: 251
This means that both red and green channels are already at maximum, while the blue channel is nearly there. This makes a straightforward white balance approach impractical. In order to properly correct the image color, we must first look at the histogram of the pixel values along the color range and take the any given percentile as our maximum value. For this case, we’ll be using the 95th percentile.
for channel, color in enumerate('rgb'):
print(channel)
channel_values = marty[:,:,channel]
plt.step(np.arange(256),
np.bincount(channel_values.flatten(), minlength=256)*1.0/
channel_values.size,
c=color)
plt.xlim(0, 255)
plt.axvline(np.percentile(channel_values, 95), ls='--', c=color)
plt.xlabel('channel value')
plt.ylabel('fraction of pixels');
Using this piece of code lets us determine the exact numerical value of the 99th percentile of each channel
for channel, color in enumerate('RGB'):
print('99th percentile of %s channel:'%channel,
np.percentile(marty[:,:,channel], 95))99th percentile of 0 channel: 147.0
99th percentile of 1 channel: 148.0
99th percentile of 2 channel: 157.0
#channel 0 = Red
#channel 1 = Green
#channel 2 = Blue
Using this information, we then apply the 95th percentile to the previous approach of White-patch algorithm
marty_wp2 = img_as_ubyte((marty*1.0 / np.percentile(marty, 95, axis=(0, 1)))
.clip(0, 1))
skio.imshow(marty_wp2)
Gray-world Algorithm — Gray is essentially a neutral color (a mix of black and white). With this concept, this algorithm also assumes that pixels are gray on average and that users should be able to adjust each color channel such that their mean values are the same.
Translating the concept into code gets you this output
marty_gw = ((marty * (marty.mean() / marty.mean(axis=(0, 1))))
.clip(0, 255).astype(int))
skio.imshow(marty_gw);
Ground-truth algorithm — Based from the observed image, users can set a specific part of an image as the reference when enhancing the entire canvas. With a set reference, users can then either apply the White Patch or the Gray-world Algorithm depending on the set reference that the ground truth is based on.
Let’s try it out with Marty and his companions. First, we set “ground truths”, where we set reference colors for the correction of the entire image to be based on. For this, we’ll be trying out the skin tones of Dr. Brown, Jennifer, and Marty himself. Let’s see the code in action:
fig, ax = plt.subplots(1,1,dpi=100)
gt1 = Rectangle((1700,600),20,20,edgecolor='r',facecolor='none')
gt2 = Rectangle((1000,350),20,20,edgecolor='g',facecolor='none')
gt3 = Rectangle((500,700),20,20,edgecolor='b',facecolor='none')
ax.add_patch(gt1)
ax.add_patch(gt2)
ax.add_patch(gt3)
ax.imshow(marty)
Now let’s take a closer look at how these references look like
fig, ax = plt.subplots(1,3,figsize=(12,10))
ax[0].imshow(marty[int(xy1[0][1]):int(xy1[1][1]),
int(xy1[0][0]):int(xy1[1][0])])
ax[0].set_title('Doc')
ax[1].imshow(marty[int(xy2[0][1]):int(xy2[1][1]),
int(xy2[0][0]):int(xy2[1][0])])
ax[1].set_title('Jen')
ax[2].imshow(marty[int(xy3[0][1]):int(xy3[1][1]),
int(xy3[0][0]):int(xy3[1][0])])
ax[2].set_title('Marty')
fig.tight_layout()
Now that we have the ground truths, we can do at least two approaches on image correction: either to normalize each color channel to the maximum value of each channel of the ground truth, or to make the mean values of each color channel match those of the mean value of the ground truth. We can show that with the following code below:
doc_patch = marty[int(xy1[0][1]):int(xy1[1][1]),
int(xy1[0][0]):int(xy1[1][0])]
jen_patch = marty[int(xy2[0][1]):int(xy2[1][1]),
int(xy2[0][0]):int(xy2[1][0])]
marty_patch = marty[int(xy3[0][1]):int(xy3[1][1]),
int(xy3[0][0]):int(xy3[1][0])]patches = [doc_patch, jen_patch, marty_patch]for patch in patches:
image_max = (marty / patch.max(axis=(0,1))).clip(0,1)
image_mean = ((marty * patch.mean())
/ marty.mean(axis=(0,1))).clip(0,255).astype(int)
fig, ax = plt.subplots(1, 2, dpi=100)
ax[0].imshow(image_max)
ax[0].set_title('Max Method')
ax[0].set_axis_off()
ax[1].imshow(image_mean)
ax[1].set_title('Mean Method')
ax[1].set_axis_off()
fig.tight_layout()
3.Histogram Manipulation
Given that images are superposition of wave patterns, the numerical distribution generated by these images can be further analyzed, which can be seen through a histogram. Manipulating the histogram means that the cumulative distribution function of the original image will be matched to a target cumulative distribution function as well, wherein all the values from the image will be replaced with the value that matches from the target. This will be done across all corresponding intensity values, using interpolation to lookup their respective replacement values.
Let’s start with color channel exploration for now with Marty.
def rgb_splitter(image):
rgb_list = ['Reds','Greens','Blues']
fig, ax = plt.subplots(1, 3, figsize=(17,7), sharey = True)
for i in range(3):
ax[i].imshow(image[:,:,i], cmap = rgb_list[i])
ax[i].set_title(rgb_list[i], fontsize = 22)
ax[i].axis('off')
fig.tight_layout()
rgb_splitter(marty)
Taking a look at each color’s intensity values give us the following visualization
def df_plotter(image):
freq_bins = [cumulative_distribution(image[:,:,i]) for i in
range(3)]
target_bins = np.arange(255)
target_freq = np.linspace(0, 1, len(target_bins))
names = ['Red', 'Green', 'Blue']
line_color = ['red','green','blue']
f_size = 20
fig, ax = plt.subplots(1, 3, figsize=(17,5))
for n, ax in enumerate(ax.flatten()):
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.step(freq_bins[n][1], freq_bins[n][0], c=line_color[n],
label='Actual CDF')
ax.plot(target_bins,
target_freq,
c='gray',
label='Target CDF',
linestyle = '--')
df_plotter(marty)
As expected, there are a lot of pixels already have high intensity values. Correcting this image would mean that we want to make the distribution of the intensity values uniform, generally making the plots above look linear as the broken line shown in the plots.
#linear adjuster function
def rgb_adjuster_lin(image):
target_bins = np.arange(255)
target_freq = np.linspace(0, 1, len(target_bins))
freq_bins = [cumulative_distribution(image[:,:,i]) for i in
range(3)]
names = ['Reds', 'Greens', 'Blues']
line_color = ['red','green','blue']
adjusted_figures = []
f_size = 20
fig, ax = plt.subplots(1,3, figsize=[15,5])
for n, ax in enumerate(ax.flatten()):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[image[:,:,
n]].astype(int))
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.imshow(adjusted_image, cmap = names[n])
adjusted_figures.append([adjusted_image])
fig.tight_layout()
fig, ax = plt.subplots(1,3, figsize=[15,5])
for n, ax in enumerate(ax.flatten()):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[image[:,:,
n]].astype(int))
freq_adj, bins_adj = cumulative_distribution(adjusted_image)
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.step(bins_adj, freq_adj, c=line_color[n], label='Actual CDF')
ax.plot(target_bins,
target_freq,
c='gray',
label='Target CDF',
linestyle = '--')
fig.tight_layout()
return adjusted_figuresrgb_adjuster_lin(marty)
The final adjusted image can then be generated by the following codes
target_bins = np.arange(255)
target_freq = np.linspace(0, 1, len(target_bins))
freq_bins = [cumulative_distribution(marty[:,:,i]) for i in
range(3)]
adjusted_figures = []for n in range(3):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[marty[:,:,
n]].astype(int))
adjusted_figures.append([adjusted_image])
channel_figures = adjusted_figuresimshow(np.dstack((channel_figures[0][0],
channel_figures[1][0],
channel_figures[2][0])));
Here’s the original image for reference
Aside from a linear behavior, we can also try out different probabilistic distributions to apply to our image. For this case, we’ll try out a Gaussian distribution, as well as a Sigmoid one.
Gaussian adjustment:
#Gaussian adjuster
gaussian = norm(128,64)
def rgb_adjuster_gauss(image):
target_bins = np.arange(255)
target_freq = gaussian.cdf(np.arange(255))
freq_bins = [cumulative_distribution(image[:,:,i]) for i in
range(3)]
names = ['Reds', 'Greens', 'Blues']
line_color = ['red','green','blue']
adjusted_figures = []
f_size = 20
fig, ax = plt.subplots(1,3, figsize=[15,5])
for n, ax in enumerate(ax.flatten()):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[image[:,:,
n]].astype(int))
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.imshow(adjusted_image, cmap = names[n])
adjusted_figures.append([adjusted_image])
fig.tight_layout()
fig, ax = plt.subplots(1,3, figsize=[15,5])
for n, ax in enumerate(ax.flatten()):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[image[:,:,
n]].astype(int))
freq_adj, bins_adj = cumulative_distribution(adjusted_image)
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.step(bins_adj, freq_adj, c=line_color[n], label='Actual CDF')
ax.plot(target_bins,
target_freq,
c='gray',
label='Target CDF',
linestyle = '--')
fig.tight_layout()
return adjusted_figuresrgb_adjuster_gauss(marty)
We can change some part the code from the linear adjustment to extract the gaussian adjusted image
target_bins = np.arange(255)
target_freq = gaussian.cdf(np.arange(255))
freq_bins = [cumulative_distribution(marty[:,:,i]) for i in
range(3)]
adjusted_figures = []for n in range(3):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[marty[:,:,
n]].astype(int))
adjusted_figures.append([adjusted_image])
channel_figures = adjusted_figuresimshow(np.dstack((channel_figures[0][0],
channel_figures[1][0],
channel_figures[2][0])));
Sigmoid adjustment:
#Sigmoid adjuster
from scipy.stats import logistic
def rgb_adjuster_sigmoid(image):
target_bins = np.arange(255)
target_freq = logistic.cdf(np.arange(255))
freq_bins = [cumulative_distribution(image[:,:,i]) for i in
range(3)]
names = ['Reds', 'Greens', 'Blues']
line_color = ['red','green','blue']
adjusted_figures = []
f_size = 20
fig, ax = plt.subplots(1,3, figsize=[15,5])
for n, ax in enumerate(ax.flatten()):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[image[:,:,
n]].astype(int))
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.imshow(adjusted_image, cmap = names[n])
adjusted_figures.append([adjusted_image])
fig.tight_layout()
fig, ax = plt.subplots(1,3, figsize=[15,5])
for n, ax in enumerate(ax.flatten()):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[image[:,:,
n]].astype(int))
freq_adj, bins_adj = cumulative_distribution(adjusted_image)
ax.set_title(f'{names[n]}', fontsize = f_size)
ax.step(bins_adj, freq_adj, c=line_color[n], label='Actual CDF')
ax.plot(target_bins,
target_freq,
c='gray',
label='Target CDF',
linestyle = '--')
fig.tight_layout()
return adjusted_figuresrgb_adjuster_sigmoid(marty)
We can’t really see much of the image already when applied with the Sigmoid function. Let’s see how this works out
target_bins = np.arange(255)
target_freq = logistic.cdf(np.arange(255))
freq_bins = [cumulative_distribution(marty[:,:,i]) for i in
range(3)]
adjusted_figures = []for n in range(3):
interpolation = np.interp(freq_bins[n][0], target_freq,
target_bins)
adjusted_image = img_as_ubyte(interpolation[marty[:,:,
n]].astype(int))
adjusted_figures.append([adjusted_image])
channel_figures = adjusted_figuresimshow(np.dstack((channel_figures[0][0],
channel_figures[1][0],
channel_figures[2][0])));
Hmmmm. There seems to be no detail seen here on the Sigmoid one. That could be expected since the intensity values of a lot of the pixels are at max value. This means there is no difference between intensities of pixels to show any detail at all.
4.Contrast Stretching
Unlike Histogram Manipulation, Contrast Stretching rescales the intensities of each color space that are within the target percentile range. The resulting image only has specific elements enhanced, and not the entire canvas. This can be beneficial for those who wish to emphasize only certain parts of an image.
Let’s show the impact of contrast stretching by putting in a high contrast image. To do that, we’ll be converting Marty and his friends into a grayscale image with the following code
from skimage.color import rgb2graydark_marty = rgb2gray(marty)
imshow(dark_marty)
Now we change the contrast. We’ll be using skimage.exposure.rescale_intensity to perform contrast stretching
from skimage.exposure import rescale_intensity
from skimage import img_as_ubyte
dark_marty_intensity = img_as_ubyte(rgb2gray(dark_marty))#stretching the values between the 5th and 95th percentile
dark_marty_contrast = rescale_intensity(dark_marty_intensity,
in_range=tuple(np.percentile(dark_marty_intensity, (5, 95))))
imshow(dark_marty_contrast);
Congratulations! Now you have a better idea of how image enhancements work.
Interested in more of my stories? You can check my profile below
