In this blog post we will cover how to increase vibrance in your images without blowing out the already bright colors that might be present. The code comes from the Metal GPUImage 3 framework.
Saturation vs Vibrance
In a previous blog post, I covered a saturation filter. As a quick recap, the saturation filter takes the luminance value and adds it to the original pixel color. This operation is applied uniformly to every pixel regardless of the saturation already present in the image.
This means that if you already have some saturated colors in your image, they can get blown out with a saturation adjustment:
Vibrance, on the other hand, is a slightly less blunt instrument. The equation takes into consideration how saturated the original value is. The higher the value, the less the equation impacts it.
Here is the Vibrance fragment function (with the VibranceUniform structure omitted):
First, we are finding the average luminance of the current pixel. We are doing this by adding the red, green, and blue values together and dividing the sum by three.
Next, we are determining which color is most prominent in the currently sampled pixel. This is a nested max() function. It checks if blue or green is greater, then it pits that value against red’s value and whichever value is larger emerges as the victor.
Next we need to determine the weight we will apply to our final color mix. This is determined using the following components:
The maximum color value
The average color value
The user input value from the UI determining the degree of vibrance
You need to determine the difference between the brightest color and the average color. The larger this difference, the more of an impact the shader has on that specific pixel.
Like the saturation shader, the vibrance function also uses a mix function mixing in the maximum color value. However, instead of directly pulling in that value from the slider, it’s applying the color difference value calculated above.
Color operations on images are interesting in that there are varying degrees of fidgeting you can do with the operations. There are blunt force operations that tend to be good enough for quick and dirty processing, but there are also options available if you want more control over the output of your image.
One classification of shader present in GPUImage 3 is blend operations. Blends differ from the color processing operations we’ve covered so far by using multiple texture inputs. These inputs are combined in a few different ways to generate interesting effects.
In this blog post I will be covered the simple arithmetic blend operations. These operations include:
One of the simplest types of blends is the dissolve blend. The dissolve blend is what you would use in an editing program like Final Cut Pro to transition from one scene to another without a sharp cut. You’re slowly fading the first image out while bringing the second image in.
The main thing to notice in this function is that the parameters have changed. Instead of bringing in a single input texture, we’re bringing in two input textures. These are each occupying a slot in our texture buffer.
The currently sampled pixel color in each of the input textures are mixed together based on a mixture percent that is passed in as user input. The mix() function is explained in detail in a previous post if you need a refresher.
The dissolve blend used a weighted percentage to determine how much of each image is present in the final output. Let’s say you didn’t want to use that determination and would rather simply choose the brightest pixel present between the two images. That is the gist of the Lighten function:
This function is slightly more complicated than Lighten, but the principle is similar. It compares the base color and the overly color. If the base color is darker, then it choose the base color. If the overlay is darker, then the overlay is chosen.
Pre-Multiplied Alpha Channel
This Metal function, along with several others in this blog post, have a complex set of arithmetic operations associated with multiplying the alpha channel.
The alpha channel controls the transparency of the image being displayed. GPUImage was originally designed for streaming video processing, which always has an alpha value of 1.0. However, GPUImage is also used for still image processing, which can have an alpha channel value of something below 1.0.
This filter, along with a few others, utilizes pre-multiplied alpha. There is an excellent explanation of this concept here. Essentially, if you have a base image with an alpha of 1.0 and an overlay that has an alpha of 0.3, you don’t want them to be weighted equally.
The Darken filter can be done as follows without the pre-multiplied alpha:
return min(overlay, base);
Since GPUImage is an open source project, many people have contributed changes over the years. One contributor added the pre-multiplied alpha configuration to several shaders, but not all of them. In the future we may transition all filters to either accounting for pre-multiplied alpha or remove it for the sake of simplicity. For now, my goal with the port to Metal is to maintain fidelity between GPUImage 2 and GPUImage 3. I want them to function exactly the same if you are transitioning between the two. After we complete the port we will evaluate how we want to maintain the shaders moving forward.
Subtract is the first of four general arithmetic operations we are all familiar with: Add, Subtract, Multiply, and Divide.
You might think we should start with the Add Blend, but it’s actually surprisingly more complex and we will deal with it later. First, here is the subtract blend:
This is fairly straightforward and does exactly what you would expect it to do. It takes the red, green, and blue values of the first texture and subtracts from them the red, green, and blue values of the second texture.
Multiply is the next-simplest arithmetic blend operation:
This is another blend that was modified to utilize pre-multiplied alpha. We’re multiplying the RGB values of the base and overlay colors. We’re then factoring in the transparency of the colors by subtracting them from one and adding them together. If the alpha of both textures is 1.0, then it has no impact on the final output color.
The Add blend is slightly more complex than previous blends. For this one, we separated the different colors into their separate color channels:
For each color channel, we’re essentially checking to see if the alpha channel is 1.0 or not. If the alpha channel is not 1.0, then the pre-multiplication alpha adjustments are applied. Else, each color channel value is simply added together and returned at the end.
The final blend mode in this blog post is the Divide function.
For the Divide function, you not only have to account for an alpha value of something other than 1.0, but you also have to account for an alpha value of 0.0. Trying to divide a value by zero will cause unexpected behaviors.
As I am writing this post as of August 11, 2018, we believe that there is an issue with the divide blend. The divide blend was contributed in 2012 and has been a part of the repository since the original GPUImage. We have concerns that if/else statements are not actually preventing or guarding against divide by zero errors. I will come back to this post and update it once we have had a chance to look into this issue.
Blends are a pretty easy way to add some impressive visual effects to your video and images. When I was going to school for graphic design I fell in love with blend modes. They added an interesting sophisticated effect to my designs. Having the opportunity to learn how they work and implement them has been very special to me.