Metal Shaders: Vibrance

In this blog post we will cover how to increase vibrance in your images without blowing out the already bright colors that might be present. The code comes from the Metal GPUImage 3 framework.

Saturation vs Vibrance

In a previous blog post, I covered a saturation filter. As a quick recap, the saturation filter takes the luminance value and adds it to the original pixel color. This operation is applied uniformly to every pixel regardless of the saturation already present in the image.

This means that if you already have some saturated colors in your image, they can get blown out with a saturation adjustment:

Blown out colors cause by saturation adjustment.

Vibrance, on the other hand, is a slightly less blunt instrument. The equation takes into consideration how saturated the original value is. The higher the value, the less the equation impacts it.

Shader Code

Here is the Vibrance fragment function (with the VibranceUniform structure omitted):

fragment half4 vibranceFragment(
	SingleInputVertexIO fragmentInput [[stage_in]],
	texture2d inputTexture [[texture(0)]],
	constant VibranceUniform& uniform [[ buffer(1) ]])
{
	constexpr sampler quadSampler;
	half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);
	
	half average = (color.r + color.g + color.b) / 3.0;
	half mx = max(color.r, max(color.g, color.b));
	half amt = (mx - average) * (-uniform.vibrance * 3.0);
	color.rgb = mix(color.rgb, half3(mx), amt);
	
	return half4(color);
}

First, we are finding the average luminance of the current pixel. We are doing this by adding the red, green, and blue values together and dividing the sum by three.

Next, we are determining which color is most prominent in the currently sampled pixel. This is a nested max() function. It checks if blue or green is greater, then it pits that value against red’s value and whichever value is larger emerges as the victor.

Next we need to determine the weight we will apply to our final color mix. This is determined using the following components:

  • The maximum color value
  • The average color value
  • The user input value from the UI determining the degree of vibrance

You need to determine the difference between the brightest color and the average color. The larger this difference, the more of an impact the shader has on that specific pixel.

Like the saturation shader, the vibrance function also uses a mix function mixing in the maximum color value. However, instead of directly pulling in that value from the slider, it’s applying the color difference value calculated above.

Increased vibrance without the blow out.

Conclusions

Color operations on images are interesting in that there are varying degrees of fidgeting you can do with the operations. There are blunt force operations that tend to be good enough for quick and dirty processing, but there are also options available if you want more control over the output of your image.

Metal Shaders: Blending Basics

One classification of shader present in GPUImage 3 is blend operations. Blends differ from the color processing operations we’ve covered so far by using multiple texture inputs. These inputs are combined in a few different ways to generate interesting effects.

In this blog post I will be covered the simple arithmetic blend operations. These operations include:

  • Dissolve
  • Lighten
  • Darken
  • Subtract
  • Multiply
  • Add
  • Divide

The calculations for these modes came from an addendum to Adobe’s PDF specification. If you’re interested in the original PDF specification all 700+ pages are available here.

Dissolve

One of the simplest types of blends is the dissolve blend. The dissolve blend is what you would use in an editing program like Final Cut Pro to transition from one scene to another without a sharp cut. You’re slowly fading the first image out while bringing the second image in.

Here is the shader for the dissolve blend:

typedef struct
{
    float mixturePercent;
} DissolveBlendUniform;

fragment half4 dissolveBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]],
    constant DissolveBlendUniform& uniform [[ buffer(1) ]])
{
    constexpr sampler quadSampler;
    half4 textureColor = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 textureColor2 = inputTexture2.sample(quadSampler,
        fragmentInput.textureCoordinate2);

    return mix(textureColor, textureColor2, half(uniform.mixturePercent));
}

The main thing to notice in this function is that the parameters have changed. Instead of bringing in a single input texture, we’re bringing in two input textures. These are each occupying a slot in our texture buffer.

The currently sampled pixel color in each of the input textures are mixed together based on a mixture percent that is passed in as user input. The mix() function is explained in detail in a previous post if you need a refresher.

Dissolve blend at a 50/50 mix

Lighten

The dissolve blend used a weighted percentage to determine how much of each image is present in the final output. Let’s say you didn’t want to use that determination and would rather simply choose the brightest pixel present between the two images. That is the gist of the Lighten function:

fragment half4 lightenBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 textureColor = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 textureColor2 = inputTexture2.sample(quadSampler,
        fragmentInput.textureCoordinate2);
	
    return max(textureColor, textureColor2);
}

This function is looking at the value of both the first texture and the second texture and returning whichever value is largest. Larger values are brighter and smaller values are dimmer.

Lighten filter. Since the watercolor image is much lighter than the macarons, the macarons essentially don’t show up.

Darken

The opposite of Lighten is clearly Darken. The Darken function looks like this:

fragment half4 darkenBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    return half4(min(overlay.rgb * base.a, base.rgb * overlay.a) + 
			overlay.rgb * (1.0h - base.a) + base.rgb * 
			(1.0h - overlay.a), 1.0h);
}

This function is slightly more complicated than Lighten, but the principle is similar. It compares the base color and the overly color. If the base color is darker, then it choose the base color. If the overlay is darker, then the overlay is chosen.

Conversely, in the darken filter, the watercolor image barely registers as the macarons are so much darker.

Pre-Multiplied Alpha Channel

This Metal function, along with several others in this blog post, have a complex set of arithmetic operations associated with multiplying the alpha channel.

The alpha channel controls the transparency of the image being displayed. GPUImage was originally designed for streaming video processing, which always has an alpha value of 1.0. However, GPUImage is also used for still image processing, which can have an alpha channel value of something below 1.0.

This filter, along with a few others, utilizes pre-multiplied alpha. There is an excellent explanation of this concept here. Essentially, if you have a base image with an alpha of 1.0 and an overlay that has an alpha of 0.3, you don’t want them to be weighted equally.

The Darken filter can be done as follows without the pre-multiplied alpha:

return min(overlay, base);

Since GPUImage is an open source project, many people have contributed changes over the years. One contributor added the pre-multiplied alpha configuration to several shaders, but not all of them. In the future we may transition all filters to either accounting for pre-multiplied alpha or remove it for the sake of simplicity. For now, my goal with the port to Metal is to maintain fidelity between GPUImage 2 and GPUImage 3. I want them to function exactly the same if you are transitioning between the two. After we complete the port we will evaluate how we want to maintain the shaders moving forward.

Subtract

Subtract is the first of four general arithmetic operations we are all familiar with: Add, Subtract, Multiply, and Divide.

You might think we should start with the Add Blend, but it’s actually surprisingly more complex and we will deal with it later. First, here is the subtract blend:

fragment half4 subtractBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 textureColor = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 textureColor2 = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    return half4(textureColor.rgb - textureColor2.rgb, textureColor.a);
}

This is fairly straightforward and does exactly what you would expect it to do. It takes the red, green, and blue values of the first texture and subtracts from them the red, green, and blue values of the second texture.

Subtract blend at 50%.

Multiply

Multiply is the next-simplest arithmetic blend operation:

fragment half4 multiplyBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    return overlay * base + overlay * 
        (1.0h - base.a) + base * 
        (1.0h - overlay.a);
}

This is another blend that was modified to utilize pre-multiplied alpha. We’re multiplying the RGB values of the base and overlay colors. We’re then factoring in the transparency of the colors by subtracting them from one and adding them together. If the alpha of both textures is 1.0, then it has no impact on the final output color.

With Multiply you can see some of the texture from the watercolor on the macarons.

Add

The Add blend is slightly more complex than previous blends. For this one, we separated the different colors into their separate color channels:

fragment half4 addBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    half r;
    if (overlay.r * base.a + base.r * overlay.a >= overlay.a * base.a) {
	r = overlay.a * base.a + overlay.r * 
            (1.0h - base.a) + base.r * (1.0h - overlay.a);
    } else {
	r = overlay.r + base.r;
    }
	
    half g;
    if (overlay.g * base.a + base.g * overlay.a >= overlay.a * base.a) {
	g = overlay.a * base.a + overlay.g * (1.0h - base.a) + 
            base.g * (1.0h - overlay.a);
    } else {
	g = overlay.g + base.g;
    }
	
    half b;
    if (overlay.b * base.a + base.b * overlay.a >= overlay.a * base.a) {
	b = overlay.a * base.a + overlay.b * (1.0h- base.a) + 
            base.b * (1.0h - overlay.a);
    } else {
	b = overlay.b + base.b;
    }
	
    half a = overlay.a + base.a - overlay.a * base.a;
	
    return half4(r, g, b, a);
}

For each color channel, we’re essentially checking to see if the alpha channel is 1.0 or not. If the alpha channel is not 1.0, then the pre-multiplication alpha adjustments are applied. Else, each color channel value is simply added together and returned at the end.

With the Add blend, you can see the outline of the macarons added to the watercolor texture.

Divide

The final blend mode in this blog post is the Divide function.

fragment half4 divideBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    half ra;
    if (overlay.a == 0.0h || 
        ((base.r / overlay.r) > (base.a / overlay.a))) {
	ra = overlay.a * base.a + overlay.r * (1.0h - base.a) + 
             base.r * (1.0h - overlay.a);
    } else {
	ra = (base.r * overlay.a * overlay.a) / overlay.r + 
             overlay.r * (1.0h - base.a) + base.r * 
             (1.0h - overlay.a);
    }

    half ga;
    if (overlay.a == 0.0h || 
        ((base.g / overlay.g) > (base.a / overlay.a))) {
	ga = overlay.a * base.a + overlay.g * (1.0h - base.a) + 
             base.g * (1.0h - overlay.a);
    } else {
	ga = (base.g * overlay.a * overlay.a) / overlay.g + 
             overlay.g * (1.0h - base.a) + base.g * 
             (1.0h - overlay.a);
    }
	
    half ba;
    if (overlay.a == 0.0h || 
        ((base.b / overlay.b) > (base.a / overlay.a))) {
	ba = overlay.a * base.a + overlay.b * (1.0h - base.a) + 
             base.b * (1.0h - overlay.a);
    } else {
	ba = (base.b * overlay.a * overlay.a) / overlay.b + 
             overlay.b * (1.0h - base.a) + base.b * 
             (1.0h - overlay.a);
    }
	
    half a = overlay.a + base.a - overlay.a * base.a;
	
    return half4(ra, ga, ba, a);
}

For the Divide function, you not only have to account for an alpha value of something other than 1.0, but you also have to account for an alpha value of 0.0. Trying to divide a value by zero will cause unexpected behaviors.

The Divide blend is a little more pronounced than the Multiply blend. Blends require some tweaking so far as texture combinations and amount applied. Some combinations work better than others.

As I am writing this post as of August 11, 2018, we believe that there is an issue with the divide blend. The divide blend was contributed in 2012 and has been a part of the repository since the original GPUImage. We have concerns that if/else statements are not actually preventing or guarding against divide by zero errors. I will come back to this post and update it once we have had a chance to look into this issue.

Conclusions

Blends are a pretty easy way to add some impressive visual effects to your video and images. When I was going to school for graphic design I fell in love with blend modes. They added an interesting sophisticated effect to my designs. Having the opportunity to learn how they work and implement them has been very special to me.