Metal Shaders: Blending Basics

One classification of shader present in GPUImage 3 is blend operations. Blends differ from the color processing operations we’ve covered so far by using multiple texture inputs. These inputs are combined in a few different ways to generate interesting effects.

In this blog post I will be covered the simple arithmetic blend operations. These operations include:

  • Dissolve
  • Lighten
  • Darken
  • Subtract
  • Multiply
  • Add
  • Divide

The calculations for these modes came from an addendum to Adobe’s PDF specification. If you’re interested in the original PDF specification all 700+ pages are available here.

Dissolve

One of the simplest types of blends is the dissolve blend. The dissolve blend is what you would use in an editing program like Final Cut Pro to transition from one scene to another without a sharp cut. You’re slowly fading the first image out while bringing the second image in.

Here is the shader for the dissolve blend:

typedef struct
{
    float mixturePercent;
} DissolveBlendUniform;

fragment half4 dissolveBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]],
    constant DissolveBlendUniform& uniform [[ buffer(1) ]])
{
    constexpr sampler quadSampler;
    half4 textureColor = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 textureColor2 = inputTexture2.sample(quadSampler,
        fragmentInput.textureCoordinate2);

    return mix(textureColor, textureColor2, half(uniform.mixturePercent));
}

The main thing to notice in this function is that the parameters have changed. Instead of bringing in a single input texture, we’re bringing in two input textures. These are each occupying a slot in our texture buffer.

The currently sampled pixel color in each of the input textures are mixed together based on a mixture percent that is passed in as user input. The mix() function is explained in detail in a previous post if you need a refresher.

Dissolve blend at a 50/50 mix

Lighten

The dissolve blend used a weighted percentage to determine how much of each image is present in the final output. Let’s say you didn’t want to use that determination and would rather simply choose the brightest pixel present between the two images. That is the gist of the Lighten function:

fragment half4 lightenBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 textureColor = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 textureColor2 = inputTexture2.sample(quadSampler,
        fragmentInput.textureCoordinate2);
	
    return max(textureColor, textureColor2);
}

This function is looking at the value of both the first texture and the second texture and returning whichever value is largest. Larger values are brighter and smaller values are dimmer.

Lighten filter. Since the watercolor image is much lighter than the macarons, the macarons essentially don’t show up.

Darken

The opposite of Lighten is clearly Darken. The Darken function looks like this:

fragment half4 darkenBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    return half4(min(overlay.rgb * base.a, base.rgb * overlay.a) + 
			overlay.rgb * (1.0h - base.a) + base.rgb * 
			(1.0h - overlay.a), 1.0h);
}

This function is slightly more complicated than Lighten, but the principle is similar. It compares the base color and the overly color. If the base color is darker, then it choose the base color. If the overlay is darker, then the overlay is chosen.

Conversely, in the darken filter, the watercolor image barely registers as the macarons are so much darker.

Pre-Multiplied Alpha Channel

This Metal function, along with several others in this blog post, have a complex set of arithmetic operations associated with multiplying the alpha channel.

The alpha channel controls the transparency of the image being displayed. GPUImage was originally designed for streaming video processing, which always has an alpha value of 1.0. However, GPUImage is also used for still image processing, which can have an alpha channel value of something below 1.0.

This filter, along with a few others, utilizes pre-multiplied alpha. There is an excellent explanation of this concept here. Essentially, if you have a base image with an alpha of 1.0 and an overlay that has an alpha of 0.3, you don’t want them to be weighted equally.

The Darken filter can be done as follows without the pre-multiplied alpha:

return min(overlay, base);

Since GPUImage is an open source project, many people have contributed changes over the years. One contributor added the pre-multiplied alpha configuration to several shaders, but not all of them. In the future we may transition all filters to either accounting for pre-multiplied alpha or remove it for the sake of simplicity. For now, my goal with the port to Metal is to maintain fidelity between GPUImage 2 and GPUImage 3. I want them to function exactly the same if you are transitioning between the two. After we complete the port we will evaluate how we want to maintain the shaders moving forward.

Subtract

Subtract is the first of four general arithmetic operations we are all familiar with: Add, Subtract, Multiply, and Divide.

You might think we should start with the Add Blend, but it’s actually surprisingly more complex and we will deal with it later. First, here is the subtract blend:

fragment half4 subtractBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 textureColor = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 textureColor2 = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    return half4(textureColor.rgb - textureColor2.rgb, textureColor.a);
}

This is fairly straightforward and does exactly what you would expect it to do. It takes the red, green, and blue values of the first texture and subtracts from them the red, green, and blue values of the second texture.

Subtract blend at 50%.

Multiply

Multiply is the next-simplest arithmetic blend operation:

fragment half4 multiplyBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    return overlay * base + overlay * 
        (1.0h - base.a) + base * 
        (1.0h - overlay.a);
}

This is another blend that was modified to utilize pre-multiplied alpha. We’re multiplying the RGB values of the base and overlay colors. We’re then factoring in the transparency of the colors by subtracting them from one and adding them together. If the alpha of both textures is 1.0, then it has no impact on the final output color.

With Multiply you can see some of the texture from the watercolor on the macarons.

Add

The Add blend is slightly more complex than previous blends. For this one, we separated the different colors into their separate color channels:

fragment half4 addBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    half r;
    if (overlay.r * base.a + base.r * overlay.a >= overlay.a * base.a) {
	r = overlay.a * base.a + overlay.r * 
            (1.0h - base.a) + base.r * (1.0h - overlay.a);
    } else {
	r = overlay.r + base.r;
    }
	
    half g;
    if (overlay.g * base.a + base.g * overlay.a >= overlay.a * base.a) {
	g = overlay.a * base.a + overlay.g * (1.0h - base.a) + 
            base.g * (1.0h - overlay.a);
    } else {
	g = overlay.g + base.g;
    }
	
    half b;
    if (overlay.b * base.a + base.b * overlay.a >= overlay.a * base.a) {
	b = overlay.a * base.a + overlay.b * (1.0h- base.a) + 
            base.b * (1.0h - overlay.a);
    } else {
	b = overlay.b + base.b;
    }
	
    half a = overlay.a + base.a - overlay.a * base.a;
	
    return half4(r, g, b, a);
}

For each color channel, we’re essentially checking to see if the alpha channel is 1.0 or not. If the alpha channel is not 1.0, then the pre-multiplication alpha adjustments are applied. Else, each color channel value is simply added together and returned at the end.

With the Add blend, you can see the outline of the macarons added to the watercolor texture.

Divide

The final blend mode in this blog post is the Divide function.

fragment half4 divideBlendFragment(
    TwoInputVertexIO fragmentInput [[stage_in]],
    texture2d inputTexture [[texture(0)]],
    texture2d inputTexture2 [[texture(1)]])
{
    constexpr sampler quadSampler;
    half4 base = inputTexture.sample(quadSampler, 
        fragmentInput.textureCoordinate);
    constexpr sampler quadSampler2;
    half4 overlay = inputTexture2.sample(quadSampler, 
        fragmentInput.textureCoordinate2);
	
    half ra;
    if (overlay.a == 0.0h || 
        ((base.r / overlay.r) > (base.a / overlay.a))) {
	ra = overlay.a * base.a + overlay.r * (1.0h - base.a) + 
             base.r * (1.0h - overlay.a);
    } else {
	ra = (base.r * overlay.a * overlay.a) / overlay.r + 
             overlay.r * (1.0h - base.a) + base.r * 
             (1.0h - overlay.a);
    }

    half ga;
    if (overlay.a == 0.0h || 
        ((base.g / overlay.g) > (base.a / overlay.a))) {
	ga = overlay.a * base.a + overlay.g * (1.0h - base.a) + 
             base.g * (1.0h - overlay.a);
    } else {
	ga = (base.g * overlay.a * overlay.a) / overlay.g + 
             overlay.g * (1.0h - base.a) + base.g * 
             (1.0h - overlay.a);
    }
	
    half ba;
    if (overlay.a == 0.0h || 
        ((base.b / overlay.b) > (base.a / overlay.a))) {
	ba = overlay.a * base.a + overlay.b * (1.0h - base.a) + 
             base.b * (1.0h - overlay.a);
    } else {
	ba = (base.b * overlay.a * overlay.a) / overlay.b + 
             overlay.b * (1.0h - base.a) + base.b * 
             (1.0h - overlay.a);
    }
	
    half a = overlay.a + base.a - overlay.a * base.a;
	
    return half4(ra, ga, ba, a);
}

For the Divide function, you not only have to account for an alpha value of something other than 1.0, but you also have to account for an alpha value of 0.0. Trying to divide a value by zero will cause unexpected behaviors.

The Divide blend is a little more pronounced than the Multiply blend. Blends require some tweaking so far as texture combinations and amount applied. Some combinations work better than others.

As I am writing this post as of August 11, 2018, we believe that there is an issue with the divide blend. The divide blend was contributed in 2012 and has been a part of the repository since the original GPUImage. We have concerns that if/else statements are not actually preventing or guarding against divide by zero errors. I will come back to this post and update it once we have had a chance to look into this issue.

Conclusions

Blends are a pretty easy way to add some impressive visual effects to your video and images. When I was going to school for graphic design I fell in love with blend modes. They added an interesting sophisticated effect to my designs. Having the opportunity to learn how they work and implement them has been very special to me.