# Metal Shaders: Vibrance

In this blog post we will cover how to increase vibrance in your images without blowing out the already bright colors that might be present. The code comes from the Metal GPUImage 3 framework.

## Saturation vs Vibrance

In a previous blog post, I covered a saturation filter. As a quick recap, the saturation filter takes the luminance value and adds it to the original pixel color. This operation is applied uniformly to every pixel regardless of the saturation already present in the image.

This means that if you already have some saturated colors in your image, they can get blown out with a saturation adjustment:

Vibrance, on the other hand, is a slightly less blunt instrument. The equation takes into consideration how saturated the original value is. The higher the value, the less the equation impacts it.

Here is the Vibrance fragment function (with the VibranceUniform structure omitted):

```fragment half4 vibranceFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant VibranceUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

half average = (color.r + color.g + color.b) / 3.0;
half mx = max(color.r, max(color.g, color.b));
half amt = (mx - average) * (-uniform.vibrance * 3.0);
color.rgb = mix(color.rgb, half3(mx), amt);

return half4(color);
}
```

First, we are finding the average luminance of the current pixel. We are doing this by adding the red, green, and blue values together and dividing the sum by three.

Next, we are determining which color is most prominent in the currently sampled pixel. This is a nested `max()` function. It checks if blue or green is greater, then it pits that value against red’s value and whichever value is larger emerges as the victor.

Next we need to determine the weight we will apply to our final color mix. This is determined using the following components:

• The maximum color value
• The average color value
• The user input value from the UI determining the degree of vibrance

You need to determine the difference between the brightest color and the average color. The larger this difference, the more of an impact the shader has on that specific pixel.

Like the saturation shader, the vibrance function also uses a mix function mixing in the maximum color value. However, instead of directly pulling in that value from the slider, it’s applying the color difference value calculated above.

## Conclusions

Color operations on images are interesting in that there are varying degrees of fidgeting you can do with the operations. There are blunt force operations that tend to be good enough for quick and dirty processing, but there are also options available if you want more control over the output of your image.

# Metal Shaders: Blending Basics

One classification of shader present in GPUImage 3 is blend operations. Blends differ from the color processing operations we’ve covered so far by using multiple texture inputs. These inputs are combined in a few different ways to generate interesting effects.

In this blog post I will be covered the simple arithmetic blend operations. These operations include:

• Dissolve
• Lighten
• Darken
• Subtract
• Multiply
• Divide

The calculations for these modes came from an addendum to Adobe’s PDF specification. If you’re interested in the original PDF specification all 700+ pages are available here.

## Dissolve

One of the simplest types of blends is the dissolve blend. The dissolve blend is what you would use in an editing program like Final Cut Pro to transition from one scene to another without a sharp cut. You’re slowly fading the first image out while bringing the second image in.

Here is the shader for the dissolve blend:

```typedef struct
{
float mixturePercent;
} DissolveBlendUniform;

fragment half4 dissolveBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]],
constant DissolveBlendUniform& uniform [[ buffer(1) ]])
{
half4 textureColor = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 textureColor2 = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

return mix(textureColor, textureColor2, half(uniform.mixturePercent));
}
```

The main thing to notice in this function is that the parameters have changed. Instead of bringing in a single input texture, we’re bringing in two input textures. These are each occupying a slot in our texture buffer.

The currently sampled pixel color in each of the input textures are mixed together based on a mixture percent that is passed in as user input. The `mix()` function is explained in detail in a previous post if you need a refresher.

## Lighten

The dissolve blend used a weighted percentage to determine how much of each image is present in the final output. Let’s say you didn’t want to use that determination and would rather simply choose the brightest pixel present between the two images. That is the gist of the Lighten function:

```fragment half4 lightenBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]])
{
half4 textureColor = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 textureColor2 = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

return max(textureColor, textureColor2);
}
```

This function is looking at the value of both the first texture and the second texture and returning whichever value is largest. Larger values are brighter and smaller values are dimmer.

## Darken

The opposite of Lighten is clearly Darken. The Darken function looks like this:

```fragment half4 darkenBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]])
{
half4 base = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 overlay = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

return half4(min(overlay.rgb * base.a, base.rgb * overlay.a) +
overlay.rgb * (1.0h - base.a) + base.rgb *
(1.0h - overlay.a), 1.0h);
}
```

This function is slightly more complicated than Lighten, but the principle is similar. It compares the base color and the overly color. If the base color is darker, then it choose the base color. If the overlay is darker, then the overlay is chosen.

Pre-Multiplied Alpha Channel

This Metal function, along with several others in this blog post, have a complex set of arithmetic operations associated with multiplying the alpha channel.

The alpha channel controls the transparency of the image being displayed. GPUImage was originally designed for streaming video processing, which always has an alpha value of 1.0. However, GPUImage is also used for still image processing, which can have an alpha channel value of something below 1.0.

This filter, along with a few others, utilizes pre-multiplied alpha. There is an excellent explanation of this concept here. Essentially, if you have a base image with an alpha of 1.0 and an overlay that has an alpha of 0.3, you don’t want them to be weighted equally.

The Darken filter can be done as follows without the pre-multiplied alpha:

`return min(overlay, base);`

Since GPUImage is an open source project, many people have contributed changes over the years. One contributor added the pre-multiplied alpha configuration to several shaders, but not all of them. In the future we may transition all filters to either accounting for pre-multiplied alpha or remove it for the sake of simplicity. For now, my goal with the port to Metal is to maintain fidelity between GPUImage 2 and GPUImage 3. I want them to function exactly the same if you are transitioning between the two. After we complete the port we will evaluate how we want to maintain the shaders moving forward.

## Subtract

Subtract is the first of four general arithmetic operations we are all familiar with: Add, Subtract, Multiply, and Divide.

You might think we should start with the Add Blend, but it’s actually surprisingly more complex and we will deal with it later. First, here is the subtract blend:

```fragment half4 subtractBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]])
{
half4 textureColor = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 textureColor2 = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

return half4(textureColor.rgb - textureColor2.rgb, textureColor.a);
}
```

This is fairly straightforward and does exactly what you would expect it to do. It takes the red, green, and blue values of the first texture and subtracts from them the red, green, and blue values of the second texture.

## Multiply

Multiply is the next-simplest arithmetic blend operation:

```fragment half4 multiplyBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]])
{
half4 base = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 overlay = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

return overlay * base + overlay *
(1.0h - base.a) + base *
(1.0h - overlay.a);
}
```

This is another blend that was modified to utilize pre-multiplied alpha. We’re multiplying the RGB values of the base and overlay colors. We’re then factoring in the transparency of the colors by subtracting them from one and adding them together. If the alpha of both textures is 1.0, then it has no impact on the final output color.

The Add blend is slightly more complex than previous blends. For this one, we separated the different colors into their separate color channels:

```fragment half4 addBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]])
{
half4 base = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 overlay = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

half r;
if (overlay.r * base.a + base.r * overlay.a >= overlay.a * base.a) {
r = overlay.a * base.a + overlay.r *
(1.0h - base.a) + base.r * (1.0h - overlay.a);
} else {
r = overlay.r + base.r;
}

half g;
if (overlay.g * base.a + base.g * overlay.a >= overlay.a * base.a) {
g = overlay.a * base.a + overlay.g * (1.0h - base.a) +
base.g * (1.0h - overlay.a);
} else {
g = overlay.g + base.g;
}

half b;
if (overlay.b * base.a + base.b * overlay.a >= overlay.a * base.a) {
b = overlay.a * base.a + overlay.b * (1.0h- base.a) +
base.b * (1.0h - overlay.a);
} else {
b = overlay.b + base.b;
}

half a = overlay.a + base.a - overlay.a * base.a;

return half4(r, g, b, a);
}
```

For each color channel, we’re essentially checking to see if the alpha channel is 1.0 or not. If the alpha channel is not 1.0, then the pre-multiplication alpha adjustments are applied. Else, each color channel value is simply added together and returned at the end.

## Divide

The final blend mode in this blog post is the Divide function.

```fragment half4 divideBlendFragment(
TwoInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
texture2d inputTexture2 [[texture(1)]])
{
half4 base = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half4 overlay = inputTexture2.sample(quadSampler,
fragmentInput.textureCoordinate2);

half ra;
if (overlay.a == 0.0h ||
((base.r / overlay.r) > (base.a / overlay.a))) {
ra = overlay.a * base.a + overlay.r * (1.0h - base.a) +
base.r * (1.0h - overlay.a);
} else {
ra = (base.r * overlay.a * overlay.a) / overlay.r +
overlay.r * (1.0h - base.a) + base.r *
(1.0h - overlay.a);
}

half ga;
if (overlay.a == 0.0h ||
((base.g / overlay.g) > (base.a / overlay.a))) {
ga = overlay.a * base.a + overlay.g * (1.0h - base.a) +
base.g * (1.0h - overlay.a);
} else {
ga = (base.g * overlay.a * overlay.a) / overlay.g +
overlay.g * (1.0h - base.a) + base.g *
(1.0h - overlay.a);
}

half ba;
if (overlay.a == 0.0h ||
((base.b / overlay.b) > (base.a / overlay.a))) {
ba = overlay.a * base.a + overlay.b * (1.0h - base.a) +
base.b * (1.0h - overlay.a);
} else {
ba = (base.b * overlay.a * overlay.a) / overlay.b +
overlay.b * (1.0h - base.a) + base.b *
(1.0h - overlay.a);
}

half a = overlay.a + base.a - overlay.a * base.a;

return half4(ra, ga, ba, a);
}
```

For the Divide function, you not only have to account for an alpha value of something other than 1.0, but you also have to account for an alpha value of 0.0. Trying to divide a value by zero will cause unexpected behaviors.

As I am writing this post as of August 11, 2018, we believe that there is an issue with the divide blend. The divide blend was contributed in 2012 and has been a part of the repository since the original GPUImage. We have concerns that if/else statements are not actually preventing or guarding against divide by zero errors. I will come back to this post and update it once we have had a chance to look into this issue.

## Conclusions

Blends are a pretty easy way to add some impressive visual effects to your video and images. When I was going to school for graphic design I fell in love with blend modes. They added an interesting sophisticated effect to my designs. Having the opportunity to learn how they work and implement them has been very special to me.

The first few shaders I’ve covered in this series utilized a constant that was applied uniformly to the entire image. However, many algorithms we use in image processing, we would like to control the degree of effect we want to apply to an image. We don’t always want all or nothing. The purpose of this post will to cover several adjustment filters present in GPUImage 3:

• Brightness
• Contrast
• Exposure
• Gamma
• Saturation
• Red-Green-Blue Channel Adjustment

Before we jump into these shaders, I would like to briefly cover how you can receive user input to determine the percentage of effect you would like to apply to the image.

## Encoding Parameters in Metal

There are two different ways to get parameters to the GPU:

• Hard code them to constant memory space
• Encode them to buffers to be accessed by the GPU

In the previous post about luminance we utilized the first method. The luminance algorithm doesn’t change and it is being used by multiple shaders in the GPUImage library.

For our adjustments in this section, we have a slider value to pass from the UI to the GPU. This value is encoded into a buffer that can be accessed by the GPU. We are already encoding the image we are processing as a texture. You can read more about encoding here.

Here is an example of one of our Swift classes defining a new shader operation with a single parameter to encode:

```public class BrightnessAdjustment: BasicOperation {
public var brightness:Float = 0.0 { didSet { uniformSettings[0] = brightness } }

public init() {
super.init(fragmentFunctionName:"brightnessFragment", numberOfInputs:1)

uniformSettings.appendUniform(0.0)
}
}
```

Pay close attention to this line:

```uniformSettings.appendUniform(0.0)
```

This is where we are encoding a value into our uniform buffer. We want to start out with the value set to 0.0. We want the uniform to update and respond to user input, so we take advantage of Swift’s `didSet` functionality:

```public var brightness:Float = 0.0 { didSet { uniformSettings[0] = brightness } }
```

Any time the `brightness` variable changes, the value of the uniform setting gets updated to the current value. Since we only have one value, these are set to the `[0]` index of the buffer.

So we have a buffer with a value of 0.0 that can be accessed by the shader. But the shader doesn’t know what this value correlates to. In order to do that, we need to set up a custom data structure:

```typedef struct
{
float brightness;
} BrightnessUniform;
```

We use this in the function signature to help the shader “decode” what this value is used for in our fragment equations:

```fragment half4 brightnessFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant BrightnessUniform& uniform [[ buffer(1) ]])
```

All of the shaders I detail in this blog post follow this pattern. The only real change between them is the name of the fragment function, the uniform structure, and the constant passed into the fragment function. Let’s look at the math that goes into these effects next.

## Brightness

Brightness is the intensity of color within an image. Here is the brightness filter’s code:

```fragment half4 brightnessFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant BrightnessUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

return half4(color.rgb + uniform.brightness, color.a);
}
```

You need to augment the intensity of each color by increasing the amount. The brightness shader is very similar to the saturation filter, except the adjustment colors are not weighted. Each of the red, green, and blue values are being augmented by the same amount. This isn’t a super refined color adjustment algorithm but it gets the job done. Changing the brightness does not fundamentally change the dynamic range of the image.

## Contrast

While brightness represents the overall intensity of an image, contrast represents the difference between the lightest parts of an image and the darkest parts. The larger the difference between these values, the more contrast you have in an image. A black and white graphic novel page has incredible contrast because each point on the page is either all or nothing.

Here is our contrast filter:

```fragment half4 contrastFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant ContrastUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

return half4(((color.rgb - half3(0.5)) * uniform.contrast + half3(0.5)), color.a);
}
```

While brightness was an additive operation, contrast is a multiplicative operation. Smaller, darker values are impacted less from these operations than larger, brighter values. This means that the larger the contrast value you use, the wider the disparity will be from the darkest and lightest pixel values.

## Exposure

In photography, exposure is the amount of light you allow allow through the lens. If you are in a low light situation, such as astrophotography, you want the exposure set very high. In full light situations like a mid-afternoon picnic, you need to tamp down the exposure to avoid having your image be blown out.

Here is the shader that we use to emulate exposure:

```fragment half4 exposureFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant ExposureUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

return half4((color.rgb * pow(2.0, uniform.exposure)), color.a);
}
```

This formula utilizes a new math function: the `pow` function. The `pow` function takes two parameters:

• The value to be multiplied
• The power to which it will be multiplied

So, for example, if you had

```pow(2, 8);
```

the result would be 256 (two raised to the eighth power).

The shader is taking the base value passed into the shader and multiplying it by two to the power of the exposure value. The exposure value can be anywhere between 0.0 and 1.0. Any number, besides zero, that is raised to the zero power is one. So because of how we clamp the exposure values, the exposure result will always be a value between 1.0 and 2.0.

## Gamma

Light, similarly to sound, are not experienced by humans linearly. Small amounts of light are perceived to be much brighter and increases in brightness at the full end of the spectrum do not appear to be significantly brighter despite having similar proportional increases.

Here is a good link to an article about what gamma is.

```fragment half4 gammaFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant GammaUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

return half4(pow(color.rgb, half3(uniform.gamma)), color.a);
}
```

For our gamma correction, we are passing a value set by the user and raising each of the color channel’s values to that value.

## Saturation

Saturation is how much chrominance is present in an image. In our earlier post about luminance we discussed how to create a monochromatic image. We are going to take this a step further and allow the user to adjust the amount of color they want in their image. Here is the shader:

```fragment half4 saturationFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant SaturationUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

half luminance = dot(color.rgb, luminanceWeighting);

return half4(mix(half3(luminance), color.rgb, half(uniform.saturation)), color.a);
}
```

First we create a three component vector to contain the same luminance weighting we initially created back in our luminance shader. We will use these values to adjust the color saturation.

Back when we simply wanted the luminance, we uniformly applied this value to all three color channels. Now we need to use a portion of this value along with a portion of another value. For this, we need to use another new Metal function to this blog: `mix`.

`mix` takes three parameters:

• First color value
• Second color value
• Percentage of first color

The actual math behind `mix` looks like this:

```T mix(T x, T y, T a)
x + (y – x ) * a
```

The first value is added to the second value after the second value is subtracted from the first value and multiplied by the percentage. In our case, the first value is the luminance value of the color. We want to determine how much of the original color value should be added back to the image. We subtract the smaller luminance value from the original color value and then multiply it by the percentage the user wants.

This shader is a good example of how many shaders in GPUImage are composed and build upon smaller, simpler shaders. One reason for this series of blog posts starting with very simple shaders is to show the reader how to intuitively build more complex shaders.

## RGB

So far all of our shader functions have affected all color channels equally. One powerful aspect of having multiple color channels is that they can be adjusted independently.

In order to adjust each channel independently, we need more than one uniform setting:

```public class RGBAdjustment: BasicOperation {
public var red:Float = 1.0 { didSet { uniformSettings[0] = red } }
public var blue:Float = 1.0 { didSet { uniformSettings[1] = blue } }
public var green:Float = 1.0 { didSet { uniformSettings[2] = green } }

public init() {

uniformSettings.appendUniform(1.0)
uniformSettings.appendUniform(1.0)
uniformSettings.appendUniform(1.0)
}
}
```

This could be slightly confusing, so I’ll break it down a little. We have three uniform variables for each color channel. These are connected to three separate sliders in the UI. Any time one of those sliders is changed, it updates the value of the specific variable it is attached to. This is the same as our previous shaders, but if you look at the initial uniform settings, we append three identical uniforms. We have to explicitly set the uniforms once upon launch, which is why we have those three identical lines in the Swift file. Above in the public variables, we explicitly associate the red, green, and blue slider variables with a “slot” in the uniform buffer. Once the uniforms are initially set, we don’t care anymore about the code in the initializer. The sliders take over responsibility for their own specific slot.

In order to keep this strait on the GPU side, we create a data structure emulating these public variables so the GPU can sort out how the data it’s being sent is laid out:

```typedef struct
{
```

This buffer of data is again passed into the shader as a parameter:

```fragment half4 rgbAdjustmentFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]],
constant RGBAdjustmentUniform& uniform [[ buffer(1) ]])
{
half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

return half4(
color.a);
}
```

Each color channel is multiplied by the percentage associated with the slider in the UI. We access each color channel by name using dot notation.

## Conclusions

Many of the most common image processing functions we take for granted in programs like Photoshop are surprisingly simple. From these simple building blocks we can build many large and impressive effects. These posts might seem like humble beginnings, but big things come from small beginnings.

# Metal Shaders: Luminance

Another shader in the GPUImage framework that doesn’t take any inputs other than the current pixel color is the luminance filter. This blog post will not only go over this very simple shader, but will also go into the science behind how the algorithm was developed.

## How Humans Perceive Color

Luminance is the overall brightness of a specific image. If you look at a black and white image, you’re looking at the light present in the image minus the color. The luminance filter should accurately represent the image minus the color.

You might think that you would need an equal representation of red, green, and blue in the image, but humans perceive the brightness of different colors differently. Specifically, humans are most sensitive to colors within the green spectrum.

In fact, video data is commonly encoded using a YCbCr color spectrum, emphasizing the Y component, which represents luminance. Many image sensors have twice as many green sensors as red and green. This will be covered in more detail in later blog posts.

Fun fact: In the 1960 movie Psycho, the substance used for blood during the stabbing scene in the shower was chocolate syrup. Using a blood red substance for blood does not show up well on black and white film, so they needed to use a darker substance to show up accurately.

The flip side of luminance is chrominance. Chrominance is the saturation and color of the specific pixel. Human beings are more sensitive to brightness than saturation. This makes sense evolutionarily. If you’re wandering around at night where there are a lot of predators, your vision needs to be sensitive to differences in brightness and movement rather than being sensitive to how green a plant is. Color can give you information about whether a plant is poisonous or not, but fine differences in shades of red don’t really give you an advantage the way that brightness does.

in GPUImage 3, we have a file called `OperationShaderTypes.h`. This file contains our shared vertex structures and common constant values used in multiple shaders. One of those is our algorithm for luminance weighting:

```constant half3 luminanceWeighting = half3(0.2125, 0.7154, 0.0721);
```

This algorithm came from Graphics Shaders: Theory and Practice. We use these values in the luminance fragment shader:

```fragment half4 luminanceFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]])
{
half4 color = inputTexture.sample(quadSampler,
fragmentInput.textureCoordinate);
half luminance = dot(color.rgb, luminanceWeighting);

return half4(half3(luminance), color.a);
}
```

The meat of this function is this line:

```half luminance = dot(color.rgb, luminanceWeighting);
```

One of the better explanations I have found on the dot product is on this site. What this line of code is doing is taking the red, green, and blue values of the input pixel and multiplying them by the weights we set for the luminance then summing the results. The red value is being multiplied by 0.2125. The green is weighed by a whopping 0.7154 while the blue only gets 0.0721. These three weights added together equal 1.

If you were to calculate the luminance of white, it would look something like this:

```(1.0 * 0.2125) + (1.0 * 0.7154) + (1.0 * 0.0721) = 1.0
```

The goal is that you do not wind up with a value that can ever be above 1.0. If all of the red, green, and blue values were weighted equally, then a desaturated red would look exactly the same as desaturated green and desaturated blue. Even though these would all be the same shade of gray, it would still look off to our eyes because we perceive these colors differently, as seen below:

This weight is then applied as the value for all three color channels to ensure the image is a shade of gray:

```return half4(half3(luminance), color.a);
```

## Conclusions

This is still a relatively simple shader. Many of the shaders in the GPUImage framework build off of luminance, so understanding this concept will help you later with more complex shaders.

Next, we’re going to start covering shaders that require properties to be shared between the CPU and the GPU and you will learn how to encode those values into buffers.

# Metal Shaders: Color Inversion

This is the first in a series of blog posts about the math behind image manipulation filters used in GPUImage 3. I am hoping that these posts will give the reader a good foundation around common shader functions that can be used to build more complex shaders.

One of the most basic color manipulation functions is color inversion. Color inversion required just a single input: the original pixel color. The color is inverted and the value is returned to the rest of the rendering pipeline.

How Does Color Inversion Work?

In order to work with and understand shaders and image/graphics manipulation, you need to understand how the computer processes images.

If you’ve ever worked with Photoshop, you’ve probably worked a bit with the color tools. You were given options for color choice based on a red, green, and blue value between 0 and 255.

Most of the visible color spectrum we can see can be expressed as a mixture of red, green, and blue. Further, most of those values can be expressed using 8 bits of data for the value of each color component. 8 bits represents 256 values, hence the value between 0 and 255.

## Color Representation in Metal

In the Metal Shading Language, the fragment function returns a `half4` value. `half4` is a four element data structure composed of floats at half precision. A regular float has 32 bits of precision and a half float has 16 bits of precision. Metal is natively optimized for 16 bit data types, so use those when possible.

You might be wondering why we have a `half4` if we only have red, green, and blue values. The final value is for the alpha channel, which controls the opacity of the color output.

Apple’s Cocoa frameworks represent colors as a percentage between 0.0 and 1.0. This means that to get the inverse of the percentage of each color, you simply need to subtract the value from 1.0. You can take my word for it, or we can look over a few simple examples of this in practice.

White is created by outputting 100% of red, green, and blue. This is represented as:

```half4 = (1.0, 1.0, 1.0, 1.0);
```

Subtract 1.0 from each of those values and you wind up with:

```half4 = (0.0, 0.0, 0.0, 1.0)
```

That example is pretty easy and self explanatory. Let’s look at a slightly more complex example. Let’s invert blue. To have pure blue on the screen, you have 100% blue and 0% green and red:

```half4 = (0.0, 0.0, 1.0, 1.0)
```

Each of these values is subtracted from one:

Red = 1.0 – 0.0 = 1.0
Green = 1.0 – 0.0 = 1.0
Blue = 1.0 – 1.0 = 0.0

The inverted blue value is:

```half4 = (1.0, 1.0, 0.0, 1.0)
```

This results in yellow:

So far all of these examples have been of either 0% or 100%. Does this still work at values in the middle? Absolutely.

```half4 = (0.5, 0.5, 0.5, 1.0)
```

This is gray. The inversion of gray should stay exactly the same. Let’s try it out:

Each of these values is subtracted from one:

Red = 1.0 – 0.5 = 0.5
Green = 1.0 – 0.5 = 0.5
Blue = 1.0 – 0.5 = 0.5

As you can see, none of these values changed, which is as it should be:

```half4 = (0.5, 0.5, 0.5, 1.0)
```

## Color Inversion Shader

For GPUImage, we didn’t create a separate vertex shader for every fragment shader. Many classes of shaders need the same inputs, so we set up a single vertex shader for all fragment shaders that require a single input. The output value for this single input is `SingleInputVertexIO`:

```struct SingleInputVertexIO
{
float4 position [[position]];
float2 textureCoordinate [[user(texturecoord)]];
};
```

Each of our single input shaders requires the current vertex position and the coordinate of the texture. This is the output of the single input vertex function:

```vertex SingleInputVertexIO oneInputVertex(
device packed_float2 *position [[buffer(0)]],
device packed_float2 *texturecoord [[buffer(1)]],
uint vid [[vertex_id]])
{
SingleInputVertexIO outputVertices;

outputVertices.position = float4(position[vid], 0, 1.0);
outputVertices.textureCoordinate = texturecoord[vid];

return outputVertices;
}
```

The vertex function is pulling in the position and texture that were encoded into the buffers on the CPU side. Since most of our processing will happen in the individual fragment shaders, the purpose of this vertex shader is to basically pass the current frame to the fragment function.

The final color inversion shader from GPUImage is here:

```fragment half4 colorInversionFragment(
SingleInputVertexIO fragmentInput [[stage_in]],
texture2d inputTexture [[texture(0)]])
{
half4 color = inputTexture.sample(
fragmentInput.textureCoordinate);

return half4((1.0 - color.rgb), color.a);
}
```

The fragment function has two parameters:

• The current interpolated position
• The current texture

The texture tells us what image we’re processing and the position tells us which specific pixel the fragment shader will be processing.

First, we’re creating a sampler to sample from the texture. Next, we’re creating a variable to hold the current sample color by referencing the texture sampler at the specific position coordinate we received in the parameters.

Finally, we are doing our color inversion calculation. The first three values are the red, green, and blue values we are inverting. The final value is the alpha/opacity value. We do not want to invert that value, so that is simply passed through as is.

## Conclusion

Sorry if this post feels like beating a dead horse by stating the obvious. For me personally, I had to change the way I think about programming to grok how to create shaders. I found that breaking down a shader into these simple truths and components, it helped me to see that there was a reason this formula exists instead of just copying and pasting an algorithm online.

With graphics, everything is expressed mathematically. It’s important to realize that the people who wrote these algorithms were attempting to create an effect and had to think about how to accomplish that mathematically. These aren’t magic. Every shader I go through for the rest of this series builds on the ideas I express here.

# Introducing GPUImage 3

Yesterday I presented a talk at iOS Dev Camp DC about, what I call, “Metal with Training Wheels.” Apple has introduced a lot of abstracted frameworks on top of Metal to allow you to take advantage of Metal without having to set up the entire pipeline for bit manipulation and memory allocation.

Along with Apple’s build it, abstracted frameworks, I also announced the launch of a new (old) third party framework that works with streaming video, one of the few use cases without an abstracted framework: GPUImage.

## History of GPUImage

Back in 2012, Brad Larson introduced GPUImage to the world. GPUImage started as a sample project about GPU-accelerated live video filtering for Second Conf. So many people requested access to the sample code from that talk that Brad created a full open source project around it.

The project was wildly popular and many people incorporated it into their projects. But then in 2014, Apple introduced Swift. Over the years, many people moved away from wanting to use Objective-C frameworks. Additionally, there are a lot of language features introduced in Swift that cut down on a lot of code within GPUImage. As such, in 2016, Brad introduced GPUImage 2.

GPUImage 2 was leaner than GPUImage, but it was still very backward compatible. With the open sourcing of Swift, Brad felt it was important to keep the rendering pipeline in OpenGL ES to make GPUImage cross platform. He was able to get GPUImage 2 working on a Raspberry Pi using Linux.

With the recent WWDC announcement of the deprecation of OpenGL on the Mac, the writing is on the wall for OpenGL. However, the writing is not on the wall for GPUImage as a framework. GPUImage has evolved in the past and it will continue to evolve to account for evolutions in the Apple developer ecosystem.

## Why is GPUImage Still Relevant?

From one perspective, it might make sense to retire GPUImage. When it was introduced, there was no framework for image processing on iOS as Core Image was Mac-only. Even after Core Image came to iOS, it did not initially allow for the user to write their own image processing kernels. Eventually this functionality was introduced. As of iOS 11, you can even write your own kernels in Metal and not the Core Image kernel language.

However, I believe there are several good reasons to update GPUImage for Metal.

The main reason, in my mind, is that people still use the framework. Many applications have been built on GPUImage. I feel it’s important to continue to support those users by making sure their applications don’t suddenly stop working whenever Apple throws the switch on OpenGL ES for iOS.

Another reason is that, despite all of the advances made to the Cocoa ecosystem, there is still not an abstracted framework for live video filtering. Capturing and filtering frames of live video requires you to dig into AVFoundation to capture each frame, convert it to a texture, add it to a processing pipeline, write shaders, and then output it. This year I have had multiple people contact me about doing GPU-accelerated live video filtering. There isn’t really a reason to not have an abstracted framework around this boilerplate code and GPUImage fulfills that niche.

The last important reason, in my mind, is the shader library. Many people are interested in learning about shaders and there simply isn’t a lot of great learning materials for people who are just starting out with shaders. There are 3D math books that explain what vectors are without any context as to how they are used. Most of the GLSL books I have found spend half the book explaining OpenGL specific state pipeline stuff that doesn’t exist in Metal.

The filter library has a wide range of image processing shaders in varying degrees of complexity. It’s an invaluable resource for beginning shader developers to reference to see how a shader is assembled. There are sites like Shader Toy that are way too complicated for a beginner to figure out.

You can use Core Image to quickly get image filtering up and running, but the logic isn’t exposed to the programmer. You can’t really use them to learn how to make complex shaders.

I believe having a framework full of publicly exposed shaders that allows you to easily set up live video filtering is important. I want to see GPUImage weather the transition away from OpenGL. I want people who are interested in Metal to have access to a working project to learn from.

It is my intention to create a series of blog posts about the functionality of most of the shaders. Some shaders are quite similar and may be covered under a single blog post. But it’s my intention for the reader to be able to understand that the shaders are not magic and to be able to think algebraically so they can approach making their own shaders with the right tools.

GPUImage was one of the things that brought Brad and I together. I knew who he was because of GPUImage and I reached out to him because of my interest in graphics programming. I am very happy that he felt comfortable handing the reigns to his framework over to me to shepherd it into the Metal ecosystem. I would like to be a good steward of this framework so that it can continue to be a useful asset and tool for the people who come after me.

# Metal Integration With SceneKit: SCNShadable Fragment Injection

I spent a year working on The Metal Programming Guide for Pearson. Working on this book was a tremendous privilege and I feel quite fortunate to have had the opportunity to work on it. However, while I worked on the book, I kept feeling there had to be a better way to do a lot of the things I wrote about in the book. I didn’t think you should need a hundred lines of code to output a box to the screen.

Everyone I talk to (myself included) who thinks about learning Metal tends to want to do so because we like to think of ourselves as low level programmers. We’ve been told Metal is difficult and we want to make it our bitch. We feel that way until we realize just how many things you have to set up that are the same for nearly everything you are doing. You spend a week trying to get a triangle on the screen. It seems excessive and you wonder why you need to keep writing the same set up code over and over again. Why hasn’t someone created an open source framework to make all this crap easier?

No one has created an open source framework because it already exists.

Apple has two game frameworks that do 2D and 3D graphics specifically for game development. SpriteKit was announced with iOS 7 and SceneKit was originally announced for the Mac. Neither has gotten quite the amount of traction I think they both should have gotten and now they’re old news and not new and shiny any longer. However, they do serve an important purpose.

One major advantage of SceneKit is that it takes care of a lot of boring low level stuff that is tedious to do in Metal. It is fairly simple to set up a lot of work easily in SceneKit and drop down to Metal only when you need to. SceneKit has a lot of functionality that is easily extended by knowing a little bit of Metal.

There are three levels of Metal shader integration with SceneKit:

• `SCNShadable`
• `SCNProgram`
• `SCNTechnique`

`SCNShadable` is the easiest way to quickly integrate Metal shader code into a SceneKit project. `SCNShadable` is a protocol that all built in `SCNGeometry` objects conform to. SceneKit has an extensive library of primitive `SCNGeometry` objects that you may use individually or combined to create more complex shapes. This means that if you want to insert a small bit of Metal code to a cone or a plane, you can create one in SceneKit and inject shader code into the rendering pipeline without being responsible for setting up the whole thing.

`SCNShaderModifierEntryPoint` allows you to customize SceneKit at four different points in the rendering pipeline:

• Geometry
• Surface
• Lighting Model
• Fragment

The remainder of this blog post will focus on the final option, which is the fragment shader. Not only is the fragment shader the most focused on point in the rendering pipeline, it also happens to be the simplest one for us to hack.

## Setting up SceneKit

Since most people are relatively unfamiliar with SceneKit, I will go over the steps you need to take to put together about the most basic SceneKit project possible. The first property you must have for a SceneKit project is an ` SCNView`:

```// Properties
var scnView: SCNView!
```

I like to create functions to set up the various properties on their own, so here is the code to set up the `SCNView`:

```// Setup Functions
func setupView() {
scnView = self.view as! SCNView
scnView.showsStatistics = true
scnView.allowsCameraControl = true
scnView.autoenablesDefaultLighting = true
}
```

The bottom three method calls are not really necessary, but they are nice to add unless you know you absolutely don’t need them. For example, if you were going to roll your own lighting, you probably would not want to auto-enable default lighting. Since that’s a more advanced move, let’s just let SceneKit handle that for now.

Next, you need an `SCNScene`:

```var scnScene: SCNScene!
```

As before, you need to set up this scene in a method:

```func setupScene() {
scnScene = SCNScene()
scnView.scene = scnScene
scnScene.background.contents = UIColor.purple
}
```

Here, you are initializing the `SCNScene` and ensuring that the `scnView` object has set the newly initialized `scnScene` as its scene. I am also specifying for the background color to be purple. If you don’t want your background to be purple, then there is nothing I can do to help your taste level or bring joy to your life.

Finally, we need to be able to see what is going on in our scene, so we need a camera. The camera is an attribute on `SCNNode` so rather than initialize a straight `SCNCamera` you need to create a node for it to be attached to:

```var cameraNode: SCNNode!
```

Finally you set up the camera node:

```func setupCamera() {
cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
cameraNode.position = SCNVector3(x: 0, y:0, z: 10)
}
```

The camera is placed directly in front of but slightly up from the origin of the scene. It is then added as a child to the `scnScene`.

If you run this code right now none of this shows up. We haven’t called any of these set up methods yet, but more importantly, we haven’t create any geometry yet. That is what we’re doing next.

## Adding the SCNGeometry Object and Custom Fragment Shader

SceneKit has a large variety of built in primitive objects. These include:

• Boxes
• Cones
• Capsules
• Pyramids
• Spheres

For this demo, I will simply be creating a box. An `SCNBox` has four required properties for initialization:

• Width
• Height
• Length

You need to specify how long, tall, and wide you want your box to be. It could be a square box or it could be a long and skinny box. It’s up to you. You also need to specify how rounded you want the corners to be.

Once you have your `SCNGeometry` object, you can now manipulate its shader modifiers property. Each type of shader modifier has its own data structures. The `SCNShaderOutput` structure for fragment shaders has only one property, which is color:

```struct SCNShaderOutput {
vec4 color;
} _output;
```

Color is represented by a four element vector that has a space for a red, green, blue, and alpha value. By creating a fragment injection you can highjack whatever color would have been determined by the internal fragment shader and substitute your own. To keep things simple, this snippet simply sets the color of the box to red.

```func addBox() {
let box = SCNBox(width: 2.0, height: 2.0, length: 2.0, chamferRadius: 0.0)

// Pure red box in Metal (step 1)
"""
_output.color.rgb = float3(1.0, 0.0, 0.0);
"""
]

let geometryNode = SCNNode(geometry: box)
}
```

Finally, now that you have all of your setup methods in place, you can call them in the `viewDidLoad()` function:

```override func viewDidLoad() {
setupView()
setupScene()
setupCamera()