Jan 23, 20237 min read

LAYER MASKS

Updated: Jan 29

Layer Masks are arguably the most powerful, most frequently used tool in the whole Photoshop universe. We'll create pixel-based Layer Masks to achieve different results.

LESSON #2 : Creating a file + Learning the layout, tools, and shortcuts.

Download the files below and follow directions listed.

Layer Masks are used to:

Erase (mask) areas of a photo.
Select an area of a photo to apply an effect.
Apply Adjustment Layers selectively.
Combine photos in one canvas.
…and much, much more.

The benefit of using a mask is that it's part of a non-destructive workflow in Photoshop. The alternative would be to erase pixels, rather than hiding (or masking) them, which is never advisable.

Change This Setting

Go Photoshop menu > Preferences > Tools. Check the box indicated above.

Important Keyboard Commands

d = Default colors in foreground/background swatches.
x = Swap foreground/background swatches.
⌥-delete = Fill with foreground color
⌘-delete = Fill with background color.
⌘-a = Select all.
⌘-d = Deselect all.
Shift-⌘-i = Invert selection.

Raster Layer Masks

A raster layer mask is a greyscale image, where darker pixels hide the underlying photo to increasing degrees. It works a bit like a dimmer on a light switch. The darker the pixels are on the mask, the less you can see the photo on that layer.

On a mask, darker the gradient gets, the more the photo will be hidden.

You can mask content by:

creating a mask, then painting with black, white or grey.
creating a selection, then creating a mask.

There are more techniques for creating masks, but these two will get you started.

Simple Mask

Lets start with replacing the background on a photo by hiding the original one to reveal the one on the layer below. In this case, we’ll change the background to a beach scene.

In the Properties panel, click on Remove Background. Voilà!

You can use Select & Mask to refine your mask’s edge. Simply double-click on the mask’s icon in the Layers panel to access it.

Refine Edge tool
Choose preview mode
Set auto edge detection
Sharpen or feather edges
Set how to output new mask

Use the Refine Edge tool to paint along the edge. Photoshop will detect edges, then try to guess how to enhance the selection.

Masking to Blend Photos

We can also use masks to blend photos. We’ll use the Gradient Tool on the mask. Since black hides and white reveals, this will make the photo on the top layer gradually become transparent to partially reveal the photo below.

A black to white gradient replaces itself each time to drag one. A black to transparent gradient adds to itself each time you drag one.

A Layer Mask is a greyscale image. Where the pixels are dark on the mask, they’re hiding whatever’s on the layer below. Where they’re light, they reveal what’s on the layers below. It stands to reason that if we paste a photo onto a mask, it will do the same, hopefully giving us interesting results.

Masking with a Photo

Masking Hair

The goal of this assignment is to cut an object or person out of a photograph, despite very fine detail. A clipping path would be unmanageable and painting with the brush tool is plain ‘ol ugly.

One of the strengths of masking in Photoshop is that you can use the characteristics of the image to create a mask. By delving into the channels, we can use higher-contrast areas to mask even the most whispy hair.

Your Workspace

For this exercise, its your decision, but it’s best to clear the panels and only show the ones we want. To that end, hit Shift-Tab. All panels disappear except for the Options bar and tools. Now go to the Window menu, then select the Layers and Channels only.

Now go Window > Workspace to save the workspace. Name it Masking. Now every time you wish to do this type of work, simply select this workspace.

*** This is where keyboard shortcuts come in handy. If you know your Zoom shortcuts, you can zoom in and out to check your work, while in a modal dialog. Hint: ⌘-space is your friend. ***

If you hold Control-Option as you hold down the Brush Tool, you'll get this handy heads up display of your brush tip.

Manual Vs. One-Click Methods.

Considering that Adobe is automating many of these challenging tasks, we always want to compare what we can do manually versus what Photoshop can do automatically. We want to gauge whether it’s still worth going the long way, or just clicking a button in Photoshop.

On the left, you see the manual method of masking hair. On the right, I used Photoshop's Remove Background function.

The above image proves that the automated method still has a way to go before it reaches the quality of a manual selection.

Layers

An Alpha Channel in Photoshop is a saved selection represented as a greyscale image. In the panel below, what’s white on the channel is selected. What’s black is deselected. What’s grey is a partial selection – kind of like onion paper. The lighter the color on the channel, the more it’s selected.

The Channels panel has stored selections called Alpha Channels.

The best way to start is to leverage the present content of the photo. We need to increase the contrast of the photo until we end up with the subject in solid white and the background in solid black (or vice-versa). The black and white version will be loaded as a selection which will be used in a mask.

Throughout this process, we’ll pay special attention to never touch the original layer.

The first step in the process is the simplest way to increase contrast. Simply duplicate the background layer with Option-⌘-J. Name it High Contrast. On the new Layer, go ⌘-L or Image >Adjustments >Levels.

We create a high-contrast version of the photo to separate the subject from the background.

Move the Black Point + White Point sliders inward until the subject gets darker. Stop before the background gets dark. We’re just trying to punch-up the subject a little. Click OK.

The result we want is getting the background as white as possible and her as dark as possible without losing any of the whispy hairs. Note that you can zoom while the Levels dialogue is open.

Channels

The Channels panel is where most of the magic happens. While still on our new darker layer, go to the Channels panel. Find the channel where the subject has most contrast with the background. The edges are what counts.

Duplicate that channel with a right-click. We’re going to kind of ruin this channel in the process, so let’s call it Damaged.

Once on this channel, we need to run Levels with ⌘-L. In this case, we want to be way more extreme. Bring the black + white point sliders in until we get the subject as black as possible.

We want to stop with the Levels adjustments before we start to loose detail in the whispy hair, so zoom in to check before you accept.

Now we have a subject which is almost completely black. Some highlights probably still remain. We need to get rid of them. I simply use a big brush and paint with pure black. Make sure you stop before you paint over the whispy edges of the subject. Just fill in the center with black.

Use a soft-edged brush. Just be careful when you get close to the edges. You don’t want to paint any black on the white background.

Wrap-up

By now, you should have a completely black subject, except for the areas where you can see through the hair.

Now we can go Select > Load Selection. From the dialogue, choose your new channel’s name. Click OK. You should see marching ants on your canvas.

Go back to the layers panel.
Turn off the visibility of the Damaged layer.
Double-click on the Background Layer.
Name it something meaningful. Click OK.
Add a mask on this layer by clicking on the Add Layer Mask button at the bottom of the Layers panel.
Voila!

With a contrasting layer behind your newly-masked subject, you may notice that there are some areas of the photo that are transparent and shouldn’t be. Simply go to the mask and paint them white.

You also have the option of running a Levels command on the mask. This can fine-tune it. But be subtle at this point.

You’re done!

The Process

Duplicate main layer
Run Levels
Duplicate contrast channel
Run Levels
Paint black (or white)
Load selection
Create layer mask
Optionally run Levels