Preface

In the past, people's perception of the Galacean Engine has always been in the 3D field. We have supported the implementation of many 3D interactive projects. As the number of businesses we serve increases and their complexity grows, providing only 3D capabilities can no longer fully meet business needs. Therefore, this year we began to expand our 2D capabilities. The most basic components in 2D are SpriteRenderer and SpriteMask. In engine version 0.3, we have completed the refactoring of SpriteRenderer, and this article mainly shares the development process of SpriteMask. The final effect is as follows (the left image is the inner mask VisibleInsideMask, and the right image is the outer mask VisibleOutsideMask):

Research

The main function of SpriteMask is to collaborate with SpriteRenderer to achieve the sprite masking effect. Before starting formal development, we first conducted research from two aspects: how some engines in the industry use masks from the developer's perspective, and what technical solutions are available for mask implementation at the underlying level.

Usage Methods

From the developer's perspective, the usage methods of masks in the industry can be roughly divided into two types: based on the node tree hierarchy and based on the rendering order.

Based on Node Tree Hierarchy

The usage method based on the node tree hierarchy is roughly as follows:

The mask will affect all rendering components in its child nodes. This usage method relies heavily on the node tree hierarchy. When a sprite needs multiple masks, multiple layers of masks need to be nested. Moreover, if a mask needs to be dynamically changed, the structure of the entire node tree may also need to be adjusted accordingly.

Based on Rendering Order

In the usage method based on rendering order, the mask will set the rendering range affected by the two masks [front, back) through some parameters. Combined with the rendering order of the sprite (taking the screen outward as the positive direction of Z, when two sprites overlap, the one with a larger Z will be rendered on top, covering the one with a smaller Z), it is roughly as follows:

It can be seen that the mask is strongly related to the rendering order, making it relatively natural to implement but not flexible enough. For example, in the above image, if we want the mask to affect the sprite with Z=0 while keeping others unchanged, it cannot be achieved.

Based on Mask Layers

Whether based on the node tree hierarchy or rendering order, both are not flexible enough. The masking of SpriteMask on SpriteRenderer will be affected by some external factors, such as the node tree hierarchy or rendering order. We hope that SpriteMask can quickly match with SpriteRenderer (matching: a SpriteMask can mask a SpriteRenderer is called matching) without being affected by external factors. Therefore, we designed the concept of mask layers in the usage method. When the mask layer affected by SpriteMask intersects with the mask layer where SpriteRenderer is located, they can match, as follows:

Technical Selection

The masking capabilities implemented in the industry mainly include: rectangular mask, rotated rectangular mask, image mask, geometric polygon mask, and inner and outer masks. Since Galacean Engine is a mobile-first web graphics engine, we can implement various masking effects based on WebGL. The main solutions include: stencil, framebuffer, scissor, and shader. Next, we will consider both functionality and performance.

Functionality

From the perspective of functionality, the comparison is as follows:

• [x]

|

• [x]

|

• [x]

|

• [x]

|

• [x]

| | framebuffer |

• [x]

|

• [x]

|

• [x]

|

• [x]

|

• [x]

| | scissor |

• [x]

| | | | | | shader |

• [x]

|

• [x]

|

• [x]

| |

• 
  |

Performance

From the perspective of functionality, we can exclude the scissor and shader solutions. Next, we need to compare the stencil and framebuffer solutions in terms of performance. We implemented the stencil and framebuffer solutions using WebGL, continuously increasing the number of masks, and calculated the average time per frame over 100 frames (unit: ms). The results are as follows:

Test Environment Device: MacBook Pro Processor: 2.4 GHz Quad-Core Intel Core i5 Browser: Chrome 90.0.4430.212

For detailed test examples: stencil: https://codepen.io/chengkong/pen/poPerVy framebuffer: https://codepen.io/chengkong/pen/xxdYNro

Conclusion

Through the comparative analysis from two dimensions, from the perspective of functionality, we can exclude other solutions, leaving only stencil and framebuffer. From the perspective of performance, the performance of the framebuffer solution is almost 10 times slower than that of the stencil solution. Therefore, we finally decided to use the stencil solution to implement masking.

Key Design and Implementation

After the research is completed, the usage and technical solutions are clear. Next is the design of the core classes. Here is a brief introduction to several core concepts that need to be understood: mask layer, mask area, and mask type.

Design

The final usage by developers is as follows:

const sprEntity = rootEntity.createChild("Sprite");
// 1.1 添加一个 SpriteRenderer
const renderer = sprEntity.addComponent(SpriteRenderer);
renderer.sprite = sprite;
// 1.2 设置遮罩类型
renderer.maskInteraction = SpriteMaskInteraction.VisibleInsideMask;
// 1.3 设置精灵所属遮罩层
renderer.maskLayer =  SpriteMaskLayer.Layer0;
 
const maskEntity = rootEntity.createChild("Mask");
// 2.1 添加一个 SpriteMask
const mask = maskEntity.addComponent(SpriteMask);
// 2.2 设置遮罩区域
mask.sprite = maskSprite;
// 2.3 设置影响的遮罩层，和精灵所属遮罩层进行匹配用
mask.influenceLayers = SpriteMaskLayer.Layer0;

The relationship diagram of related classes is as follows:

Mask Layer

The mask layer determines how SpriteMask and SpriteRenderer quickly match. Let's first define all the mask layers as follows:

/**
 * Sprite mask layer.
 */
export enum SpriteMaskLayer {
    /** Mask layer 0. */
  Layer0 = 0x1,
  /** Mask layer 1. */
  Layer1 = 0x2,
  .
  .
  .
  /** Mask layer 31. */
  Layer31 = 0x80000000,
  /** All mask layers. */
  Everything = 0xffffffff
}

There are a total of 32 mask layers. Why??? Although the Number type is 64-bit, all bitwise operations are performed on 32-bit binary numbers. Each bit can represent a layer, allowing us to quickly filter through bitwise operations. Reserving 32 mask layers in a scene should meet all needs (I haven't encountered any project that uses so many masks simultaneously ^-^). Next, we add mask layer-related properties to SpriteRenderer and SpriteMask as follows:

class SpriteRenderer extends Renderer {
  /**
   * The mask layer the sprite renderer belongs to.
   */
  get maskLayer(): number;
  set maskLayer(value: number);
}
 
class SpriteMask extends Renderer {
  /** The mask layers the sprite mask influence to. */
  influenceLayers: number = SpriteMaskLayer.Everything;
}

Mask Area

In the current version, we plan to first implement image masking, where the mask area is determined by the image set for the mask. Therefore, we add a property to SpriteMask to set the mask image as follows:

class SpriteMask extends Renderer {
  /** The mask layers the sprite mask influence to. */
  influenceLayers: number = SpriteMaskLayer.Everything;
  
  /**
   * The Sprite used to define the mask.
   */
  get sprite(): Sprite;
  set sprite(value: Sprite);
}

Mask Type

After designing the mask layer and clarifying how SpriteMask and SpriteRenderer can quickly match, the next important design is whether the masked sprite should display the content inside or outside the mask area. First, we define the enumeration of mask types as follows:

/**
 * Sprite mask interaction.
 */
export enum SpriteMaskInteraction {
  /** The sprite will not interact with the masking system. */
  None,
  /** The sprite will be visible only in areas where a mask is present. */
  VisibleInsideMask,
  /** The sprite will be visible only in areas where no mask is present. */
  VisibleOutsideMask
}

The choice of mask type should be decided by SpriteRenderer, so we add a property in SpriteRenderer to mark it, as follows:

class SpriteRenderer extends Renderer {  
  /**
   * Interacts with the masks.
   */
  get maskInteraction(): SpriteMaskInteraction;
  set maskInteraction(value: SpriteMaskInteraction);
  
  /**
   * The mask layer the sprite renderer belongs to.
   */
  get maskLayer(): number;
  set maskLayer(value: number);
}

Implementation

Let's first look at the final implementation in the entire rendering pipeline as shown in the flowchart below:

Mask Layer Matching

Basic Principle

Although SpriteMask inherits from Renderer, when calling _render every frame, we do not directly send SpriteMask into the rendering queue but cache it in the rendering pipeline, as follows:

export class SpriteMask extends Renderer {
  _render(camera: Camera): void {
    // ...
    
    // 如果是 SpriteMask 渲染组件，直接在渲染管线中缓存
    camera._renderPipeline._allSpriteMasks.add(this);
    
    // ...
  }
}

Why design it this way? To answer this question, we need to first understand how the Galacean Engine sends the content to be rendered into the final rendering, as follows: Generally, after the rendering component throws itself into the rendering queue, it is just a bunch of rendering elements for the entire rendering pipeline. After the rendering queue is sorted, it will be rendered one by one (the green part in the flowchart). So far, we still cannot explain the above question. Don't worry, let's see how to use stencil to implement the mask. We always set the reference value of the stencil test to 1, as follows:

1. Send all SpriteMasks that affect the sprite to the GPU for stencil testing and update the stencil buffer value.
2. When rendering the sprite, choose the comparison function (gl.stencilFunc) according to the mask type.
3. Pixels that pass the stencil test can be rendered.

Did you find the problem? The first step requires sending all the SpriteMasks that affect the sprite to the GPU. Suppose a SpriteMask affects two different sprites, it must be sent twice. According to the existing rendering process, it is obviously impossible, so we need to cache the SpriteMask separately (the blue part in the flowchart). When rendering a sprite, find all matching SpriteMasks to update the stencil buffer.

Optimization Techniques

Here is a question to consider. Suppose we render two sprites consecutively, but the SpriteMasks matched by the two sprites differ by only one. In this case, there is no need to update the stencil buffer one by one. Just do a diff between the mask layers of the two sprites. This can effectively reduce interaction with the GPU. Based on this, we add SpriteMaskManager to handle this part of the logic. The core idea is to record the mask layer of the previous sprite (called preSprite). When rendering a new sprite (called curSprite), find the difference between the mask layers of the two sprites, divided into 3 situations: commonLayer, addLayer, reduceLayer. commonLayer is the overlapping layer of the two sprites, addLayer is the layer that curSprite has more than preSprite, and reduceLayer is the layer that curSprite has less than preSprite. The relationship is as follows:

The core code to find the mask layer difference is as follows:

const commonLayer = preMaskLayer & curMaskLayer;
const addLayer = curMaskLayer & ~preMaskLayer;
const reduceLayer = preMaskLayer & ~curMaskLayer;

Next, we need to find the corresponding SpriteMask through the mask layer difference and then perform the corresponding operations. SpriteMask identifies which mask layers it will affect through influenceLayers. Therefore, we only need to perform simple bitwise operations with the above 3 layers. The core code is as follows:

// Traverse masks.
for (let i = 0, n = allMasks.length; i < n; i++) {
  const mask = allMaskElements[i];
  const influenceLayers = mask.influenceLayers;
 
  // Do nothing for commonLayer.
  if (influenceLayers & commonLayer) {
    continue;
  }
 
  // Stencil value +1 for mask influence to addLayer.
  if (influenceLayers & addLayer) {
    const maskRenderElement = mask._maskElement;
    maskRenderElement.isAdd = true;
    this._batcher.drawElement(maskRenderElement);
    continue;
  }
 
  // Stencil value +1 for mask influence to reduceLayer.
  if (influenceLayers & reduceLayer) {
    const maskRenderElement = mask._maskElement;
    maskRenderElement.isAdd = false; 
    this._batcher.drawElement(maskRenderElement);
  }
}

Mask Area

When a SpriteMask matches, the stencil buffer needs to be updated. For addLayer, we need to add 1 to the corresponding position in the buffer, and for reduceLayer, we need to subtract 1 from the corresponding position in the buffer. The core code is as follows:

// Set the op that the stencil test passed.
const stencilState = material.renderState.stencilState;
const op = spriteMaskElement.isAdd ? StencilOperation.IncrementSaturate : StencilOperation.DecrementSaturate;
stencilState.passOperationFront = op;
stencilState.passOperationBack = op;

Mask Type

After finding all SpriteMasks through mask layer matching and updating the stencil buffer data, we need to set the stencil test function according to the set mask type. The core code is as follows:

if (maskInteraction === SpriteMaskInteraction.None) {
  // When the mask is not needed, the stencil test always passed.
  stencilState.enabled = false;
  stencilState.writeMask = 0xff;
  stencilState.referenceValue = 0;
  stencilState.compareFunctionFront = stencilState.compareFunctionBack = CompareFunction.Always;
} else {
  stencilState.enabled = true;
  stencilState.writeMask = 0x00;
  // When a mask is needed, set ref to 1, inside mask ref <= stencil, outside mask ref> stencil.
  stencilState.referenceValue = 1;
  const compare =
        maskInteraction === SpriteMaskInteraction.VisibleInsideMask
  ? CompareFunction.LessEqual
  : CompareFunction.Greater;
  stencilState.compareFunctionFront = compare;
  stencilState.compareFunctionBack = compare;
}

Summary

Currently, our SpriteMask only implements the ability to mask images, which is sufficient to meet most needs. In the future, based on developers' actual needs, we will consider whether to support rectangular masks, elliptical masks, custom shape masks, etc. Additionally, masks will support the entire 2D ecosystem, not just limited to SpriteRenderer.