Screen-space deformations in XNA for Windows Phone 7

Working on our games, we came across the need to do some screen-space deformations for all sorts of special effects like explosions, ‘drunk filter’, warp effects and other similar stuff. But since we’re targeting Windows Phone 7, we don’t have access to any custom shaders (which is a bummer), so we had to be creative. In the end, the solution turned out to be rather simple, and the performance is good enough (if you don’t go overboard with the effects, of course).

Yearning for pixel shaders, we looked over some effects we wanted to achieve and realized that most (though not all) of them simply involved modifying texture coordinates in the pixel shader. The straightforward idea was to map the final image to a grid with cells having a size of 10×10 pixels. For the next step, the first thought was to modify the positions of the vertices in the grid. However, this approach led to severe artifacts when the vertices overlap or when the triangles defined by them get their orientation reversed. It was soon clear that the best way was to process the texture coordinates on each vertex of the grid, just as you would do in the pixel shader. While the precision is not quite ‘per-pixel’ (duh!) the results look quite well, especially on a small screen (like the phone).

Having the grid set up, it’s easy to go through each vertex, make the adjustments to texture coordinates based on what effect you want to achieve, and then draw the original image with the new texture coordinates. You can also apply several effects to the same grid, and let them combine.

The Grid

Let’s look at some code first. We started by creating a class to hold the grid, which we called GridVertexPositionColorTexture. Yes, it is a grid containing VertexPositionColorTexture elements There’s nothing special in it, except for the function for initializing the geometry (i.e. the vertex buffer and the index buffer), the function for drawing the grid, and functions for getting and setting the texture coordinates of a given element. The full source code can be found in the accompanying archive.

public class GridVertexPositionColorTexture
{
    public VertexPositionColorTexture[] Vertices { get; set; }
    public short[] Indices { get; set; }
    public int Width { get; protected set; }
    public int Height { get; protected set; }
    public Vector2 CellSize { get; protected set; }

    public void BuildGeometry(int colums, int rows, Vector2 cellSize)
    {...}

    public void Draw(GraphicsDevice graphicsDevice)
    {...}

    public void ResetUVs()
    {...}

    public Vector2 GetUV0(int index)
    {...}

    public void SetUV0(int index, Vector2 value)
    {...}

    public Vector2 GetDefaultUV(int index)
    {...}
}

Before writing and applying any effect, let’s setup the Game class to load a texture and draw it over the whole screen using the grid defined above.

Choose a suitable image, and add it to your content project. Normally, this image will be the result of your scene rendering and drawn into a Render Target. However, this article will only focus on applying the post processing effects on an image, no matter how you obtained it.

You will need to add some members to your Game class, to hold the background texture, to manage the Grid and a BasicEffec to draw the grid.

 GridVertexPositionColorTexture grid;
 Texture2D background;
 BasicEffect basicEffect;

protected override void LoadContent()
{
        [...] //other initializations

        //initialize the Grid
        grid = new GridVertexPositionColorTexture();
        grid.BuildGeometry(48, 80, new Vector2(10, 10));

        //load the background texture
        background = Content.Load<Texture2D>("background");

        //load the BasicEffect, and set it up for orthographic projection
        basicEffect = new BasicEffect(GraphicsDevice);
        basicEffect.TextureEnabled = true;
        basicEffect.Projection = Matrix.CreateOrthographicOffCenter(0, 480, 800, 0, 0, 10);
        basicEffect.View = basicEffect.World = Matrix.Identity;
}

It’s all basic initialization. We set the grid to 48×80, with each cell being 10×10 pixels, thus covering the screen of 480×800. For the BasicEffect we use an Orthographic projection matrix.

Lastly, in the Draw method, add a few lines of code to draw the texture using the grid.

protected override void Draw(GameTime gameTime)
{
    GraphicsDevice.Clear(Color.Black);

    GraphicsDevice.SamplerStates[0] = SamplerState.LinearClamp;
    basicEffect.Texture = background;
    basicEffect.CurrentTechnique.Passes[0].Apply();
    grid.Draw(GraphicsDevice);

    base.Draw(gameTime);
}

You’ll notice we use the LinearClamp sampler state, to avoid having to use power-of-two textures. Running the code now will (normally) show you the chosen background image.

First Run

Now comes the fun part.

The Effect

The effect we will showcase today is what we call the Pinch effect. I personally have no idea if there’s a better name out there for it, but this is what we use internally. So, straight to the code, we have the following method.

public void ApplyPinch(GridVertexPositionColorTexture grid, Vector2 center, float intensity)
{
    Vector2 gridSize = new Vector2(grid.Width, grid.Height) * grid.CellSize;

    for (int i = 0; i < grid.Vertices.Length; i++)
    {
        //Get the position in pixels
        Vector2 pos = grid.GetUV0(i) * gridSize;

        //Get the direction from the current grid vertex to the center-of-pinch
        Vector2 dir = pos - center;
        float length = dir.Length();
        float minDisplacement = -length;
        if (dir.Length() != 0)
        {
            dir.Normalize();
        }

        //Compute the displacement, but not lower than the minDisplacement
        Vector2 displacement = dir * Math.Max(intensity, minDisplacement);
        //Find the new position
        Vector2 newPos = pos + displacement;
        //Set the new Texture Coordinates
        grid.SetUV0(i, newPos / gridSize);
    }
}

What we do is take each vertex, find its position, transform it and save it back into the vertex’s data as a new set of Texture Coordinates.

For the pinch effect that we’re after, the transformation is rather basic: compute the normalized unit vector towards the center, and displace it by a factor indicated by the ‘intensity’ parameter. The only extra step is making sure the displacement is not below a certain threshold, otherwise some artifacts can appear for negative intensities.

Now go to the Update function and add in the following code.

protected override void Update(GameTime gameTime)
{
    grid.ResetUVs();
    ApplyPinch(grid, new Vector2(240, 400), 100);
    base.Update(gameTime);
}

Running the game now should show you the effect in action.

Simple pinch

That’s rather dull, isn’t it? With a little bit of work we can make it more interesting. Just update your Update code like below, to add pinches based on the touch positions on the screen, and use a sine wave for the intensity.

protected override void Update(GameTime gameTime)
{
    grid.ResetUVs();
    float intensity = (float)Math.Sin(gameTime.TotalGameTime.TotalSeconds) * 100;

    TouchCollection collection = TouchPanel.GetState();
    foreach (TouchLocation point in collection)
    {
        if (point.State == TouchLocationState.Moved)
        {
            ApplyPinch(grid, point.Position, intensity);
        }
    }
    base.Update(gameTime);
}

The following looks much better in motion.

That’s the basic technique. All you need to do now is do some over-engineering to create a complex system of classes that can be used to apply various effects to an arbitrary input texture, and then plug that into your games. And of course, vary the parameters of the effects based on what the user does. For some practice to put you on the right path, try the following exercise:

Aaaand… this is the end of the article. You can download the source code here.

Obligatory legal stuff: All code is released under Ms-PL, so you can use it freely. However, you may not redistribute and/or use the art assets ( background.png ) beyond your own educational purposes.

See you next time for a small demo of using Alpha Blending to obtain the negative of an image. ( Oh shaders, how I miss you! )