Augmented reality video projection onto out-of-plane surfaces: tag any surface with a few stickers and use it as a virtual display.

Here's a video demo of what the end result looks like:

1.5 Order Approximation

We use the 1.5 order polynomial method to "warp" the source image onto the target plane. This method requires solving a set of linear equations:

$$
U = a_0 + a_1 X + a_2 Y + a_3 X Y \\
V = b_0 + b_1 X + b_2 Y + b_3 X Y
$$

By substituting the positions of our 4 target dots (X, Y) and the 4 coordinates in our virtual plane (U, V) and solving for parameters a and b, we produce a mapping from the video plane to the virtual plane.

This solution has the unfortunate side effect of not preserving straight lines; however, it is significantly faster than the bilinear and perspective transforms which we outline in the below sections.

Bilinear Transform

Although the 1.5 order polynomial provides a reasonable approximation, the transformation we actually want is shown below:

$$
X = a_0 + a_1 U + a_2 V + a_3 U V \\
Y = b_0 + b_1 U + b_2 V + b_3 U V
$$

Unfortunately, solving this set of equations for U and V results in

$$
U = \frac{X - a_0 - a_2 V}{a_1 + a_3 V} \\
V = \frac{-B \pm \sqrt{B^2 - 4 A C} }{2 A}
$$

This allows us to preserve straight lines, but diagonals - i.e. top-left to bottom-right lines - are still slightly warped.

Perspective Transform

Finally, we can use the perspective transform, which involves solving:

$$
U = \frac{a_0 + a_1 X + a_2 Y}{1 + c_1 X + c_2 Y} \\
V = \frac{b_0 + b_1 X + b_2 Y}{1 + c_1 X + c_2 Y}
$$

Since this is a pain to solve for, and the difference is negligible except at extreme angles - who wants to look at a display at a ~80 degree angle anyways? - we use the 1.5 order polynomial approximation in our implementation.

implementation

Our code is available online at:

https://github.com/k15z/skyi-board

We use OpenCV for manipulating image/video files and Eigen for computing the projection matrix. The actual projection is computed by manually copying pixels over to the target image, so significant performance gains can be achieved by using OpenCL bindings.

board_0

board_1

board_2