Many people have asked how to do some q&d projection mapping, so here’s the rundown:
- Put the projector up, screw it in so it doesn’t move.
- Plug in a computer and open a drawing program (I used Gimp, you can duplicate the working working image and output it to the projector to see the full resolution image as you edit it) and draw over the features of the castle to figure out where things need to be in the videos (position of the clock, the tower, etc). This honestly feels surreal, it looks like you’re painting the castle as you draw over it… I also marked parts of the image I didn’t want to project anything to as the projector covers a bit of the walls beside the table. That’s why there are weird diagonal black bars on each side of the image.
- Load your sketch and the video in Natron and start playing the the corner pin transform until you get something close. The Merge block allows you to put the sketch from the previous step at 50% opacity over the video to align the features, but you can make the sketch transparent when you render the final video. The transform is different for each video as the position of the camera relative to the castle is different depending on where the recording was made. The transform also takes into consideration the relative position of the projector and the Lego castle, which would be different if you did it.
- Export the video
- Play it back. My projector plays videos straight from a usb drive, so that was easy enough…
Sketch used to align features 
Natron Graph 
Example distorted frame
This is a VERY poor man’s projection mapping, but the illusion is “good enough” for suspension of disbelief. Normally you’d want to calibrate the projector, get the camera projection parameters from the original video and compute the real transformation, but that’s too much work for a Lego set…
