Sky replacement in video with Python and AI (epic!)

Sky replacement in video with Python and AI (epic!)

In this article, I will show you how to create dynamic sky replacement in video with static images. The tools that I use in this guide are Python 3.7 and vision-based AI open-source SkyAR library of Zhengxia Zou.

I think that SkyAR is an amazing neural network! It can automatically replace and repaint the sky in video with a background image template. And that stuff goes on in real-time!

When I look at a boring code description at Github, sometimes I don’t even want to try it. But this work is a good case when there is something to show in a code presentation. Look at these “District 9 ship” and “castle in the sky”!

SkyAR works fast and pretty easy to set up and run. I’m going to show you how I took a cloudy sky from my video and replaced it with a futuristic sky.

So, let’s get started!

Setting up your python project

All the demonstration code was tested with Python 3.7 and CUDA 10.1 drivers on Windows 10. It should work with other setup, but there may be subtle differences.

I’m assuming that you’ve already got some experience in Python.

Clone the repo

git clone https://github.com/jiupinjia/SkyAR.git
cd SkyAR

Download the pre-trained sky matting model from Google Drive, and unzip it into the repo directory.

Setup virtual environment

A virtual environment enables you to install packages in an isolated environment. It’s very useful when you want to try some packages or projects without messing with your system-wide installation. You can learn more about them in Python Documentation.

Run the following commands to make and run a virtual environment inside the SkyAR directory:

python3 -m venv skyar
source ./skyar/bin/activate

When you activate a virtual environment, any package that you install will be installed inside the environment without affecting your system-wide installation.

Install dependencies

pip install -r Requirements.txt
If you are facing errors with PyTorch on this step you should go to PyTorch’s official site and choose build for your drivers and system configuration.
Choose the PyTorch build
Choosing the PyTorch build

Getting started to sky replacement in video

The main thing is to choose the proper sky visibility in the experimental video. I realized from my tests that video sample must meet the following requirements:

  • the video should be daytime with good illumination;
  • the sky should not drop from the frame at the start of the video;
  • the horizon rotation in the video should neither be rapid nor exceed 180 degrees around the shooting point;
  • the sky should not be blocked by a lot of small objects such as trees, buildings, lighting poles, etc.

Getting a glitch-free epic video is guaranteed if you follow these tips.

Configure your settings

Project settings located at ./config/ in json files. I use the source file config-annarbor-supermoon.json and change the path to my video which I added to the ./test_videos/ folder.

...
"datadir": "./test_videos/wakeboard.mp4",
...

Also, you can try to play with other settings. For example, you can change the picture of the sky (“skybox”) or a different blending style (“relighting_factor” and the following).

Starting sky replacement

To start the process of sky replacement in your video, simply run:

python skymagic.py --path ./config/config-annarbor-supermoon.json

Processing time may vary depending on your system configuration, video length and quality. It took me about 30 seconds on the ASUS AMD Radeon RX 580 8GB video card. Your video will appear in the root folder of SkyAR.

Finally, look at how the SkyAR handled my video:

From time to time the background picture of the sky is a little wobbly when the camera is moving. But in general, it turned out to be quite good. Quality is quite suitable to post the video to Insta Story or Tik Tok and surprise friends, isn’t it?

Possible issues

If you are facing issues with video codecs or framerate of your video go to ./skymagic.py to line 36 (self.video_writer).

self.video_writer = cv2.VideoWriter('demo.mp4', cv2.VideoWriter_fourcc(*'MP4V'), 20.0, (args.out_size_w, args.out_size_h))

You can change the video codec and framerate of future videos. If you have questions about parameters, you can read the OpenCV documentation.

Conclusion

The SkyAR is a great library to play with python at leisure. I am sure that the author will finalize some shortcomings. I will be excited to see it applied to any of the industrial products shortly. Maybe like a plugin for Adobe After Effects sky replacement or filter for Instagram or Snapchat.

Siarhei Palishchuk

Siarhei constantly looking for exploiting google with the best SEO practices. A lifelong tech geek and cryptocurrency enthusiast.

Comments