Timelapse/Processing

Overview

Procedure

The procedure for post-processing videos looks like this:

• Obtain and wrangle a large number of sequentially numbered jpeg files (xargs)
• Turn mass of jpeg files into video (ffmpeg)
• Downsample (xargs/unix)
• Figure out what effects to apply - single jpeg file (lightroom)
• Apply desired effects en-masse - all jpeg files (lightroom)
• Make more videos (ffmpeg)

Tools

ffmpeg: One of the most useful tools for post-processing video is ffmpeg. There are some notes on ffmpeg on the wiki already: Ffmpeg. Most of the material here will be a variation on that.

xargs: Another useful tool for post-processing video is xargs. Notes on xargs on the wiki already: Xargs. This is a unix command-line utility that forks a single process or single command to multiple inputs. This is extremely useful to streamlining image processing, which can be done in parallel. (It is also useful for mass-renaming files.)

ImageMagick: the command-line "convert" tool is the interface for Image Magick, and this is an extremely handy tool if you need to do any mass-manipulation of images from the command line - for example, shrinking, cropping, stretching, filtering, masking, sharpening, blurring. You name it, you can probably do it with ImageMagick. When combined with convert and xargs, you become unstoppable.

There are also some interesting side topics, such as image averaging using the Python_Imaging_Library. This allows for smoother, more stretched-out timelapse videos.

Wrangling Large Numbers of Files

When I finished with my first timelapse, I was left with 16,636 jpg files over an approximately 9 hour period. That's a pretty unwieldy number of files, and at 200 KB a pop, it's also a lot of disk space. That means stitching together movies, applying effects, &c will take a long time.

Normally, I would turn all the photos into a "baseline" movie - one that shows everything, and makes it easy to narrow in on the interesting parts. However, with 16,000 files, this would be a long, big movie. I decided to throw out half the photos (can get more later).

Getting the filename list

With 16,000 files, if you try and run ls *.jpg, ls will raise an error (too many arguments). Instead, just list everything (no arguments), and search for jpg files:

/bin/ls -1 | grep jpg > files

Now if you examine files you'll see the interval between photos is very erratic - it was programmed to be 2 seconds, but it is all over the place - sometimes 3, sometimes 2.

I wanted to throw out every other photo, and I couldn't do it based on the digits or the numbering. The intervals were uneven, and there were multiple hours of photos. So, I decided to just cut out every other photo.

Chopping up the filename list

To remove every other file, I needed to feed the output of the above command, ls -1 | grep jpg, to a program that would print out every other line - good task for a one-liner in sed or awk.

Here's an awk one-liner to print only odd lines (only the even lines in awk, which numbers lines starting at 0):

awk 'NR % 2 {print} !(NR % 2) && /pattern/ {print}'

Here's that script in action:

$ cat multiline
line1
line2
line3
line4
line5
line6
line7
line8
line9
line10

$ cat multiline | awk 'NR % 2 {print} !(NR % 2) && /pattern/ {print}'
line1
line3
line5
line7
line9

Now that can be applied to an external file,

$ awk 'NR % 2 {print} !(NR % 2) && /pattern/ {print}' files

This prints out a list of files to keep. The easiest way to do this is probably to make a directory to hold the downsampled photos, and (if desired) delete the remaining photos. To move the list of photos obtained in the prior step, you can use xargs to feed the filenames to the "mv" command:

xargs -I% mv % my_timelapse/.

To stitch it all together, save your odd-numbered files in a directory, and remove all the other remaining photos:

$ mkdir my_timelapse/
$ /bin/ls -1 | grep jpg > files
$ awk 'NR % 2 {print} !(NR % 2) && /pattern/ {print}' files | xargs -I% mv % my_timelapse/.
$ # optional: remove all other images
$ # rm *.jpg

That awk command starts with the list of all the files (the "files" file), then parses it down to every other one, then passes that reduced list on to xargs.

Before:

$ ls -1 | wc -l
    16636

After:

$ ls -1 | wc -l
    8320

and counting the number of downsampled photos:

$ ls -1 my_timelapse | wc -l
    8318

Applying Effects with Lightroom

How It Works: The Theory

For editing photos - single or multiple - I recommend using a tool like Lightroom, which allows you to "save" your edit steps and apply them en masse to a large number of photos at once.

Start by modifying a single photo. Do this in "Develop" mode. You can adjust brightness, colors, whatever you want. Now select all the photos in that folder by pressing Control+A or Command+A. (Or, use Control+Click or Command+Click and pick the photos you want out of the photo tray on the bottom). Then click the big "Sync" button. This will synchronize all of your adjustments to ALL of the other photos.

Once you're ready, and all the photos have chnages synced, you'll need to EXPORT them. Pick a location, a naming schema, a format, etc. Then export them to a folder. The content of this folder contain your modified time lapse photos and will be used to make a (modified, with effects) time lapse video.

Problems

The problem is, sometimes you don't have control over the camera's brightness setting, or you set it to auto. Or the white balance is also set to automatic, and adjusting the colors to be reasonable on one makes the colors look awful on another. This can make your videos look awful. Case in point: in this video, I picked one frame and brightened it up, and adjusted the colors to eliminate an awful green tint.

Then I applied those changes to the rest of the photos, and made them into a video. Here's the awful result:

Raspberry Pi Timelapse: Adjusted Colors (Yuck) from charlesreid1 on Vimeo.

Creating Video from Image Files

These notes are based on the notes at the Ffmpeg page.

Renaming Files

One of the quirks of ffmpeg is that, if you are converting a large number of files, they have to be sequentially numbered, starting with 0 OR 1.

Renaming Files: Test Run

The best way to do this is to put all the photos you want to add to your timelapse into one big folder, in the order you want them. Then do a /bin/ls -1 to list them all in order. Then use a bash for loop to name each file in a sequential fashion: 0001.jpg, 0002.jpg, etc.

#!/bin/sh

i=1
for file in `/bin/ls -1 *.jpg`; do
    originalprefix=$(echo "$file" | sed 's/\.jpg//')
    newprefix="img"
    newfile=$(printf "${newprefix}_%04d.jpg" "$i")
    echo "mv ${file} ${newfile}"
    i=$((i+1))
done

The output of this file, run on a dummy directory with 8 dummy files:

$ ./doit.sh
mv f101G1.jpg img_0001.jpg
mv f101G2.jpg img_0002.jpg
mv f101G3.jpg img_0003.jpg
mv f102G1.jpg img_0004.jpg
mv f102G2.jpg img_0005.jpg
mv f102G3.jpg img_0006.jpg
mv f103G1.jpg img_0007.jpg
mv f103G2.jpg img_0008.jpg

Renaming Files: Real Deal

The real filenames were more complicated.

Each file had a six-digit timestamp prefix, then a dash, and then a six-digit time suffix that changed from photo to photo, by random amounts. Like this:

20160730-165955_8313.jpg
20160730-170000_8314.jpg
20160730-170005_8315.jpg

which we wanted to turn into

20160730_8313.jpg
20160730_8314.jpg
20160730_8315.jpg

To do that, we used this modified version of the above script:

#!/bin/sh

i=1
for file in `/bin/ls -1 *.jpg`; do
    filestrip=$(echo "$file" | sed 's/-[0-9]\{6\}\.jpg//')
    newfile=$(printf "${filestrip}_%04d.jpg" "$i")
    echo "mv ${file} ${newfile}"
    mv ${file} ${newfile}
    i=$((i+1))
done

Let's break down that sed expression:

sed 's/-[0-9]\{6\}\.jpg//'

This looks for an expression like -000000 (the [0-9] means any digit, and the \{6\} means 6 in a row), followed by the expression .jpg. This is the six digit hour-minute-second timestamp at the end of the filename, before the suffix. We want to get rid of both the jpg suffix and the non-unique timestamp, so that all the photos will have the same filename prefix.

The double slash means, replace whatever you find with the empty string. A few before/afters:

20160730-165955_8313.jpg
20160730_8313.jpg

20160730-170000_8314.jpg
20160730_8314.jpg

20160730-170005_8315.jpg
20160730_8315.jpg

20160730-170010_8316.jpg
20160730_8316.jpg

20160730-170015_8317.jpg
20160730_8317.jpg

20160730-170020_8318.jpg
20160730_8318.jpg

Using ffmpeg

Now we get to use ffmpeg to stitch these image files together into movies.

A full treatment of ffmpeg would take a lot more space, so we'll leave that to the Ffmpeg page.

Here we'll just go through the bare minimum to make a viable timelapse movie.

Reviewing Filenames

Just a review: ffmpeg is very picky about filenames. It expects files to have a similar prefix, and to have a numbering scheme that starts with 0 or 1. Example of a scheme it likes:

$ /bin/ls -1
20160730_0001.jpg
20160730_0002.jpg
20160730_0003.jpg
20160730_0004.jpg
20160730_0005.jpg
20160730_0006.jpg
20160730_0007.jpg
20160730_0008.jpg
20160730_0009.jpg
20160730_0010.jpg
20160730_0011.jpg
20160730_0012.jpg
20160730_0013.jpg
20160730_0014.jpg
20160730_0015.jpg
20160730_0016.jpg
20160730_0017.jpg
20160730_0018.jpg
20160730_0019.jpg

Make sure that's what you've got. That's what all the filename wrangling nonsense is about anyway.

ffmpeg -i 20160730_%04d.jpg -q:v 1 -vcodec mpeg4 movie_output.mp4

The %04d just means, a 4-digit number that increments, starting at 0 or 1.

Running ffmpeg

When I ran the above ffmpeg command, I saw several lines like this:

$ ffmpeg -i 20160730_%04d.jpg -q:v 1 -vcodec mpeg4 movie1.mp4
[...]
frame= 3670 fps= 58 q=1.0 size=   32379kB time=00:02:26.80 bitrate=1806.9kbits/s speed=2.31x
frame= 4906 fps= 58 q=1.0 size=   42788kB time=00:03:16.24 bitrate=1786.2kbits/s speed=2.32x
frame= 6611 fps= 56 q=1.0 size=   70847kB time=00:04:24.44 bitrate=2194.8kbits/s speed=2.2
frame= 6639 fps= 56 q=1.0 size=   71354kB time=00:04:25.56 bitrate=2201.1kbits/s speed=2.2

This didn't make much sense to me... 56 frames per second? However, the videos is 5:32, or 332 seconds long, and 332 seconds * 26 frames per second gives 8632, much closer to the actual count of photos in the directory. So, the video looks like it is actually 26 fps.

Stock ffmpeg

The "stock" or baseline ffmpeg command I started with was:

$ ffmpeg -i 20160730_%04d.jpg -q:v 1 -vcodec mpeg4 movie1.mp4

This resulted in the movie below:

Raspberry Pi Timelapse: Midnight to Mid-Morning from charlesreid1 on Vimeo.

Slower Framerate/Stop Motion

Next I tried a different frame rate, 15 fps, with the -r 15 flag. This gives it much more of a stop-motion look - I'm not sure how it's filling in all the frames, but it moves at the same rate. In other words, halving the frame rate will not double the length of your video. Frames are simply removed from the original video. (It may be as crude as showing 15 frames, then stopping for 15 frames.)

$ ffmpeg -i 20160730_%04d.jpg -r 15 -q:v 1 -vcodec mpeg4 movie1.mp4

The slower framerate requires a faster capture rate as well - it's great for stop motion, but if people are running across the screen in 4 or 5 frames, you don't get much of a stop motion effect. Much better is to calibrate the frame rate to the height of the camera and the subject matter. A 15th floor camera above a city street can have a slower frame rate due to its wider field of vision and the slower scale of motion from the 15th floor. On the other hand, a 4th floor view of construction workers at a construction site may be closer to the movement, have a narrower range of view, and therefore require a faster frame rate to capture the relative scale of motion.

Proof the timing is exactly the same in both videos: here's a screenshot from both videos of 3:17, when the street lamp is on:

26 fps	15 fps

then of 3:18, when the street lamp turns off:

26 fps	15 fps

Capturing 1 frame every 2 seconds, and setting to 15 fps, almost looked right for stop motion, but everything was too fast by a factor of about 2.

Raspberry Pi Timelapse: Midnight to Mid-Morning (15 fps) from charlesreid1 on Vimeo.

Continued Ffmpeg Notes

Find continued notes on the use of Ffmpeg specifically for timelapses over at the Timelapse/Ffmpeg page.