speen's space

using pixel sorting to animate photos

September 28, 2023 · photography, programming, art


This was my final project for my photography class for the Spring 2023 semester. This post is an adaptation of a presentation that I gave in May on the work I did for the project. I hope in this format that I can cover things in more depth that I brushed over in the presentation.

what is pixel sorting?

"An image of mountains with a pixel sorting effect applied. The effect is glitchy and produces several gradients along each row of the image."

Pixel sorting is the application of computer sorting to the data present in images to produce the above effect. This particular image was created by Kim Asendorf, who is one of the earliest pioneers of this technique. His original 2010 code can be viewed here.

applying sorting to images

The first step to sorting pixel data in an image is deciding how to sort it. This is done by choosing a sorting function. The responsibility of the sorting function is to take each pixel’s data, and map it to a singular value. While most commonly a pixel is split into red, green, and blue channel, this sometimes makes use of other representations such as HSL or HSV. These are the most common ways that each pixel is mapped to a singular value:

Depending on the composition of the original image, selecting a sorting function can drastically change the output of the pixel sorter.

selectively sorting pixels

"A blurred mess of white, gray, brown, and black."

Although you can’t really tell, this is a pixel-sorted picture of my friends dog, Frog. This is an example of what happens if you apply a pixel sorting filter to the entirety of an image. To prevent an image from becoming a blurred mess, pixel sorting software makes use of a mask to split the image up into distinct intervals that will then be sorted.

"A black and white image of a beagle sitting on a bed."

This is an example of a mask, it was created manually using a threshold filter. Essentially, pixels over a certain brightness has been set to white, pixels below a certain brightness has been set to black. The pixel sorter uses this image to determine what parts of the original source image are to be sorted.

"An image of a black beagle, with a vertical pixel sorting effect applied"

The sorter looks at each column of the image and sorts each interval of white pixels. In this context, the intervals are each contiguous group of white pixels along the columns of the image. Note: Intervals can also be the rows of the image, diagonal lines, or even concentric arcs within the image.

For instance, if we were to label the horizontal intervals for all the white pixels in an image with a different color, it would look like this:

"An animation with two frames. One shows a pixel art picture of a blue fish with a white background. The second frame is that same fish, but the white background has each contiguous horizontal white line of pixels marked in a different color."

creating animations using pixel sorting

To animate the pixel sorting effect, I used some bash scripting to generate multiple images from one source image, where each output image was generated with slightly different parameters.

Consider this bash script snippet that I used for this animation:

for upper in $(seq 1 -.1 0.3);
	pixel-sorter "seaweed.jpg" -a 90 -u $upper

This code uses a for loop to run the pixel sorter with the upper limit of the threshold mask set to 1, 0.9, 0.8…0.3. Then, the images generated from this loop can be stitched together into the frames of a gif.

The programs I ended up using were primarily satyarth’s pixelsort python module and RusticFlare’s pixel sort program written in Java. The use of each program is best explained in the documentation, which I’ve linked.

For combining the frames into a single gif file I used ezgif, as it is well, easy. I also made some of the gifs in ffmpeg, when the file sizes were too large and I didn’t want to compress the frames prior to animation.

That’s pretty much all there is to it. Making these images was fairly easy as a process but the challenge mostly comes with selecting good images and getting the parameters for the program right to achieve the desired effect.

other results

closing thoughts

Image processing takes a lot of time. I hadn’t really thought about it at the time, but if I had scaled down my images before applying the sorting filters I could’ve experimented with the parameters of the programs much more quickly.

Sometime in the future I think it could be fun to try and implement some of these effects on my own. When I was first starting the project, I had the idea to try and implement them in javascript so that they could be accessible through my site, but I never got around to it.

If you have any questions about this project, reach out! I’d be happy to help anyone with their attempts to reproduce this effect.