An Imaginative Computer

Science of Imagination Laboratory

At Carleton University I am the director of the Science of Imagination Laboratory. In it, we try to understand imagination in human beings and computers. We try to make computer programs that imagine the same way people do. If the software behaves similarly to human beings, then it supports our theories of how imagination works.

We're interested in understanding visual imagination, that is, how people picture things. What we think happens when someone is asked, for example, to picture a "dog," is that they refer to the memories they have of seeing dogs, and flesh out a picture based on what else was in those scenes. So, for example, if you're used to seeing dogs outside on leashes, being walked by their owners, then we would expect a person to imagine a sidewalk, a leash, and a person along with the dog in their visual imagination.

So how do we get this to work on a computer?

First, we have a big slew of photographs that have been labeled. 50,000 pictures, with about 12 labels for each picture. So we know what's in the picture and where. We use this as a substitute for human visual memory. That is, we assume that the pictures represent the kind of realistic things that people tend to see during their lives. Of course, this is just a rough and small approximation.

Over the summer we finished a version of the Science of Imagination Lab Imagination Engine, or SOILIE, and published a paper on it. Here's what it can do:

You input a single word, such as "car" or "mouse." SOILIE looks at "co-occurrence" with that word. That is, what other labels tended to be in the same pictures with the query? So, for "car," it might be "road," "sky," and "man." Then SOILIE sees where in these pictures the other elements tend to go. It then grabs pixels from various images in the database and puts them on a canvas.

The results look like some weird surrealist art projects, but we're pretty happy with them

Let's look at some.

SOILIE's imagination of "car"

In this picture, we can see that SOILIE put the pixels for a "car" in the center. It knows that pictures of cars tend to also feature a "sky," so it put a sky somewhere above the car. Beneath the car is a "road."

All of these sub-images are from other pictures, which is why they don't look like they belong together.

Another interesting thing is that "wheel" also tends to co-occur with "car," so SOILIE put in a wheel. This is a mistake, though, because SOILIE should know that any image of a car is likely to also have an image of a wheel already in it. This is something we will fix for future versions .

SOILIE imagination of "eye"

This next image is of SOILIE imagining an "eye." Above is hair, to the left is an ear, and below is a mouth.

Although it looks strange, you can see the sense SOILIE is making of it!

Personally, I find these images rather charming. We have a long way to go to make them realistic, and lots of problems to solve, but for now I'm happy with our start. We are also talking to an art gallery about featuring some prints and a demo of the software for an exhibit.

Is it real imagination? Before your answer, answer these questions for yourself:

1. Does a calculator really calculate?
2. Does an airplane really fly?
3. Does a boat swim? Why not?

If you are interested in the paper that describes this version of "SOILIE," you can read it here:

• Breault, V., Ouellet, S., Somers, S. & Davies, J. (2013). SOILIE: A computational model of 2D imagination. In R. West and T. Stewart (Eds.)Proceedings of the 12th International Conference on Cognitive Modeling (ICCM2013), Ottawa: Carleton University (unnumbered six page online proceedings paper).

Here are some other links for further information:

• My TEDx talk on imagination
• My video interview for Nautilus Magazine.
• Imagining Nothing blog entry
• Imagination Mistakes blog entry
• What Imagination Is blog entry
• My website