The Man Who Mistook His Y for a Hat

One of the reasons people love reading Oliver Sacks is that he manages to convey the magnificent powers of our brains, powers we spend most of our lives taking for granted. Sacks accomplishes this not by enumerating the brain's specs (like its processing speed or number of neurons), but by showing us what happens when our powers are switched off, usually by disease. His stories of switched-off brain powers make us say, "Gosh, I didn't even know I had brain regions devoted to doing that!"

Recently Sacks wrote a piece in The New Yorker on the neuroscience of reading, specifically centered on a writer who lost his ability to read. This story fits Sacks' modus operandi well: by gawking at the poor chap who lost his reading power, we say, "Gosh, I didn't even know I had brain regions devoted to reading!"

Except that this would be a mistaken conclusion in this case because, as Sacks reminds us, writing was invented only around five thousand years ago, far too recently to have affected our brains. In fact, most of us don't have to look back more than several generations to find ancestors who couldn't read.

How, then, do we have reading areas for a brain that didn't evolve to read?

Stanislas Dehaene, neuroscientist and author of Reading in the Brain, argues that our brains have undergone "neuronal recycling," where writing has shaped itself over time to be easy on our visual systems.

And what's the trick to getting writing to fit into our illiterate visual system?

In my own research I have suggested how it happened: culture shaped letters to look "like nature."

Oliver Sacks describes the research this way:

"Such a redeployment of neurons is facilitated by the fact that all (natural) writing systems seem to share certain topological features with the environment, features that our brains have evolved to decode. Mark Changizi and his colleagues at Caltech examined more than a hundred ancient and modern writing systems, including alphabetic systems and Chinese ideograms, from a computational point of view. They have shown that all of them, while geometrically very different, share certain topological similarities. (This visual signature is not evident in artificial writing systems, such as shorthand, which are designed to emphasize speed more than visual recognition.) Changizi et al. have found similar topological invariants in a range of natural settings, and this is has led them to hypothesize that the shapes of letters "have been selected to resemble the conglomerations of contours found in natural scenes, thereby tapping into our already-existing object recognition mechanisms."

It is worth delving much more deeply here into our ability to read, where I will try to better communicate how powerful reading is, and how our brain has come to cope with a capability it never evolved to accommodate. I will take you through several topics, eventually moving more closely to what I call "nature-harnessing", the method culture used to turn our non-reading visual system into reading machines.

Super Reading Medium

Communicating with the dead is a standard job requirement for a psychic such as the infamous medium John Edward of the television show Crossing Over who claims to be able to listen to what the deceased family members of his studio audience have to say. Hearing the thoughts of the dead would appear to be a power we certainly do not possess. Surely this power must remain firmly in the realm of fiction (Edward included). However, a little thought reveals that we in fact do this all the time. ...by simply reading. With the invention of writing, the ability for the dead to speak to the living suddenly became real. (Progress in communicating in the other direction has been slower going.) For all you know, I'm dead, and you're exercising your spirit-reading skills right now. Good for you!

Before the advent of writing, in order to have our thoughts live on after we had gone we had to invent a great story or catchy tune and hope that they're singing it by the fire for generations. Only a few would be lucky enough to have a song or story with such legs (e.g., Homer's Illiad), and at any rate, if our ancestors were anything like us, their greatest hits probably tended to include "ooh-la-la" and "my baby left me" much more often than "here's my unsolicited advice" and "beware of milk-colored berries." Getting your children to be your audio tape in this fashion is probably futile (and aren't they just as likely to purposely say the opposite?), but at least it relies on spoken words, something readily understandable by future generations. The problem is getting your voice to last. Voices are just too light and insubstantial, like a quarterback finding an open receiver and throwing to him a marshmellow. Marshmellows are great to hold, but impossible to throw far. I suppose if you were to speak loudly enough during a heavy volcanic ash storm, ripples on the rapidly accumulating layers of ash might record your spoken words, one day to be recovered by clever archeologist decoders. However, much of what you're likely to say in such circumstances will be unrepeatable in polite company.

What prehistoric people did successfully leave behind for us to read tended to be solid and sturdy, like Stonehenge or the moai statues of Easter Island. These were quarterback passes that got to the receiver all right, except that now the quarterback is throwing something that is uncatchable, like porcupines or anvils. Massive monuments are great if your goal is to impress the neighboring tribes or to brag to posterity. But if your goal is to actually say something that can be understood, this tact is worse than writing abstruse poetry, and literally much heavier. The only thing we're sure of about such communications is that they had too much free time on their hands. Not the most informative spirit-reading.

The invention of writing changed spirit-reading forever. It also changed the world. Reading now pervades every aspect of our daily lives, so much so that one would be hardpressed to find a room in a modern house without words written somewhere inside. Lots of them. Many of us now read more sentences in a day than we listen to. And when we read we must process thousands of tiny shapes in a short period of time. A typical book may have more than 300,000 strokes, and many long novels will have well over one million. Not only are we highly competent readers, but our brains even appear to have regions devoted to recognizing words. Considering all this, a Martian just beginning to study us humans might be excused for concluding that we had evolved to read. But, of course, we haven't. Reading and writing is a recent human invention, going back only several thousand years, and much more recently for many parts of the world. We are reading using the eyes and brains of our illiterate ancestors. And this brings us to a deep mystery: Why are we so good at such an unnatural act? We read as if we were designed to read, but we have not been designed to read. How did we come to have this super power?

Reading as a super power? Isn't this, you might ask, a bit of an exaggeration? No, it really is super. To better appreciate it, when you next have the illiterate caveman neighbors over to the house--the ones who always bring the delicious cave-made bundt cake--wow them with how you can transmit information between you and your spouse without speaking to one another. ...by writing and reading. They'll certainly be impressed. It's not your use of symbols that will impress them, however, because they leave symbols for one another all the time, like a shrunken head in front of the cave to mean the other is at the witchdoctor's. And they have spoken language, after all, and realize that the sounds they utter are symbols. What will amaze them about your parlor trick is how freakishly efficient you are at it. How did your spouse read out the words from the page so fast? Although they appreciate that there's nothing spooky in principle about leaving marks on a page that mean something, and someone reading them later, they conclude that you are just way too good at it, and that, despite your protestations, you must be magical shamans of some kind. They also don't fail to notice that your special power would work even if the writer was far away. Or long dead. Their hairs stand on end, the conversation becomes forced, they skip dessert, and you notice that their cavekids don't come around to throw spears at your kids any more. As the saying goes, one generation's maelstrom is a later generation's hot tub. We're just too experienced with writing to appreciate how super it is, but not so for your cave neighbors.

We have the super power of reading not because we evolved to read--and certainly not because we're magical in any way--but because culture evolved writing to be good for the eye. Just as Captain Kirk's technology was sometimes interpreted as magic by some of the galaxy locals, your neighbors are falsely giving you credit for the power when the real credit should go to the technology. The technology of writing. And not simply some new untested technology, but one that has been honed over many centuries, even millenia, by cultural evolution. Writing systems and visual signs tended to change over time, the better variants surviving, the worse ones being given up. The resultant technology we have today allows meanings to flow almost effortlessly off the page and straight into our minds. Instead of seeing a morass of squiggles we see the thoughts of the writer, almost as if he or she is whispering directly into our ears.

The special trick behind the technology is that human visual signs have evolved to look like nature. Why? Because that is what we have evolved over millions of years to be good at seeing. We are amazingly good at reading the words of the dead (and, of course, the living) not because we evolved to be spirit-readers. Rather, it is because we evolved for millions of years to be good at quickly visually processing nature, and culture has evolved to tap into this ability by making letters look like nature. Our power to quickly process thousands of tiny shapes on paper is our greatest power of all, changing our lives more than our other powers. Literacy is power, and it's all because our eyes evolved to see well the natural shapes around us and we, in turn, put those shapes to paper.

Good Listening

"How did your date go?" I asked.
"Great. Wow. What a guy!" she replied. "He listened so attentively the entire dinner, just nodding and never interrupting, and-"
"Never interrupting?" I interjected.
"That's right. So supportive and interested. And so in tune with me, always getting me without even needing to ask me questions, and his-"
"He asked you no questions?" I interrupted, both eyebrows now raised.
"Yes! That's how close the emotional connection was!"
It struck me that any emotional connection she felt was a misreading of his eyes glazing over, because her date was clearly not listening. At least not to her! I didn't mention to her that the big game was last night during her dinner, and I wondered whether her date might have been wearing a tiny ear phone.

Good listeners don't just sit back and listen. Instead, they are dynamically engaged in the conversation. I'm a good listener in the fictional conversation above. I'm interrupting, but in ways that show I'm hearing what she's saying. I am also able to get greater details of the story where I might need them. In this case about her date's conversational style. That's what good listeners do. They rewind the story if needed, or forward it to parts they haven't heard, or ask for greater detail about parts. And good communicators tend to be those who are able to be interacted with while talking. If you bulldoze past all attempts by your listener to interrupt you, your listener will probably soon not be listening. Perhaps he's heard that part before and is tuning out now. Or perhaps he was confused by something you said fifteen minutes earlier, and gave up trying to make sense of what you're saying. Good listeners require good communicators. My fictional friend above appears to be a good communicator because she dynamically reacts to my queries midstream. The problem lies in her date, not her, and I politely suggest he may not be the right one for her.

Even though we (arguably) evolved to speak and listen, but didn't evolve to read, there is a sense in which writing has allowed us to be much better listeners than speech ever did. That's because readers can easily interact with the writer, no matter how non-present the writer may be. Readers can pause the communication, skim ahead, rewind back to something not understood, and delve deeper into certain parts. We listeners can, when reading, manipulate the speaker's stream of communication far beyond what the speaker would let us get away with in conversation--"Sorry, can you repeat the part that started with, ‘The first day of my trip around the world began without incident'?"--making us super-listeners, and making the writer a super-communicator.

We don't always prefer reading to listening. For example, we listen to books on tape, lectures, and talk radio, and in each case the speakers are difficult to interrupt. However, even these cases help illustrate our preference for reading. Although people do sometimes listen to books on tape, they tend to be used when reading is not possible, like when driving. When one's eyes are free, people prefer to read stories rather than hear them on tape, and the market for books on tape is miniscule compared to that for hard copy books. We humans have brains that may have evolved to comprehend speech, and yet we prefer to listen with our eyes, despite our eyes not having been designed for this! Television and movies have an audio stream that is not easily interruptable, and we do like that, but now the visual modality helps keep our attention. And although students have been listening for centuries to the speech of their professors, until recently with relatively little visual component, anyone who has sat through years of these lectures knows how often one's mind wanders. ...how often one is not actually listening. Talk radio has some popularity, and tends to be more engaging than traditional lectures, but notice that such shows go to great lengths to be conversational, typically having conversations with callers, and often having a pair of hosts (or a sidekick), to elicit the helpful interruptions found in good listening.

Canned speech, then, tends to be difficult to listen to, whereas genuine, dynamic, interactive conversation enables good listening. There is, however, one kind of audio stream our brains can't get enough of, where interruption is not needed for good listening, and where we're quite happy not seeing anything. Music. Audio tapes that give up on communication and aim only for aesthetics are suddenly easy listening. The rarity of books on tape, and the difficulty with listening to canned speech more generally, is not due to some intrinsic difficulty with hearing per se. The problem is that speech requires comprehension--music doesn't--and comprehension can occur most easily when the listener is able to grab the conversation by the scruff of the neck and manipulate it as needed so he can fit it into his head. Good conversation with the speaker can go a long way toward this, but even better listening can be achieved by reading because then you can literally pick up the communication with your hands and interact with it to your heart's content.

Working Hands

Having a conversation is not like passing notes in class. Although in each case two people are communicating back and forth in turn, when passing notes you tend to do little reading and lots of wiggling--either wiggling your hand in the act of writing a note, or twiddling your thumbs while waiting for your friend to write his. Note-writing takes time, so much time that passing notes back and forth is dominated by the writing, interspersed with short bouts of reading. All the work's in the writing, not the reading. Conversation--i.e., people speaking to one another--is totally unlike this. Speaking flows out of us effortlessly, and comes out nearly at the speed of our internal thoughts. That is, whereas writing is much more difficult than reading, speaking is not much more difficult than listening.

The reason for this has to do with how many people we're communicating with. When we speak there are typically only a small number of people listening, and most often there's just one person listening (and often less than that when I speak in my household). For this reason spoken language has evolved to be a compromise between the mouth and ear: somewhat easy for the speaker to utter, and somewhat easy for the listener to hear. In contrast, a single writer can have arbitrarily many readers, or "visual listeners." If cultural evolution has shaped writing to minimize the overall efforts of the community, then it is the readers' efforts that will drive the evolution of writing because there are so many of them. That's why as amazing, as writing may be, it is a gift to the eye more than a gift to the hand. For example, a book may take six months to write, but it may take only six hours to read. That's a good solution because there are usually many readers of any given book.

Is writing really for the eye, at the expense of the hands? One of the strongest arguments that this is the case is that writing has been culturally selected to look like nature, something we'll see later. That's a good thing for the eye, not the hand, because the eye has evolved to see nature--the hand has not evolved to draw it. Not only does writing tend to look like nature, but I have found that even visual symbols like trademark logos--which are typically never written by hand, and are selected to be easy on the eyes--have the fundamental structural shapes found in nature. And note that for some decades now much of human writing has not been done by hand, but instead has been done by keyboard. If the structures of letters were for the hand, we might expect that now that our hands tend to be out of the picture, the structures of letters might change somewhat. However, although there are now hundreds of varying fonts available on computers, the fundamental structural shapes have stayed the same. Shorthands, however, have been explicitly selected for the hand at the expense of the eye, and shorthands look radically different from normal writing, and I have shown that they have shapes that are not like nature. I have also taken data from children's scribbles and shown how the fundamental structures occurring in scribbles are unlike that found in writing and in nature. Finally, one can estimate how easy a letter is to write by the number of distinct hand sweeps required to produce it (this counts sweeps resulting in strokes on the page, and also sweeps of the hand between touchings of the paper), and such estimates of "motor ease" do not help to explain the kinds of shapes we find in writing.

Could culture really have given no thought whatsoever to the tribulations of the hand? Although selection would have favored the eye, it clearly would have done the eye no good to have writing be so difficult that no hands were willing to make the effort. Surely the hand must have been thrown a bone, and it probably was. The strokes in the letters you're reading, and in line drawings more generally, are quite a bit like contours in being thin, but there is an important difference. Real world contours occur when one surface stops and another starts, like the edge between two walls, or the edge of your table. Usually there is no line or stroke at all (although sometimes there can be), but a sudden change in the nature of the color or texture from one region to the next. The visual system would therefore probably prefer contours, not strokes. But strokes are still fairly easy to see by the visual system, and are much easier for the hand to produce. After all, to draw true contours rather than strokes would require drawing one color or texture on one side of the border of the intended contour and another color or texture on the other side. I just tried to use my pen to create a vertical contour by coloring lightly to the right and more darkly to the left, but after a dozen tries I've given up. I won't even bother trying to do this for an "S"! It's just too hard, which is why when we try to draw realistic scenes we often start with lines as contours, and only later add color and texture between the lines. And that's probably why writing tends to use strokes. That we use strokes and not true contours is for the benefit of the hand, but the shapes of our symbols are for the eye.

Harness the Wild Eye

You'd be surprised to see a rhinoceros with a rider on its back. In fact, a rider would seem outlandish on most large animals, whether giraffe, bison, wildebeast, bear, lion, or gorilla. But a rider on a horse seems natural. Unless you grew up on a farm and regularly saw horses in the meadows, a large fraction of your experiences with horses were likely from books, television and film where the horses typically had riders. Because of your "city-folk" experiences with horses, a horse without a rider can seem downright unnatural! In fact, if aliens were observing the relationship between humans and horses back when horses were our main mode of transportation, they may have falsely concluded that horses were designed to carry humans on their backs. But, of course, horses aren't born with bridles and saddles attached, and they didn't evolve to be ridden. They evolved over tens of millions of years of evolution in savannas and prairies, and it is only recently that one of the primates had the crazy idea to get on one. How is it that horses could have become so well adapted as "automobiles" in a human world?

Horses didn't simply get pulled out of nature and plugged into society. Instead, culture had to evolve to wrap around horses, making the fit a good one. Horses had to be sired, raised, fed, housed, steered, and scooped up after. Countless artifacts were invented to deal with the new tasks required to accommodate the entry of horses into society, and entire markets emerged for selling them. Diverse riding techniques were developed and taught, each having certain advantages for controlling these beasts. The shapes of our homes and cities themselves had to change. Water troughs in front of every saloon, stables stationed through towns and cities, streets wide enough for carriages, parking spaces for horses, and so on. That horses appear designed for riders is an illusion due to culture having designed itself so well to fit horses.

Just as horses didn't evolve to be ridden, eyes didn't evolve for the written. Your eyes reading these words are wild eyes, the same eyes and visual systems of our ancient preliterate ancestors. And yet, despite being born without a "bridle," your visual system is now saddled with reading. We have, then, the same mystery as we find in horses: how do our ancient visual systems fit so well in modern reading-intensive society?

Eyes may seem like a natural choice for pulling information stored on material, and indeed vision probably has inherent superiorities over touch or taste, just as horses are inherently better rides than rhinos. But just as horses don't fit efficiently into culture without culture evolving to fit horses, the visual system couldn't be harnessed for reading until culture evolved writing to fit the requirements of the visual system. We didn't evolve to read, but culture has gone out of its way to create the illusion that we did. We turn next to the question of what exactly cultural evolution has done to help our visual systems read so well.

From the Hands of Babes

You might presume that a two and a half year old girl couldn't have much to say. If I were struck on the head and reduced to infant-level intelligence for two and a half years, I'm fairly sure I wouldn't thereby have a flood of stories to tell you about. None, at least, that are not considerably degrading. But there she is, as you well know if you've seen these creatures, talking up a storm. A little about the few things that have happened to her, but mostly about things that never have, and never will: princesses, dragons, Spongebob, Stegosauruses. She's five now and there's been no let-up. She's talking to me as I write this!

I just gave her a piece of paper and crayons, and although she's just begun trying her hand at writing--"cat," "dug," "saac" (snake), "flar" (flower)--she's been putting her thoughts and words to the page for a long time now. By drawing. Children are instructive for the invention of writing because they invent their own writing through pictures. Through the work of Rhoda Kellogg in the mid-twentieth century we know that children world-wide draw very similar shapes, and follow a similar developmental schedule. Since they are not designed to draw, these similarities are, in a sense, parallel discoveries about how to ably communicate on paper; on how to write. Sir Herbert E. Read, an early 20th century professor of literature and arts, encountered Rhoda Kellogg's work late in his life, and wrote the following:

It has been shown by several investigators, but most effectively by Mrs. Rhoda Kellogg of San Francisco, that the expressive gestures of the infant, from the moment that they can be recorded by a crayon or pencil, evolve from certain basic scribbles towards consistent symbols. Over several years of development such basic patterns gradually become the conscious representation of objects perceived: the substitutive sign becomes a visual image. ... According to this hypothesis every child, in its discovery of a mode of symbolization, follows the same graphic evolution. ... I merely want you to observe that it is universal and is found not only in the scribblings of children but everywhere the making of signs has had a symbolizing purpose-which is from the Neolithic age onwards. [from Herbert Read, "Presidential Address to the Fourth General Assembly of the International Society for Educational Society for Eduation through Art." Montreal: Aug 19, 1963.]

Aren't children's drawings just that, drawings? It's certainly true that sometimes children are just trying to depict what they see. Those are "mere" drawings. But often their drawings are primarily aimed to say something-to tell a story. When my daughter brings me her latest drawing, she usually doesn't brag about how real it looks (nor does she tell me about its composition and balance). Sure, sometimes she asks me to count how many legs her spider has, but usually I get a story. A long story. For example, here is a Cliffs Notes version of the story behind her drawing in the nearby figure: A house with arms and eyes; the windows have faces; it is a magic house; there is a girl holding a plate of cream puffs; two people are playing with toys at the table but a tomato exploded all over the toy; there are butterflies in the house. Her drawing is intended to communicate a story, and that sounds an awful lot like writing.

But if she's truly writing, then she'd have to be using symbols. Is it really plausible that small children are putting symbols on the page before they learn formal writing, as Rhoda Kellogg and Herbert Read believe? I think so, for consider that most of their drawings have only the barest resemblance to the objects they are intended to denote. Look again at nearly any of the objects in my daughter's drawing. An attempt at realism? Hardly. We find similar kinds of symbols when even adults draw cartoons-adults who could draw realistically if they wished. These cartoon symbols, like those in the first row of Figure 2a, are ridiculously poor renderings of objects. You have surely seen similar visual signs out and about in culture. Although you'll probably have no trouble knowing what animals the drawings are intended to symbolize, your dog would have no idea what those (or my daughter's) drawings are supposed to be. They get their meaning by convention more than by resemblance. We're so used to these conventions that we have the illusion that they look like the animals they refer to, but other cultures often have somewhat different conventions for their animals. For example, I find it difficult to tell what kind of animal I'm looking at in many of today's Japanese cartoons for kids, some of them shown in the second row of the upcoming figure below.

The same is true for sound. We in the United States say "ribbit" to refer to the call made by a frog, and after growing up with that as the symbol for frog calls it can be hard to appreciate that frogs don't sound at all like that. In fact, people from different cultures use different sound symbols to refer to frog calls, and each person is initially convinced that their sound resembles frogs. Algerians say "gar gar," Chinese say "guo guo," the English say "croak," the French say "coa-coa," Koreans say "gae-gool-gae-gool," Argentinians say "berp," Turks say "vrak vrak," and so on. Just as in children's drawings, the sound "ribbit" is a symbol for the call of the frog, not a real attempt to resemble or mimic it.

Children's drawings communicate stories with symbols. That sure sounds like writing to me. Or at least the barest beginnings. If these little whippersnappers are so smart that they can spontaneously invent writing largely on their own, perhaps it couldn't hurt to look into the kinds of symbols they choose for their writing. And the answer is so obvious that it may be difficult to notice: children draw object-like symbols for the objects in their writing. Their drawings may not look much like the objects they stand for, but they look like objects, not like fractal patterns, not like footprints, not like scribbles, not like textures, and so on. The same is true for the cartoons drawn by adults, as in (a) of the nearby figure. And we find the same so-obvious-it-is-hard-to-notice phenomenon for animal calls: although there are lots of different sounds used for frog calls, they are all animal-call-like. All those frog calls sound like some possible kind of animal. What might this phenomenon mean for writing?

Word and Object

Is the strategy of object-like drawings for objects mere child's play? Apparently not, because it's not just in kids' drawings and cartoons that you find this, but among human visual signs generally. Most non-linguistic visual signs throughout history have been object-like, such as those found in pottery, body art, religion, politics, folklore, medicine, music, architecture, trademarks and traffic (see (b) in the nearby figure for a small variety). And computer desktop icons are not only object-like in appearance, but can even be moved around like objects. Much of formal writing itself has historically been of this objects-for-words form, such as Egyptian hieroglyphs, Sumerian cuneiform, Chinese, and Mesoamerican writing. Modern Chinese is still like this, used by nearly half the world. In these writing systems we find drawings with the complexity of simple objects and used as symbols to refer to objects, and also to refer to adjectives, adverbs, verbs and so on. (See (c) in the nearby figure for several examples.) Object-like symbols for objects-that trick's not just for kids.

Is there something beneficial about drawing objects for the words in writing? I suspect so, and I suspect that it is the same reason that animal-call symbols tend to be animal-call-like: we probably possess innate circuitry that responds specifically to animal-call-like sounds, and so our brain is better able to efficiently process a spoken word that means an animal call if the word itself sounds animal-call-like. Similarly, we possess a visual system designed to recognize objects and efficiently react to the information. If a word's meaning is that of an object (even an abstract object), then our visual system will be better able to process and react to the written symbol for that object if the written symbol is itself object-like. The next figure, in (b), shows a fictional case of writing with object-like symbols for words (and single strokes are shown for "function words" like ‘the' and ‘in'). To begin to grasp why this strategy might be good, consider two alternative strategies besides the objects-for-words one.

First, rather than drawing objects for words, we could be lazy and just draw a single contour for each spoken word. Writing "The rain in Spain stays mainly in the plain" would then look something like that shown in (a) of the figure below. Shorthand is somewhat akin to the lazy approach, with some words having single stroke notations. Shorthand is great for writers with fast-talking bosses, but is notoriously hard to read and has not caught on for writing. Kids also don't think it's a good idea-there's not even a single lone contour in my daughter's drawing. One reason it's not a good idea is that there are just not enough distinguishable stroke types for all the words we speak. Coming up with even 100 easily distinguishable stroke types would be tricky, and that would still be far below the tens of thousands that would be needed for writing.

There is also a more fundamental difficulty, and it has to do with the fact that the part of your brain doing the visual computations is arrayed in a hierarchy. The earlier stages of the hierarchy deal with simpler parts like contours, higher areas deal with simple combinations of contours, and eventually at the highest regions of the hierarchy full objects are recognized and perceived. The problem with using single strokes to represent spoken words is that the visual system finishes processing the strokes far too early in the hierarchy. The visual system is not accustomed to word-like (e.g., object-like) interpretations to single strokes. Single strokes are typically not perceived at all, at least not in the sense that they make the list of things we see out there. For example, when you look at the figure below with a cube and a pyramid, you perceive a cube in front of a pyramid. That's what you consciously notice and carry out judgements upon. You don't see the dozen contours in quite the same sense. Nor do you see the many object corners and junctions (intersections of contours). You don't say, "Hey, look at all those contours and corners in the scene." Our brains evolved to perceive objects, not object-parts, because objects are the clumps of matter that stay connected over time and are crucial to parsing and making sense of the world. Our brains naturally look for objects and want to interpret stimuli out there as objects, so using a single stroke for a word (or using a junction for a word) is not something our brains are happy about. Instead, when seeing the stroke-word sentence in (a) in the "rain in spain" figure, the brain will desperately try to see objects in the jumble of strokes, and if it can find one, it will interpret that jumble of strokes in an object-like fashion. But if it did this, it would be interpreting a phrase or whole sentence as an object, something that is not helpful for understanding a sentence: the meaning of a sentence is "true" or "false," not any single word meaning. Using single strokes as words is, then, a bad idea because the brain is not designed to treat single contours as meaningful. Nor is it designed to treat object junctions as meaningful. That's why spoken words tend to be written with symbols having a complexity no smaller than visual objects.

How about, instead, letting spoken words be visually symbolized by whole scenes, i.e., via multiple objects rather than just a single one? The (c) part of the "The rain in Spain..." figure shows what it might look like with this "scene-ogram" strategy. Quite an eye full. These are akin to the drawings found in some furniture assembly manuals. The problems now are the opposite to those before. First, the natural meaning of scene-ogram images is more like that of a sentence, like "Take the nail that looks like this, and pound it into the wooden frame that looks like that." Secondly, the fact that there are objects as part of these complex symbols is itself a problem because now the brain wants to inappropriately make meanings out of these, and yet these objects are now just the building blocks of a written word, having no meaning at all.

In sum, the visual system possesses innate mechanisms for interpreting object-like visual stimuli as objects. Because spoken words are the smallest meaningful entities in spoken language, and often have meanings that are at the object level (either meaning objects, or properties of objects, or actions of objects), it is only natural to have visual representations of them that the visual system has been designed to interpret, and to interpret as objects. By drawing objects for spoken words--and not smaller-than-object visual structures like contours or junctions, and not larger-than-object visual structures like scenes--the visual system is able to be best harnessed for a task it never evolved to do.

Object-like symbols might, then, be a good idea for representing words, but are the object-like symbols we find in culture a result of cultural evolution having selected for this, or might it instead be that they are just a left-over due to the first symbols having been object-like? After all, the first symbols tended to be object-like pictograms, even more object-like than the symbols in non-linguistic and linguistic signs. Perhaps our symbols are still object-like merely because of inheritance, and not because culture has designed them to be easy on the eye. The problem with this argument is that writing tended to change quickly over time, especially as cultures split. If there were no cultural selection pressure to keep symbols looking object-like, then the symbol shapes would have randomly changed over the centuries, and the object-likeness would have tended to become obliterated. But that's not what we find. Culture has seen to it that our symbols retain their object-likeness, because that's what makes us such good readers. It is interesting, though, that even the first symbols were on the right track, before cultural evolution had time to do any shaping of its own. Although, given that even small children codgeon onto this, it's perhaps not too surprising that the first scribes appreciated the benefits of object-like drawings for words.

The Trouble with Speech Writers

The brain prefers to see objects as the symbols for words, and kids and much of the world have complied. Such writing is "logographic" (symbols for words), and doesn't give the reader information on how to speak it, which is itself a great benefit, for then even people who speak different languages can utilize the same writing system and be able to communicate via it. That is, logographic writing systems can serve as universal writing systems bringing together a variety of spoken languages into harmony and friendship, Tower-of-Babel style. Japanese speakers, for example, have no idea what a Chinese speaker is saying, but can fairly well understand written Chinese because Japanese speakers also use Chinese writing (which is of the objects-for-words kind).

Brotherhood and peace may be nice, but there er jus some thangs ya cayant do when writin' with objects. For one thing, you can't communicate how to say those words. ...including putting a person's accent down on the page. A Japanese person may be glad to be able to read Chinese content, but he will be totally unprepared to actually speak to anyone in China. The kind of writing you're reading at the moment is entirely different. Rather than symbols for spoken words, the basic symbols are letters saying how to speak the words. You're reading "speech-writing." Speech-writing allows us to put Tom Sawyer's accent on paper, and it allows non-speakers of our language to obtain a significant amount of knowledge about how to speak among us by reading at home. Such a learner would have an atrocious accent, of course, but would nevertheless have a great start. A second important advantage to speech-writing is that one can get away with many fewer symbols for writing. Rather than one object-like symbol for each of the tens of thousands of spoken words, one only needs a symbol for each of the dozens of speech sounds, or phonemes, we make. That's a thousand-fold reduction in the number of written symbols we have to learn.

I have no idea whether the merits of speech-writing outweigh the benefits of logographic (symbols-for-words) writing, but there have been hundreds of speech-writing systems over history, many in use today by about half the world's population. And when culture decided to go the speech-writing route rather than the logographic route, it created for itself a big dilemma. As we've discussed, the best way to harness the natural object-recognition powers of the visual system is to have spoken words look object-like on paper. But in speech-writing the symbols are for speech sounds, and written words will consist of multiple speech sound symbols. How can our written words look like objects if written words no longer have fundamental symbols associated with them? If symbols are for fundamental speech sounds, then the look of a written word will depend upon the letters in it. That is, the word's look will be due to the vagaries of how the word sounds when spoken. Had it been spoken differently, the written word would look different. If the look of a word depends on how speakers say the word, it would seem that all hope is lost in trying to make written words look object-like in speech-writing.

There is a way out of the dilemma, however, and although no individual may have conceived of the idea, culture nevertheless eventually evolved to utilize this solution. The solution is this: If written words must be built out of multiple symbols, then to make words look object-like, make the symbols look like object parts. That's what culture did. Culture dealt with the speech-writer dilemma by designing letters that look like the object parts found in nature, object junctions, in particular. That way written words will typically be object-like, so that again our visual system can be best harnessed for reading.

-----

This piece was adapted from Chapter 4 of The Vision Revolution.

----

Mark Changizi is the author of The Vision Revolution and a professor at Rensselaer Polytechnic Institute.