Red

Time to time, the subject of subjective experience comes up: Why is red "red"? The question is universal—Most sighted people see at least one color in addition to variations in brightness. Outside of sight, the question of what determines the content of "qualia", the atoms of subjective experience, is popular. "What is it like to be a bat?", "Neural plasticity and consciousness". 

Recently, I ask a friend: Why do colors seem as they do and not some other way? How can we know that two people experience them similarly? I do not remember the exact words, but the content of their answer was thus: 

The statistics of color are not symmetric. Blue and green overlap in a specific way, as do green and red. We see correlations in color data that are inherent to the absorption spectra of the cones in our retina, the properties of light, and the chemical statistics in our environment. 

You think that color is mysterious, because it seems you might exchange red for blue, blue for green, and green for red, and the world should stay the same. But this is not the case. This would lead to fundamental differences (and presumably evolutionary disadvantages) in what the brain can know about colors. 

At first, this answer satisfied me: Color qualia were not some ineffable mystery of the universe, but something inherited by the content and processing of information in our nervous system. Similar content and computations should create comparable subjective experiences. The (closed-loop) processing of information is equivalent to its experience. 

This seemed in line with the loose academic consensus, with some philosophers adding that the effect of our actions on our senses is also vital for determining subjective content (e.g. Hurley and Noë 2003). But, these ideas are not yet quite precise, especially in the context of color vision. What is so special (and captivating) about color? Do we understand the computations underlying its perception? 

Very few people frame the problem of qualia as "Why is left "left"and right "right""? Something sets color apart. The symmetry of left and right, up and down, seem obvious. Indeed, some people can hardly tell left from right. And, if you were to start wearing prismatic glasses that flipped the world upside down, the brain adapts. In time this adaptation would restore a subjective experience demonstrably similar to the right-side-up world of the past. In these dimensions it seems that subjective experience really is symmetric. 

Left and right are distinguished because you, as an embodied agent, have a left hand and left eye, a right hand and a right eye. Their symmetry is broken only by the multitude of asymmetric objects in our environment, not to mention our numerous cultural artifacts where left and right must be kept separate (reading text, driving on the road). Brain lateralization (are you left or right handed) also breaks this symmetry. 

It seems that many qualia are explained by their informational content (and how this is affected by our actions). To me the senses of sound, grayscale vision, somatosensation, proprioception, are all "colorless". It feels intuitive and satisfying that their subjective content should arise from their relationships amongst themselves and with my own body. I cannot say the same for color. 

But, there are other senses that defy "explaining away". Smell, taste, pain come with a preexisting opinion about how they should feel that lies outside their raw information geometry. The nervous system contains hard-wired circuits for processing such sensations. 

 There are nuclei that trigger salt cravings, dedicated pathways for pain, and (at least in rodents) innate tagging of certain odors as terrifying or sexy. Evolution provides hard-wired opinions about what some sensations mean, and what actions are appropriate upon their experience. This is a reasonably satisfying explanation of more visceral sensations, although much remains unaccounted for. 

But, back to color. It seems that something "breaks the symmetry" of color perception so that, yes, red really is "red", and cannot be anything else. Yet, color is not so visceral that it might be explained by innate connections. What possible explanations are there? 

1. Are color statistics asymmetric in a stable way across time (and across individuals)? 
2. Are color channels processed in a way that creates innate quality (like taste and smell)? 
3. Can biases in how the early visual system encodes and processes color explain its subjective quality? 

Are color statistics asymmetric in a stable way across time (and across individuals)?

While we cannot permute red, green, and blue color channels, color space does have an inherent geometry that might permit other transformations. 

If it were simply the case that red, green, and blue have a particular correlation structure, then your color quality would return to baseline after a period of immersion in augmented reality that provides altered color data with similar correlations. Red, green, and blue may not be "axes of symmetry", but there would be other (probably nonlinear and non-uniform) transformations of color space that *are* symmetric and *do* preserve qualia under these symmetries. 

To some extent this is true. If you look at Bridget Riley's painting "Gather", you will see how important context and contrast are for color perception. Red is not so much "red", but rather a very specific lack of green-ness and blue-ness, in the visual vicinity. Different (R,G,B) tuples can be perceived as the same subjective color, depending on context. But, this is also true for other senses: adjacent (colorless) shapes can bias perception of size and length. All this really says is that the relevant informational dimensions of qualia are relative. 

Perhaps the set of symmetries that matter for determining whether information geometry alone determine qualia are curvature-preserving isomorphisms within a given sensory space? It is not enough to retain the 3-D nature of color vision. It is enough, however, to preserve notions of perceptual distance (similarity vs contrast) across the space? 

The circuitry of the retina and brain are specialized to filter and compress visual data. Thus, transformations must preserve color similarity not at the level of rods and cones, but in the high- level information available to our conscious brains after post- processing through the retina and V1. 

Identifying such a transformation is beyond our current abilities. We do not understand early visual color encoding that well. And, I do not think this would be sufficient. I've mentioned that the changes in sensory input due to our own actions matter for qualia. More generally, the way that sensory input changes due to other known factors matters. Colors must also change in a specific way with illumination levels, and in how they change as an apple ripens over time. 

It seems that even if all of these conditions were met, it would still not be possible to find a symmetry of color vision that preserves subjective qualia. 

Are color channels processed in a way that creates innate quality (like taste and smell)? 

It seems that different colors have innate behavioral "valences". Primates that can see red can use it to find ripe fruit or to communicate social cues. Blushes are red. And, indeed, different retinal ganglion cells project to different targets. Most visual data is eventually sent to visual cortex, but more primitive areas are also targeted. 

It seems not-unreasonable to claim that the subjective asymmetry of color, then, arises in ways similar to taste or smell. Colors mean something, and have some opinions about what behaviors they should trigger if you want to survive and reproduce. 

Somehow, this explanation remains unsatisfying to me. Color vision is part of image-processing and image-formation. If innate valance alone determined the subjective experience of color, we would need to explain how it couples to visual perception. We do not simply get a vague feeling of "that apple looks nice to eat", but a detailed, pixel-for-pixel sense of redness. 

My hunch is that innate valence does bias qualia, in the sense that it tags some color channels with specific and distinct predictions. But I am not sure this explains everything. 

There are other visual inputs which come with an innate valence. Babies detect faces innately. Faces are intensely ethologically relevant. Yet, it is the composite object of the face that is tagged with innate valance. We do not appreciate "red" only when it takes a form or context that is appetitive. 

Can biases in how the early visual system encodes and processes color explain its subjective quality? 

This seems likely. From the retina, colors are encoded differently. Contrasts in blue and yellow have coarse spatial resolution, but good temporal resolution. Contrasts in red and green have good spatial resolution, but less temporal precision. 

There is an optical illusion wherein a rotating disk with black and white markings gives rise to illusory colors. The specific color depends on the spatial pattern and rate of movement. (Look up "Flechner colors"). This shows that different colors couple differently to our perception of motion and contrast. Illusory colors experienced when viewing a white, immersive strobe light (flicker ganzfeld) further highlight this. 

This seems sufficient to ensure that different colors feel "different". The asymmetry in processing is so severe that no external transformation of color space will lead to inputs with comparable correlation structure. 

The sub-units of color processing (in cortical columns) are tiled repeatedly across visual cortex. This is also consistent with the notion that the it is the "pixels" of vision that are tagged with subjective aspect, rather than something more abstract. 

Assembling it all. 

It seems sensible that some aspects of our experience of color should be innate. A ripe apple and a blushing face are important. But, innate valence (comparable to smell and taste), would not necessarily imply "pixel-wise" subjective specificity to color. 

Furthermore, color processing in early vision is intensely asymmetric, likely in a way that makes it impossible to transform color space while preserving information geometry after early visual encoding. This provides an inherent, "pixel-wise" asymmetry to color that could lend it specific qualitative content. 

Is that all, then? The innate structure of early vision provides specific, oriented, asymmetric "labelled channels". These enforce the similarity space of color perception. It is a world where "up" is up and will never be anything else. Simultaneously, other brain structures provide innate valence that tags these channels in relation to instinctive behaviors. 

Does neural plasticity associate these two distinct sources of "innateness"? Would this further endow the "pixel-wise" color perception with vague, hard to articulate sense of innateness? I think... probably? 

Does the weirdness of color comes from this fusion? Color has innate valence, but also "pixel-like" aspects of vision. There is no other sense that operates in quite this way. Somatosensation, which contains many different touch receptors, may be the closest analogue. And, indeed, feels the closest (qualitatively) to the content of color vision. 

… To me, at least.