Real or Virtual Matter - or How I Learned to Stop Worrying and Love the Matrix

 

Subliminal User Interfaces - Computer Applications as Agents of the Mind

Alvaro Cassinelli — University of Tokyo, Ishikawa Hashimoto Laboratory, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan — alvaro@k2.t.u-tokyo.ac.jp

Originally written circa 2001 as an unpublished research proposal. This version has been lightly formatted and edited for online publication.

---

Introduction

From graphical interfaces to perceptual user interfaces (PUI)

Perceptual Human Interfaces (PUIs) may well be the future of graphical user interfaces if they succeed in making communication with machines natural enough. However, I believe there is more to human-computer interaction than simply giving computers the capacity to understand—or reply by mimicking—a human face, a gesture, or even complex attitudes and other less codified forms of human communication. My guess is that, sooner or later, there will be a paradigm shift in how we understand human-computer interaction: in the future, it may not rely on an interface at all.

An elementary PUI will certainly minimize annoyances such as searching for the right application on a graphical desktop, along with the associated memory effort—a gesture will suffice. Concurrently, the idle time involved in launching applications—at the time, a real burden on a PC running several memory-hungry applications—will eventually disappear as processors continue to grow in power.

Presumably, however, even the most advanced PUI will do little more than a skilled human assistant at a computer: respond to our wishes as smoothly as possible while making attention-consuming details such as clicking and typing invisible. I believe human-machine interfaces can go further. One day, a computer filled with sensors may achieve a degree of transparency beyond that of even an expert human intermediary.

Beyond the Perceptual User Interface Paradigm: Subliminal User Interfaces (SUI): the bottleneck of conscious commands

The problem is that PUIs—or human assistants—do not address the real issue: the frustrating, almost entirely sequential nature of conscious commands. Is this behavior so unavoidable in human-computer communication that it imposes a theoretical limit on the smoothness of any future interface?

Certain characteristics of the human mind will always constrain the way a user gives conscious orders through an interface, whether human or artificial. Attention is limited, and conscious processes appear to unfold sequentially. Even if several processes run in parallel in our minds, ideas and wishes seem to arrive one after another.

According to Minsky [1], the mind resembles a “society” of agents in competition and interaction, continuously processing information even when some of that work is ultimately discarded. In this view, conscious behavior results from an a posteriori agreement among these agents. Crucially, this agreement is reached without our awareness of what is taking place—a consequence of the very definition of consciousness, as Dennett describes it [2].

Unconscious (subliminal) communication

A person must impose some order on what is in their mind to produce a more-or-less sequential stream of words or gestural commands. Yet conscious communication is by no means the only way human beings communicate and interact with the world.

There are many situations in which the chorus of several “mind agents” finds a direct route to the external world without being forced through the bottleneck of sequential language—and without “us” becoming aware of what we are doing or intending until after it happens. A skilled pianist never consciously launches a “move finger” application when playing a chord.¹

The sequential bottleneck can therefore sometimes be avoided: different parts of the brain address different aspects of the external world in parallel. We know that non-verbal communication exists, and some people develop this skill to the point where it resembles a sixth sense. Perhaps certain agents in their minds have developed special pathways to the external world, just as pathways exist between the “musical” agents in a pianist’s mind and the keyboard.

¹ The movement of the fingers and the will to move them appear simultaneous. Indeed, they may have to be simultaneous because they refer to the same physical phenomenon, with the “will” being a post-generated conscious acknowledgement of the action [2].

Computer Applications as Agents of the Mind

Following this reasoning, the ultimate futuristic human-computer interface would be a system so tightly coupled with our unconscious parallel processing that applications could operate—and even launch—before we consciously realized we needed them. They would attract our conscious attention only when they produced an interesting outcome from an otherwise discreet background process.

Even more fascinating—and certainly more troubling—is the possibility of a computer communicating with our mind’s agents without “us” being conscious that the communication is taking place.

The central aim of this research proposal is to demonstrate the possibility of establishing a subliminal, two-way communication path between a human and a computer: a Subliminal User Interface, or SUI.²

The ultimate SUI would dispense with any perceptible interface device. It would discreetly fuse the human self and the computer, functioning like a perfect prosthesis—one the user forgets they are using. Computer applications would become full members of an enlarged “society of mind.”

² Subliminal: (1) Existing in the mind below the threshold of consciousness—as feeling rather than as clear ideas. (2) Using sensory stimuli too weak to be consciously perceived, but capable of affecting unconscious mental processes. — Webster’s Dictionary

The Proposed Experiment

A synthetic “sixth sense” and a society of software agents

In practical terms, how might this fusion be achieved? Science-fictional, cyborg-like solutions involving direct wiring between chips and the brain were—and remain—beyond the scope of this proposal, although preliminary experiments had been conducted in our laboratory under the Sensory-Motor Fusion project. The ethical implications are worth exploring, not only for their conceptual subtlety but for the sake of our shared future.

I propose staying on safer ground—though no less ethically provocative—by designing two complementary systems:

1. A SUI based on common PUI sensors, giving a computer the ability to infer unformulated thoughts: a prototype synthetic “sixth sense.”
2. A synthetic society of software agents, composed of small autonomous routines, each responsible for a different desktop application.
   

At the core of each agent, a learning or predictive neural network—or another prediction algorithm—would anticipate user behavior from data gathered by the PUI and launch applications in the background. This would resemble predictive branching and execution caching in modern microprocessors, but at a much higher level.

Experiment One: A PUI-Based Subliminal User Interface

A two-way subliminal channel

Could a computer equipped with multimodal sensors develop something comparable to a human sixth sense, at least within a specific environment?

Evidence suggests that an effective subliminal communication channel can be established from computer to human through “subliminal cueing” [3]. The authors call this a “zero-attention interface,” and their work indicates that the degree of distraction produced by the cue can be adjusted.

The proposal here is to implement the opposite direction of that channel: from the human subconscious—understood as its own society of mind agents—toward a synthetic society of software agents within the computer.

Sensing non-explicit input

The raw material gathered by the PUI might include both perceptible and imperceptible aspects of a user’s behavior: eye movement, pauses, body movement and posture, breathing patterns, skin conductivity, EEG signals, and more.

In developing a SUI, we can draw on context-aware software and techniques based on “non-explicit input,” whose goal is to “reduce the amount of information the user must explicitly provide the application” [3]. The aim, however, is to go further: to infer what is on the user’s mind before a conscious intention has formed, preprocess any relevant data, and use subliminal cueing to return a “non-explicit output” to the user.

This two-way channel would be accessible to all the agents in the computer. Such agents might become highly effective at influencing perceived intention—a possibility with profound ethical implications.

Testing whether subliminal information is processed

A two-way subliminal channel could be tested through an experiment in which:

1. Subliminal information is presented to a participant.
2. The participant’s conscious attention remains occupied elsewhere.
3. Behavioral or physiological evidence demonstrates that the information was nevertheless processed unconsciously.

In principle, the loop could then be closed: the computer would process the evidence and return a different subliminal cue to the participant, continuing the cycle.

A concrete visual-cueing experiment

The experiment could build on the setup described in [3]:

“…subliminal cueing using short-duration, visually masked video fields on a head-mounted display can be effective in improving performance on a name-recognition visual-memory task.”

My proposed modification would eliminate the learning phase. The display would be used only to prompt certain names subliminally. Then, without asking an explicit question, the participant would be shown a screen filled with names while performing a concurrent, attention-intensive task.

At the same time, eye movement would be recorded. The saccade pattern could be analyzed to determine whether the participant spent an unusually long time looking at the subliminally prompted name. Here, “processing” is treated as a simple pattern-recognition task, but other experiments could involve more complex associations.

One alternative would involve specially trained participants such as skilled musicians. A few tones in particular scales could first be prompted subliminally. Then, while the participant performs an attention-intensive task, correct and incorrect musical transcriptions could be shown subliminally and subtle indicators of stress measured.

Experiment Two: A Synthetic Society of Desktop Agents

Applications that anticipate intention

An experimental synthetic society of mind agents could consist of small software components, each controlling the state—launching, closing, and so on—of a different desktop application such as a word processor or email client.

These agents would respond to conscious commands issued through a conventional graphical interface or an experimental PUI, but they would also attempt to interpret and respond to the user’s unconscious behavior.

A synthetic agent would infer the user’s likely intention and discreetly launch applications with a high probability of subsequent use. To do this, each agent would need a basic capacity for learning, prediction, and decision-making—an elementary AI core.

Discretion, interruption, and inhibition

The agents would maintain a continuous communication channel with the user and convey information either noticeably or subliminally through the available SUI features. They should remain as discreet as possible: unless necessary, their activity should not reach conscious awareness. There should be no annoying prompts or automatically generated proposal lists.

This creates a difficult design problem. At some point, information may need to become consciously accessible—but when? It may be impossible to identify a single “correct” level of interruption. Some people may appreciate occasional interruptions from sufficiently intelligent collaborators, while others may prefer to work without acknowledging any interaction.

The system should therefore include an inhibitory control that the user can activate whenever an agent intervenes inappropriately or too often. The agent could learn from this explicit cue and adjust its degree of discretion.

A simple predictive scenario

Many straightforward demonstrations could be built from the semantics of the words being written, typing speed, and pauses that may indicate hesitation.

For example, if someone using a word processor types the word “equation” and then makes an imperceptibly longer-than-usual pause, MATLAB could launch discreetly in the background—just in case. Higher-level context awareness would gradually increase the system’s predictive capacity and autonomy.

And who knows: perhaps one day this “awareness” could grow large enough that the “narrative center of gravity” [2] would shift away from the human being and settle somewhere between the “user” and the computer.

References

1. Minsky, M. The Society of Mind. First published 1985.
2. Dennett, D. C. Consciousness Explained. First published 1991.
3. DeVaul, R. W., and Pentland, A. S. “Toward the Zero-Attention Interface: Subliminal Cueing and Proactive Information Delivery.”