[PR] Multisensory Intelligence

Created in October, 2026

A Vision for Multisensory Intelligence: Sensing, Science, and Synergy [link]

The next decade of research vision for multisensory AI is to move beyond text, images, and audio toward AI that senses the world the way we do, through touch, smell, taste, physiological signals, and the physical and social environment. MIT Media Lab organizes this agenda around three themes: (1) sensing new modalities, (2) building a science of how modalities differ and interact, and (3) learning synergy, where combined modalities do more than the sum of their parts.

Goal: Multisensory intelligence envisions an end-to-end pipeline from sensor design, data collection and processing, AI modeling, and interactive deployments to improve human experiences in physical and social contexts. We believe in the immense impact of multisensory intelligence to enhance how humans and AI experience and interact with one another, leading to improved productivity, creativity, and wellbeing for the world.

Novel Sensory Modalities

Show the eight modality groups
  • Vision: Vision research is moving beyond 2D images and videos toward 3D and 4D understanding, and beyond ordinary pixels to wireless signals, thermal imaging, depth cameras, and SPAD cameras. Its impact now reaches from medical imaging to the depths of the ocean and outer space.
  • Language: Language has been the main driver of foundation models, through Transformers and LLMs, and it has expanded from typed text to spoken language. Research now also covers low-resource languages, sign language, and performative language.
  • Audio: Audio carries information beyond words, such as prosody, rhythm, and vocal cues that reveal emotion and intent. Together with language and vision, it supports tasks such as recognizing sentiment, emotion, humor, and sarcasm in social interactions.
  • Touch: Tactile sensors such as gloves, GelSight, and DIGIT capture touch at scale. They help robots grasp and manipulate objects, and they support haptics and robotic surgery.
  • Smell and taste: These are understudied senses. Gas sensors and GC-MS now enable large-scale smell datasets, and wearable devices can release smells, transmit taste, and create temperature and mouthfeel sensations.
  • Physical sensing: IoT sensors track human activity, transportation, and health. In science, AI processes molecules, proteins, satellite and remote-sensing data, and climate, ocean, and space sensors, and it can help drive scientific discovery.
  • Medical sensing: AI combines lab tests, imaging, clinical conversations, and multi-omics data to support doctors. Wearables and smartphones allow non-invasive health monitoring.
  • Social sensing: Signals from wearables, everyday environments, and social networks let AI sense the social world. Affective computing uses speech, facial expressions, gestures, and tone of voice to infer emotions, sentiment, and personality.

Scientific Foundations of Multisensory AI

Sensory heterogeneity

Different modalities carry information with diverse qualities, structures, and representations. Heterogeneity arises from six sources:

  1. the basic elements of each modality
  2. their distribution
  3. their structure (e.g., discrete vs. continuous, spatial vs. temporal)
  4. their information content
  5. their noise and imperfections
  6. the tasks each modality suits best
Open questionsHow can we quantify heterogeneity? Should future models be modular and heterogeneity-aware, or unified and heterogeneity-agnostic?

Multimodal interconnections

Modalities are heterogeneous but connected. Connections describe information shared across modalities in the data itself, while interactions describe how modalities combine to create new information for a task. Interactions come in three types:

Redundancythe same information appears in both modalities
Uniquenessthe information exists in only one modality
Synergynew information emerges only when modalities are combined
Open questionsHow can we quantify interactions, for example with information theory? Which fusion functions best capture them?

Unification vs. modularity

Should one model handle every modality, or should separate modules be combined?

  • Unification: Models are converging in their representations, and LLMs already show visual, robotic, and social knowledge without ever seeing those modalities. This motivates unified architectures and scaling laws across modalities.
  • Modularity: Unified models struggle with compositionality, interpretability, and control, which motivates multi-agent and composable modules.
Open questionHow can we find principled frameworks for weighing the pros and cons of each paradigm?

Learning and optimization

Multimodal training brings three new challenges:

  1. Modality collapse: the model relies only on the dominant modality
  2. Modality interference: one modality brings in spurious or conflicting information
  3. Modality imbalance: modalities learn and overfit at different rates

So optimization should be data-dependent, using each modality's heterogeneity and interactions. Multimodal data also offers new training signals: optimizing for synergy, redundancy, and uniqueness, cross-modal prediction, and contrastive learning. There are also brain-inspired ideas, such as cross-modal plasticity and synesthesia.

The Next Frontier for Multisensory Intelligence

Challenge 1Integration

Learning joint representations that capture cross-modal interactions while accounting for how different the modalities are.

Integration is hard for four reasons:

  1. The modality gap: the more different two modalities are, the more specialized their encoders must be, and the less their representations line up (e.g., static vs. temporal, discrete vs. continuous).
  2. Information interactions: deciding whether to encode redundant, unique, or synergistic information, and at which stage to fuse it (from late fusion of separate predictions to early fusion of raw modalities).
  3. Foundation representations: pre-trained models are usually adapted with simple adapters or late fusion, which loses fine-grained detail, compositionality, and spatio-temporal synergy.
  4. Real-world constraints: missing, noisy, or adversarial modalities, plus compression methods that were never designed for multimodal models.
Open directions
  • A rigorous science of multimodal interactions
  • Ways to bridge the modality gap
  • Models natively trained on multiple modalities that learn synergy at fine spatial and temporal resolution
  • Methods that balance representation quality with robustness, safety, and efficiency

Challenge 2Alignment

Modeling fine-grained connections between the elements of different modalities, using the structure already present in the data (e.g., matching a speaker's gestures to their spoken words).

Alignment is hard because connections can be one-to-one, many-to-many, or missing, can span long distances, and often require cutting continuous data into segments with no clear boundaries. Alignment takes three forms:

  1. Explicit alignment: directly learning connections with contrastive learning, optimal transport, or adapters. This means choosing the granularity (token and region vs. caption and image) and the alignment function. Contrastive methods provably capture only redundant information.
  2. Implicit alignment: alignment that emerges without an explicit objective, through shared architectures, parameters, interleaved data, or scale. It breaks down when the modalities are very different or one carries information the other lacks.
  3. Aligned representations: alignment as a hidden intermediate step toward better downstream representations. It can be undirected or directed from a source to a target modality, and learned end-to-end or as a post-hoc layer on frozen encoders.
Tokenization trade-off

Discrete text is much easier to align than continuous images, video, or audio, so continuous data is usually segmented or tokenized first. This gives clear units to match, puts both modalities in the same form (e.g., text-aligned visual tokenizers, unified tokenizers), and lets standard sequence models be reused. The cost is information loss when continuous signals are forced into discrete boundaries. Recent tokenization-free methods avoid this by aligning directly in input space.

This raises the question of where alignment should happen: at the raw data and tokenizer level, at the representation level during early fusion, or after the fact between pre-trained encoders.

Open directions
  • Principled ways to decide when and how to align versus fuse
  • A rigorous study of emergent alignment and its limits on very different modalities
  • Aligned representations that capture fine-grained synergy
  • Alignment for high-frequency continuous data that needs no tokenization or segmentation

Challenge 3Reasoning

Performing multi-step inference across modalities to combine and build on knowledge, guided by the structure of the task.

Language models already reason step by step with explanations, verification, and backtracking, but models that reason jointly over text, images, audio, video, and knowledge graphs are still largely missing. Reasoning has three parts:

  1. Reasoning medium: the intermediate representation each step is written in, such as language, the model's latent space, visual or video chains of thought, or attention maps. Language is easy to read and split into steps, while vision carries more information per step, so different problems may need different mediums or combinations of them.
  2. Reasoning structure: how a problem is broken into substeps (chains, trees, or graphs). The structure should adapt the number and type of steps, keep track of state in interactive settings (often trained with RL), and be learned when experts cannot specify it.
  3. Training and inference: encouraging reasoning through dense supervision or RL. RL needs a good reward or verifier, which is easy for math and code but hard for subjective, human-centered tasks, and it is unclear what cross-modal rewards should look like.
Neuro-symbolic mediums

An emerging alternative medium that uses discrete symbols as the intermediate reasoning steps and combines them with neural models. Directions include logic-based differentiable reasoning, world models used as priors for visual reasoning, and multilingual reasoning for culture-specific tasks. The appeal is pairing the perception of neural networks with the precision and compositional structure of symbolic reasoning.

Long-horizon reasoning

A key frontier as reasoning extends to long text, audio, video, and continuous sensing. Token-efficient architectures (e.g., linear attention) and retrieval help models observe and reason over long time spans. However, current models lack the inductive biases and positional awareness of older recurrent architectures, so they need far more data to generalize to longer, more compositional tasks. Progress may require revisiting fixed-memory recurrent ideas or inventing new ones. This matters because audio, video, and sensor streams hold orders of magnitude more data than internet text.

Open directions
  • Reasoning across input and output mediums (text, images, video, symbols, languages)
  • Understanding the trade-offs between mediums, especially over long horizons
  • Reasoning over structures that reflect cross-modal dependencies
  • Incentives for reasoning when rewards are ambiguous or supervision is scarce
  • Combining reasoning with causal, counterfactual, and compositional inference

Challenge 4Generation

Learning a generative process that produces raw multimodal data while modeling cross-modal interactions and each modality's structure, and keeping the modalities coherent with one another.

Most work so far is cross-modal generation (text to image, video, or audio; image to video), with a growing trend toward unified generation across language, video, and audio. Generation faces three challenges:

  1. Synchrony: keeping generated modalities aligned in time and meaning.
  2. Streaming: generating continuously in real time as inputs change, instead of producing a fixed-length output all at once.
  3. Creativity: supporting creators rather than replacing them, with diverse outputs, user control, explanations and attribution, and scaffolding that is gradually removed as users improve.

Synchrony has three levels:

Within-modality coherence

Each modality stays internally consistent: spatial coherence in images, discourse coherence in text, and spatio-temporal coherence in video. Current unimodal generators mostly handle this already.

ExampleIn a generated video of someone walking, their shirt stays the same color and their body does not warp from frame to frame.
Cross-modal coherence

The output in one modality faithfully follows a control signal from another, such as text, a target style, an image or video, or a robot goal or action trajectory. Common methods are paired data with aligned representations, parameter-efficient fine-tuning, and classifier or classifier-free guidance.

ExampleThe prompt “a red cup to the left of a blue book” produces exactly that image, not a blue cup or a swapped layout.
Multimodal coherence

Several modality streams are generated in parallel and stay consistent with one another, through joint end-to-end training or post-hoc alignment after separate generation.

ExampleIn a generated talking-head video, the lip movements match the generated speech, and a door slam is heard at the moment the door closes.
Causal attention

Streaming generation requires each new output to depend only on what has already happened, not on the future. Most models are trained with bidirectional attention over the whole sequence, so they must see or generate everything at once. Methods such as diffusion forcing and self forcing make these models generate causally and stably, and sliding-window attention, KV caching, distillation, and memory compression keep long streams efficient.

ExampleA real-time sports commentator model must describe a goal as it happens using only the frames seen so far. A bidirectional model could only comment after watching the whole match.
Open directions
  • Synchronizing unimodal generators with limited multimodal data
  • Unified understanding and generation in one shared representation space
  • Efficient real-time streaming over long horizons and many modalities
  • Interactive conditioning on inputs, actions, and feedback
  • Generation that adapts to human intent while preserving human-AI co-creativity and agency

Challenge 5Generalization

Predicting how knowledge transfers from high-resource modalities or pre-trained models to low-resource ones, including both the gains and the risks.

Generalization comes in three types:

  1. Cross-modal transfer: using what was learned in one modality to improve another.
  2. High-modality generalization: transfer among many modalities when only some pairs of them ever appear together in the data.
  3. Scaling tradeoffs: predicting how capabilities and risks change with more data, larger models, and more training, so limited budgets are spent well.
Cross-modal transfer

Cross-modal transfer helps when a second modality reinforces the prediction (redundancy), adds new concepts (uniqueness), or creates new information when combined (synergy). There are three ways to encourage it:

Cross-modal prediction: train the model to translate from the primary modality into a secondary one, so the primary representation learns to “hallucinate” the other modality.

ExampleA sentiment model reads only text but is trained to predict the speaker's facial expression and tone of voice, so its text features absorb emotional cues.

Cross-modal alignment: train several modalities into one shared space, then use only the primary modality at test time.

ExampleCLIP is trained on image-text pairs, yet its image encoder alone often beats vision-only encoders.

Cross-modal fusion: train one model directly on many modalities with autoregressive or masked prediction, with no explicit translation or alignment objective, and let transfer emerge.

ExampleA native multimodal model trained on interleaved text, images, and audio improves on audio tasks because of what it learned from text.

A separate approach keeps unimodal models apart and lets them exchange predictions, through co-training (pseudo-labeling each other's data) or co-regularization (pushing their predictions to agree). This approach suits black-box LLMs.

High-modality generalization

As the number of modalities grows, data rarely covers every pair, and the gaps get worse in sensitive domains. Current approaches use language or vision as the pivot, extend two-modality prediction and alignment objectives to many modalities, or rely on large-scale unsupervised training across all modalities.

ExampleA health team has smartwatch motion data paired with video of people exercising, and video paired with text descriptions, but no motion data paired with text. They still want the watch to report “the user is doing push-ups” in words. Using video as a pivot, motion maps to video and video maps to text, so the model learns motion-to-text without ever seeing that pair.
Scaling tradeoffs

Practitioners with extra budget need to know whether to spend it on unimodal or multimodal data, and on unimodal or multimodal components. It is also unclear whether synergy, alignment, and transfer appear at small scale or only emerge with enough data and parameters.

Risk: scaling does not only help Some parts of multimodal learning get worse with scale:
  • Optimization difficulties: larger multimodal models become harder to train stably.
  • Modality competition: modalities compete during training, and a dominant one (often text) can suppress learning from the others, so adding a modality or scaling up can leave the weaker modality undertrained.

The long-term goal is meta scaling laws: understanding how scaling differs across modalities and tasks well enough to predict how a completely new modality will scale before collecting much data for it.

Open directions
  • New objectives that promote cross-modal transfer
  • Understanding when transfer emerges with respect to data, architecture, and scale
  • Generalization across many modalities when only a subset is fully observed
  • Principled scaling tradeoffs that cover synergy, competition, and transfer
  • Data acquisition frameworks that balance performance with sample efficiency

Challenge 6Experience

Designing systems that interact with people and the environment over many steps to improve productivity, creativity, wellbeing, and human-AI experiences.

The goal is AI that extends human perception, thinking, and emotional intelligence. Experience has three parts:

  1. Multimodal interaction: principles for human-AI interaction across modalities, drawing on AI, HCI, psychology, and cognitive science. Models must accept any modality as input, stream outputs in many modalities, and stay coherent over long interactions. Classic HCI guidelines need updating for generative AI and should cover the medium (speech, vision, haptics, AR/VR, smell), style, content, and duration of an interaction. Interaction should also follow FATE principles (fairness, accountability, transparency, ethics) and show users its ambiguity and uncertainty.
  2. Grounded multimodal agents: agents that act in websites, operating systems, databases, and social settings. They need to propose and verify actions, search for good long-term trajectories, separate high-level planning from low-level actions, and combine memory with reasoning. Beyond self-evolving agents, the paper argues for co-evolving agents that improve alongside humans, express uncertainty, defer to experts, and need less human help over time. Training increasingly relies on world models (interactive video generators conditioned on actions), which still need to be physically grounded, consistent, and fast enough for real time. Extending them to social and biological worlds could enable socially intelligent agents and active clinical assistants in homes, ICUs, and operating rooms.
  3. Human-AI co-experience: experiences that humans and AI build together over time, beyond static generation. Examples include synchronized visual, audio, and haptic feedback for games, and new smell and taste experiences for personalized diets. These systems must sense human states (attention, emotion, memory) in real time, choose the right interaction medium (text, voice, avatar, embodiment), and adapt to cognitive load. Evaluation needs to move from single-turn metrics to long-term, multi-turn measures such as rapport and mutual satisfaction.
Open directions
  • Principles for how humans and AI communicate across sensory channels
  • Modeling human intent, attention, and affect
  • Grounding world models in social and physical commonsense
  • Using AI to create or modify our senses for new experiences
  • Feedback loops through which humans and AI adapt to each other

— TC Chen, 19 Oct. 2026