2026

$N_0$-Foundation: Towards the Age of Tactile Intelligence

NeoteAI Team, Fudan TEAI Team

arXiv coming soon. 2026 Tech Report Core Contributor

$N_0$-Foundation is a tactile-centric foundation for embodied manipulation that integrates tactile sensing hardware, large-scale multimodal data, hardware-agnostic tactile representation learning, and standardized evaluation into one system. Durable vision-based tactile sensors and $N_0$-TacUMI provide a synchronized acquisition stack; NeoData supplies more than 30,000 hours and 1.4M episodes of synchronized visual-tactile interaction across six embodiments and 450+ tasks (with OpenNeoData, a 5,000-hour open-source subset); NeoForce learns transferable contact features from device-independent three-axis force fields; and the NeoReal and NeoSim suites standardize evaluation for contact-rich manipulation in the physical world and simulation.

$N_0$-Foundation: Towards the Age of Tactile Intelligence

NeoteAI Team, Fudan TEAI Team

arXiv coming soon. 2026 Tech Report Core Contributor

$N_0$-Foundation is a tactile-centric foundation for embodied manipulation that integrates tactile sensing hardware, large-scale multimodal data, hardware-agnostic tactile representation learning, and standardized evaluation into one system. Durable vision-based tactile sensors and $N_0$-TacUMI p...

$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

NeoteAI Team, Fudan TEAI Team

2026 Tech Report Core Contributor

$N_0$-VTLA is a vision-tactile-language-action (VTLA) foundation model for fine-grained contact-rich manipulation. To our knowledge, it is the first VTLA model pretrained on tactile data at scale, learning broad contact priors from NeoData, our large-scale visuo-tactile robot dataset. A predictive tactile pathway distills the contact patterns learned at scale into fine motion adjustments, and ALTER, an advantage-conditioned offline RL method, further improves policies from stored deployment data. $N_0$-VTLA wins all nine real-robot NeoReal tasks and reaches 63.8% mean success on a twenty-task simulation suite, against 44.0% for the strongest baseline.

$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

NeoteAI Team, Fudan TEAI Team

2026 Tech Report Core Contributor

$N_0$-VTLA is a vision-tactile-language-action (VTLA) foundation model for fine-grained contact-rich manipulation. To our knowledge, it is the first VTLA model pretrained on tactile data at scale, learning broad contact priors from NeoData, our large-scale visuo-tactile robot dataset. A predictiv...

$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation

NeoteAI Team, Fudan TEAI Team

2026 Tech Report Core Contributor

$N_0$-TWAM is a tactile-native world-action model for contact-rich manipulation that jointly predicts future vision and future contact. To our knowledge, it is the first tactile world-action model trained at large scale, pre-trained with visuo-tactile joint training over tactile-rich demonstrations spanning six embodiments and 450 tasks. An asymmetric Mixture-of-Transformers architecture pairs a full-width video expert with slim action and tactile experts, so touch is both predicted and observed while inference stays real-time. Evaluations on real and simulated benchmarks demonstrate strong capability across contact-rich tasks and clear benefits from data scaling for precise tactile and action prediction.

$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation

NeoteAI Team, Fudan TEAI Team

2026 Tech Report Core Contributor

$N_0$-TWAM is a tactile-native world-action model for contact-rich manipulation that jointly predicts future vision and future contact. To our knowledge, it is the first tactile world-action model trained at large scale, pre-trained with visuo-tactile joint training over tactile-rich demonstratio...

Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents
Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents

Heng Zhou, Zelin Tan, Zhemeng Zhang, Yutao Fan, Yibing Lin, Li Kang, Xiufeng Song, Rui Li, Songtao Huang, Ao Yu, Yuchen Fan, Yanxu Chen, Kaixin Xu, Xiaohong Liu, Yiran Qin, Philip Torr, Chen Zhang, Zhenfei Yin

preprint 2026 Preprint

We investigate whether LLM agent improvements come from the model or the reasoning approach. Testing six inference-time paradigms (Direct, CoT, ReAct, Plan-Execute, Reflection, ReCode) across four frontier LLMs and ten benchmarks, we find that reasoning structure helps dramatically on some tasks but hurts on others—no single paradigm consistently excels, but oracle per-task selection outperforms fixed paradigm approaches. We propose an embedding-based router that selects the optimal paradigm for each task, achieving 53.1% accuracy compared to 50.3% for the best fixed approach, recovering up to 37% of the oracle gap. Paradigm selection should be task-specific rather than architecturally fixed.

Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents
Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents

Heng Zhou, Zelin Tan, Zhemeng Zhang, Yutao Fan, Yibing Lin, Li Kang, Xiufeng Song, Rui Li, Songtao Huang, Ao Yu, Yuchen Fan, Yanxu Chen, Kaixin Xu, Xiaohong Liu, Yiran Qin, Philip Torr, Chen Zhang, Zhenfei Yin

preprint 2026 Preprint

We investigate whether LLM agent improvements come from the model or the reasoning approach. Testing six inference-time paradigms (Direct, CoT, ReAct, Plan-Execute, Reflection, ReCode) across four frontier LLMs and ten benchmarks, we find that reasoning structure helps dramatically on some tasks ...

CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment
CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment

Li Kang*, Yutao Fan*, Rui Li, Heng Zhou, Yiran Qin, Zhemeng Zhang, Songtao Huang, Xiufeng Song, Zaibin Zhang, Bruno N.Y. Chen, Zhenfei Yin, Dongzhan Zhou, Wangmeng Zuo, Lei Bai (* equal contribution)

preprint 2026 Preprint

CoEnv addresses coordination challenges in multi-agent robotic systems by introducing a compositional environment that integrates real-world and simulation components. The framework operates through three stages: digitizing physical workspaces, employing language models for action planning, and transferring validated strategies to real robots with collision safeguards. We demonstrate the effectiveness of our approach on complex multi-arm manipulation tasks, proposing a new paradigm for multi-agent embodied AI.

CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment
CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment

Li Kang*, Yutao Fan*, Rui Li, Heng Zhou, Yiran Qin, Zhemeng Zhang, Songtao Huang, Xiufeng Song, Zaibin Zhang, Bruno N.Y. Chen, Zhenfei Yin, Dongzhan Zhou, Wangmeng Zuo, Lei Bai (* equal contribution)

preprint 2026 Preprint

CoEnv addresses coordination challenges in multi-agent robotic systems by introducing a compositional environment that integrates real-world and simulation components. The framework operates through three stages: digitizing physical workspaces, employing language models for action planning, and t...

Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study
Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study

Zhiheng Xi*, Xin Guo*, Jiaqi Liu*, Jiazheng Zhang*, Yutao Fan, Zhihao Zhang, Shichun Liu, Mingxu Chai, Xiaowei Shi, Yitao Zhai, Xunliang Cai, Tao Gui, Qi Zhang, Xuanjing Huang (* equal contribution)

International Conference on Machine Learning (ICML) 2026 Regular

We investigate whether reinforcement fine-tuning effectively enables LLM agents to generalize across scenarios. Our study examines three dimensions: within-environment generalization across task difficulty, cross-environment transfer to unseen environments, and sequential multi-environment training to assess both transfer and forgetting. Findings reveal that while RFT performs well within familiar environments, it shows weaker transfer to unseen environments, correlating with shifts in semantic priors and observation/action interfaces. Sequential and mixture training approaches demonstrate promise for improving overall generalization while maintaining prior knowledge.

Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study
Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study

Zhiheng Xi*, Xin Guo*, Jiaqi Liu*, Jiazheng Zhang*, Yutao Fan, Zhihao Zhang, Shichun Liu, Mingxu Chai, Xiaowei Shi, Yitao Zhai, Xunliang Cai, Tao Gui, Qi Zhang, Xuanjing Huang (* equal contribution)

International Conference on Machine Learning (ICML) 2026 Regular

We investigate whether reinforcement fine-tuning effectively enables LLM agents to generalize across scenarios. Our study examines three dimensions: within-environment generalization across task difficulty, cross-environment transfer to unseen environments, and sequential multi-environment traini...

Reading ≠ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models
Reading ≠ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models

Heng Zhou, Ao Yu, Li Kang, Yuchen Fan, Yutao Fan, Xiufeng Song, Hejia Geng, Yiran Qin

preprint 2026 Preprint

We identify a critical limitation in current vision-language models: while they excel at "reading what text says," they struggle with understanding "how it looks." We systematically evaluate 15 state-of-the-art VLMs on typography recognition, testing font family, size, style, and color across 26 fonts, four scripts, and three difficulty levels. Findings reveal a striking hierarchy—color recognition performs near-perfectly while font style detection remains universally poor. Model scale does not correlate with improved performance, suggesting the gap stems from training-data omission rather than a capacity ceiling. Fine-tuning on synthetic data improves results substantially, though font style recognition remains resistant, indicating that deeper architectural innovations may be necessary.

Reading ≠ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models
Reading ≠ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models

Heng Zhou, Ao Yu, Li Kang, Yuchen Fan, Yutao Fan, Xiufeng Song, Hejia Geng, Yiran Qin

preprint 2026 Preprint

We identify a critical limitation in current vision-language models: while they excel at "reading what text says," they struggle with understanding "how it looks." We systematically evaluate 15 state-of-the-art VLMs on typography recognition, testing font family, size, style, and color across 26 ...

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

Yujiong Shen*, Yajie Yang*, Zhiheng Xi*, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zhang, Lei Bai, Zhenfei Yin#, Tao Gui#, Xingjun Ma, Qi Zhang, Xuanjing Huang#, Yu-Gang Jiang (* equal contribution, # corresponding author)

International Conference on Machine Learning (ICML) 2026 Regular

SciAgentGym is an interactive benchmark environment featuring 1,780 domain-specific tools across four natural science disciplines. We introduce SciAgentBench for evaluating agent capabilities and observe that success rates drop sharply from 60.6% to 30.9% as interaction horizons extend, even for leading models. We further propose SciForge, a data synthesis method using dependency graphs to generate training trajectories, enabling our SciAgent-8B model to outperform significantly larger competitors while demonstrating cross-domain transfer.

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

Yujiong Shen*, Yajie Yang*, Zhiheng Xi*, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zhang, Lei Bai, Zhenfei Yin#, Tao Gui#, Xingjun Ma, Qi Zhang, Xuanjing Huang#, Yu-Gang Jiang (* equal contribution, # corresponding author)

International Conference on Machine Learning (ICML) 2026 Regular

SciAgentGym is an interactive benchmark environment featuring 1,780 domain-specific tools across four natural science disciplines. We introduce SciAgentBench for evaluating agent capabilities and observe that success rates drop sharply from 60.6% to 30.9% as interaction horizons extend, even for ...

CoEditor++: Instruction-based Visual Editing via Cognitive Reasoning
CoEditor++: Instruction-based Visual Editing via Cognitive Reasoning

Minheng Ni, Yutao Fan, Zhengyuan Yang, Yeli Shen, Yuxiang Wei, Yaowen Zhang, Lijuan Wang, Lei Zhang, Wangmeng Zuo

preprint 2026 Preprint

CoEditor++ is a training-free framework for instruction-based image editing that decomposes editing into "what to edit" and "how to edit" through two cognitive stages with a reflective self-selection mechanism. Built from publicly available components, CoEditor++ achieves competitive results on SmartEdit and AltBear compared to both open-source and proprietary models including GPT-4o, with particularly strong performance in visual consistency while maintaining instruction-following capabilities. We show the effectiveness of CoEditor++ benefits from its structured cognitive design rather than any specific model component.

CoEditor++: Instruction-based Visual Editing via Cognitive Reasoning
CoEditor++: Instruction-based Visual Editing via Cognitive Reasoning

Minheng Ni, Yutao Fan, Zhengyuan Yang, Yeli Shen, Yuxiang Wei, Yaowen Zhang, Lijuan Wang, Lei Zhang, Wangmeng Zuo

preprint 2026 Preprint

CoEditor++ is a training-free framework for instruction-based image editing that decomposes editing into "what to edit" and "how to edit" through two cognitive stages with a reflective self-selection mechanism. Built from publicly available components, CoEditor++ achieves competitive results on S...

Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning

Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhongzhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang#, Zhenfei Yin#, Philip Torr, Lei Bai (# corresponding author)

Annual Meeting of the Association for Computational Linguistics (ACL) 2026 Oral

This work investigates how reinforcement learning shapes large language models during post-training, focusing on mathematical reasoning. Using the Qwen2.5 series (0.5B to 72B parameters), we examine the interactions between model size, data volume, and computational resources, and find that larger models learn more efficiently, performance follows predictable power-law patterns, and reusing quality data is remarkably effective when data is limited, since final performance depends primarily on total optimization steps rather than sample diversity.

Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning

Zelin Tan, Hejia Geng, Xiaohang Yu, Mulei Zhang, Guancheng Wan, Yifan Zhou, Qiang He, Xiangyuan Xue, Heng Zhou, Yutao Fan, Zhongzhi Li, Zaibin Zhang, Guibin Zhang, Chen Zhang#, Zhenfei Yin#, Philip Torr, Lei Bai (# corresponding author)

Annual Meeting of the Association for Computational Linguistics (ACL) 2026 Oral

This work investigates how reinforcement learning shapes large language models during post-training, focusing on mathematical reasoning. Using the Qwen2.5 series (0.5B to 72B parameters), we examine the interactions between model size, data volume, and computational resources, and find that large...

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

Guibin Zhang*, Hejia Geng*, Xiaohang Yu*, Zhenfei Yin#, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Francisco Piedrahita-Velez, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, Lei Bai# (* equal contribution, # corresponding author)

Transactions on Machine Learning Research (TMLR) 2026 Journal

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

Guibin Zhang*, Hejia Geng*, Xiaohang Yu*, Zhenfei Yin#, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Francisco Piedrahita-Velez, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, Lei Bai# (* equal contribution, # corresponding author)

Transactions on Machine Learning Research (TMLR) 2026 Journal

2025

LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge

Heng Zhou, Ao Yu, Yuchen Fan, Jianing Shi, Li Kang, Hejia Geng, Yongting Zhang, Yutao Fan, Yuhao Wu, Tiancheng He, Yiran Qin, Lei Bai, Zhenfei Yin

preprint 2025 Preprint

We present an automated pipeline that constructs evaluation benchmarks from recent knowledge updates by computing deltas between successive Wikidata snapshots, filtering candidate triples for quality, and synthesizing natural-language questions at varying reasoning difficulty levels with answers verified through SPARQL validation. Experiments show that model performance significantly declines when encountering post-training information, particularly on multi-hop reasoning tasks; while retrieval-augmented and larger instruction-tuned models help, they do not fully bridge the recency gap. The benchmark enables continuous assessment of LLMs against evolving knowledge, shifting focus from static memorization toward practical retrieval and reasoning.

LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge

Heng Zhou, Ao Yu, Yuchen Fan, Jianing Shi, Li Kang, Hejia Geng, Yongting Zhang, Yutao Fan, Yuhao Wu, Tiancheng He, Yiran Qin, Lei Bai, Zhenfei Yin

preprint 2025 Preprint

We present an automated pipeline that constructs evaluation benchmarks from recent knowledge updates by computing deltas between successive Wikidata snapshots, filtering candidate triples for quality, and synthesizing natural-language questions at varying reasoning difficulty levels with answers ...

Better know nothing than half-know anything: A Precise and Efficient Dataset for Scientific Reasoning in Language Models
Better know nothing than half-know anything: A Precise and Efficient Dataset for Scientific Reasoning in Language Models

Yutao Fan*, Yizhou Wang*, Zhiheng Xi*, Lintao Wang, Jianyu Wu, Aoran Wang, Guanyu Li, Jiaqi Liu, Pengze Li, Heng Zhou, Jiayang Li, Wangmeng Zuo, Lei Bai, Shixiang Tang, Philip Torr, Zhenfei Yin (* equal contribution)

Under review. 2025

Large Language Models (LLMs) have achieved remarkable progress in reasoning tasks, i.e., coding and mathematics. However, their ability to perform scientific reasoning remains significantly limited, probably hampered by the scarcity of high-quality scientific reasoning datasets. Existing approaches either rely on LLM-generated synthetic data (suffering from noise and hallucinations) or human-compiled documents (facing scarcity and non-standardization). In this paper, we empirically verify that integrating precise knowledge from original scientific documents with formalized questions and consistent answers can mitigate the need for large-scale data. Based on this insight, we design PreciSci, a pipeline for constructing multi-disciplinary scientific reasoning datasets. This pipeline involves extracting knowledge from reliable sources, refining questions for completeness and precision, applying multi-stage filtering to eliminate redundancy and noise, and refining answers to ensure reliable supervision. Leveraging PreciSci, we build Open-Sci, a precise and knowledge-dense scientific reasoning dataset. Experimental evaluations show that despite Open-Sci being less than one-sixth the size of state-of-the-art scientific reasoning datasets, it enables LLMs to achieve approximately 4.49% better performance across diverse discipline-specific benchmarks.

Better know nothing than half-know anything: A Precise and Efficient Dataset for Scientific Reasoning in Language Models
Better know nothing than half-know anything: A Precise and Efficient Dataset for Scientific Reasoning in Language Models

Yutao Fan*, Yizhou Wang*, Zhiheng Xi*, Lintao Wang, Jianyu Wu, Aoran Wang, Guanyu Li, Jiaqi Liu, Pengze Li, Heng Zhou, Jiayang Li, Wangmeng Zuo, Lei Bai, Shixiang Tang, Philip Torr, Zhenfei Yin (* equal contribution)

Under review. 2025

Large Language Models (LLMs) have achieved remarkable progress in reasoning tasks, i.e., coding and mathematics. However, their ability to perform scientific reasoning remains significantly limited, probably hampered by the scarcity of high-quality scientific reasoning datasets. Existing approach...

BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

Zhiheng Xi*, Guanyu Li*, Yutao Fan*, Honglin Guo*, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding, Wangmeng Zuo, Zhenfei Yin, Lei Bai, Tao Ji, Tao Gui, Qi Zhang, Philip Torr, Xuanjing Huang (* equal contribution)

Annual Conference on Neural Information Processing Systems (NeurIPS) 2025 Poster

In this paper, we introduce BMMR, a large-scale bilingual, multimodal, multi-disciplinary reasoning dataset for the community to develop and evaluate large multimodal models (LMMs). BMMR comprises 110k college-level questions spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple-choice, fill-in-the-blank, and open-ended QA-and sourced from both print and digital media such as books, exams, and quizzes. All data are curated and filtered via a human-in-the-loop and scalable framework, and each instance is paired with a high-quality reasoning path. The dataset is organized into two parts: BMMR-Eval that comprises 20,458 high-quality instances to comprehensively assess LMMs'knowledge and reasoning across multiple disciplines in both Chinese and English; and BMMR-Train that contains 88,991 instances to support further research and development, extending the current focus on mathematical reasoning to diverse disciplines and domains. In addition, we propose the process-based multi-discipline verifier (i.e., BMMR-Verifier) for accurate and fine-grained evaluation of reasoning paths. Extensive experiments on 24 models reveal that (i) even SOTA models (e.g., o3 and Gemini-2.5-Pro) leave substantial headroom on BMMR-Eval; (ii) reasoning models exhibit discipline bias and outperform LMMs only on specific subjects; (iii) open-source models still trail their proprietary counterparts; and (iv) fine-tuning on BMMR-Train narrows this gap. Additionally, we conduct reasoning-chain analyses using BMMR-Verifier and other in-depth studies, uncovering the challenges LMMs currently face in multidisciplinary reasoning.

BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset
BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset

Zhiheng Xi*, Guanyu Li*, Yutao Fan*, Honglin Guo*, Yufang Liu, Xiaoran Fan, Jiaqi Liu, Jingchao Ding, Wangmeng Zuo, Zhenfei Yin, Lei Bai, Tao Ji, Tao Gui, Qi Zhang, Philip Torr, Xuanjing Huang (* equal contribution)

Annual Conference on Neural Information Processing Systems (NeurIPS) 2025 Poster

In this paper, we introduce BMMR, a large-scale bilingual, multimodal, multi-disciplinary reasoning dataset for the community to develop and evaluate large multimodal models (LMMs). BMMR comprises 110k college-level questions spanning 300 UNESCO-defined subjects, spanning diverse formats-multiple...

Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning
Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning

Minheng Ni, Yutao Fan, Lei Zhang, Wangmeng Zuo

The International Conference on Learning Representations (ICLR) 2025 Poster

As large-scale models evolve, language instructions are increasingly utilized in multi-modal tasks. Due to human language habits, these instructions often contain ambiguities in real-world scenarios, necessitating the integration of visual context or common sense for accurate interpretation. However, even highly intelligent large models exhibit significant performance limitations on ambiguous instructions, where weak reasoning abilities of disambiguation can lead to catastrophic errors. To address this issue, this paper proposes Visual-O1, a multi-modal multi-turn chain-of-thought reasoning framework. It simulates human multi-modal multi-turn reasoning, providing instantial experience for highly intelligent models or empirical experience for generally intelligent models to understand ambiguous instructions. Unlike traditional methods that require models to possess high intelligence to understand long texts or perform lengthy complex reasoning, our framework does not significantly increase computational overhead and is more general and effective, even for generally intelligent models. Experiments show that our method not only significantly enhances the performance of models of different intelligence levels on ambiguous instructions but also improves their performance on general datasets. Our work highlights the potential of artificial intelligence to work like humans in real-world scenarios with uncertainty and ambiguity.

Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning
Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning

Minheng Ni, Yutao Fan, Lei Zhang, Wangmeng Zuo

The International Conference on Learning Representations (ICLR) 2025 Poster

As large-scale models evolve, language instructions are increasingly utilized in multi-modal tasks. Due to human language habits, these instructions often contain ambiguities in real-world scenarios, necessitating the integration of visual context or common sense for accurate interpretation. Howe...