Ph.D. Candidate at Institute for Artificial Intelligence, Peking UniversityMy dream is to witness the emergence of artifical general intelligence and to contribute my own efforts toward it.
During my undergraduate years, I was passionate about studying classical algorithms and participated in algorithm competitions. This not only honed my programming skills but also taught me to approach problems using computational thinking. In graduate school, I delved into deep learning and conducted research in areas such as neural architecture search and computer vision. These experiences allowed me to appreciate the fascinating world of artificial intelligence and marked the beginning of my journey as a researcher. During my doctoral studies, I focused on open-world instruction-following agents as my research topic, diving deep into both the technical and theoretical aspects of this field. Over time, I have developed the ability to examine problems in machine learning from a broader and more unified perspective, often through the lens of probability and learning. I am deeply passionate about the theory and application of general-purpose foundational decision-making models and am committed to advancing research in this area.
Research Interests: Machine Learning, Generative Models, Computer Vision, Robotics, Sequential Control
Warning
Problem: The current name of your GitHub Pages repository ("Solution: Please consider renaming the repository to "
http://".
However, if the current repository name is intended, you can ignore this message by removing "{% include widgets/debug_repo_name.html %}" in index.html.
Action required
Problem: The current root path of this site is "baseurl ("_config.yml.
Solution: Please set the
baseurl in _config.yml to "Education
-
Peking UniversityInstitute for Artificial Intelligence
Ph.D. CandidateSep. 2022 - Jul. 2026 -
University of Chinese Academy of SciencesInstitute of Computing Technology, Chinese Academy of Sciences
M.S. in Computer ScienceSep. 2019 - Jul. 2022 -
Xi'an Jiaotong UniversitySoftware College
B.Eng. in Software EngineeringSep. 2015 - Jul. 2019
Experience
-
Beijing Institute for General Artificial IntelligenceResearch InternMar. 2022 - Sep. 2024 -
Beijing Bytedance technology company LimitedResearch InternOct. 2020 - May 2021
Honors & Awards
-
Gold Medal 🏅 Top 3.3%
at the 43rd ICPC International Collegiate Programming Contest Asia Regional2018 -
Outstanding Student Award
at Xi’an Jiaotong University for undergraduate students2019
News
Selected Publications (view all )
ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment
Shaofei Cai, Zhancun Mu, Anji Liu, Yitao Liang
arxiv 2025
We aim to develop a goal specification method that is semantically clear, spatially sensitive, and intuitive for human users to guide agent interactions in embodied environments. Specifically, we propose a novel cross-view goal alignment framework that allows users to specify target objects using segmentation masks from their own camera views rather than the agent's observations.
ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment
Shaofei Cai, Zhancun Mu, Anji Liu, Yitao Liang
arxiv 2025
We aim to develop a goal specification method that is semantically clear, spatially sensitive, and intuitive for human users to guide agent interactions in embodied environments. Specifically, we propose a novel cross-view goal alignment framework that allows users to specify target objects using segmentation masks from their own camera views rather than the agent's observations.
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
Shaofei Cai, Zihao Wang, Kewei Lian, Zhancun Mu, Xiaojian Ma, Anji Liu, Yitao Liang
IEEE/CVF Computer Vision and Pattern Recognition (CVPR) 2025
We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2.
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
Shaofei Cai, Zihao Wang, Kewei Lian, Zhancun Mu, Xiaojian Ma, Anji Liu, Yitao Liang
IEEE/CVF Computer Vision and Pattern Recognition (CVPR) 2025
We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2.
MineStudio: A Streamlined Package for Minecraft AI Agent Development
Shaofei Cai*, Zhancun Mu*, Kaichen He, Bowei Zhang, Xinyue Zheng, Anji Liu, Yitao Liang (* equal contribution)
arxiv 2024
MineStudio is an open-source software package designed to streamline embodied policy development in Minecraft. MineStudio represents the first comprehensive integration of seven critical engineering components: simulator, data, model, offline pretraining, online finetuning, inference, and benchmark, thereby allowing users to concentrate their efforts on algorithm innovation.
MineStudio: A Streamlined Package for Minecraft AI Agent Development
Shaofei Cai*, Zhancun Mu*, Kaichen He, Bowei Zhang, Xinyue Zheng, Anji Liu, Yitao Liang (* equal contribution)
arxiv 2024
MineStudio is an open-source software package designed to streamline embodied policy development in Minecraft. MineStudio represents the first comprehensive integration of seven critical engineering components: simulator, data, model, offline pretraining, online finetuning, inference, and benchmark, thereby allowing users to concentrate their efforts on algorithm innovation.
GROOT: Learning to Follow Instructions by Watching Gameplay Videos
Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, Yitao Liang
International Conference on Learning Representations (ICLR) 2024 Spotlight Top 6.2%
This paper studies the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space.
GROOT: Learning to Follow Instructions by Watching Gameplay Videos
Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, Yitao Liang
International Conference on Learning Representations (ICLR) 2024 Spotlight Top 6.2%
This paper studies the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space.
Automatic Relation-aware Graph Network Proliferation
Shaofei Cai, Liang Li, Xinzhe Han, Jiebo Luo, Zhengjun Zha, Qingming Huang
IEEE/CVF Computer Vision and Pattern Recognition (CVPR) 2022 Oral Top 4.2%
This paper proposes Automatic Relation-aware Graph Network Proliferation (ARGNP) for efficiently searching GNNs with a relation-guided message passing mechanism. Specifically, we first devise a novel dual relation-aware graph search space that comprises both node and relation learning operations. These operations can extract hierarchical node/relational information and provide anisotropic guidance for message passing on a graph. Second, analogous to cell proliferation, we design a network proliferation search paradigm to progressively determine the GNN architectures by iteratively performing network division and differentiation.
Automatic Relation-aware Graph Network Proliferation
Shaofei Cai, Liang Li, Xinzhe Han, Jiebo Luo, Zhengjun Zha, Qingming Huang
IEEE/CVF Computer Vision and Pattern Recognition (CVPR) 2022 Oral Top 4.2%
This paper proposes Automatic Relation-aware Graph Network Proliferation (ARGNP) for efficiently searching GNNs with a relation-guided message passing mechanism. Specifically, we first devise a novel dual relation-aware graph search space that comprises both node and relation learning operations. These operations can extract hierarchical node/relational information and provide anisotropic guidance for message passing on a graph. Second, analogous to cell proliferation, we design a network proliferation search paradigm to progressively determine the GNN architectures by iteratively performing network division and differentiation.