Medical image segmentation is an important research topic in the field of medical image analysis. It utilizes computer vision and machine learning techniques to automatically or semi-automatically segment target structures in medical images, providing support for disease diagnosis, treatment, and monitoring. Currently, deep learning-based methods such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and attention mechanisms have achieved significant success in medical image segmentation. Traditional methods based on image features, morphology, and statistics, as well as emerging technologies such as multi-modal information integration, uncertainty learning, and augmented reality, have also been applied in medical image segmentation.Our research focuses on optimizing deep learning-based segmentation frameworks and their applications to clinical practice. Firstly, fine segmentation of human organ structures can assist doctors in disease diagnosis. For example, segmentation of structures such as blood vessels, lungs, and hearts can help doctors determine the morphology, location, and size of lesions, guiding clinical decision-making. Secondly, we also aim to integrate segmentation results with surgical navigation systems, allowing doctors to accurately locate target structures during surgery, improving surgical safety and precision.Related Projects - Diagnosis and surgical planning for pediatric congenital heart disease: In collaboration with Shanghai Children's Medical Center, this project has two main tasks. The first task is to diagnose pediatric congenital heart diseases, including Tetralogy of Fallot, single ventricle, and anomalous pulmonary venous drainage. The second task is to plan for Fontan surgery, a surgical repair for single ventricle heart defects, and assess postoperative risks.Selected Papers AEC-Net: Attention and Edge Constraint Network for Medical Image SegmentationEMBC 2020[BibTex]Wang, Jingyi; Zhao, Xu; Ning, Qingtian and Qian, Dahong

We aim to explore the possibilities of computer perception and understanding of 3D scenes. Our basic research involves specifying the position and movement patterns of subjects in the environment from sensor inputs and forming robust 3D representations of the environment and the objects within it, with the ultimate goal of providing a perceptual algorithmic basis for the vision of computers acting freely in the real world. Specifically, we aim to combine classical geometric methods in computer vision with learning-based semantic understanding to achieve this goal efficiently and accurately.On the other hand, robust 3D representation of the environment and the objects within it involves 3D reconstruction of the scene and 3D detection of the objects. To achieve this, we expect to combine computer vision and computer graphics and efficiently extract key information from multimodal sensors to build accurate and comprehensive 3D representations of the scene and objects in terms of practical requirements.Selected Papers MESA: Matching Everything by Segmenting AnythingCVPR 2024[Project][Code][BibTex]Zhang, Yesheng; Zhao, Xu3DTRIP: A General Framework for 3D Trajectory Recovery Integrated With PredictionIEEE Robotics and Automation Letters 2023[BibTex]Ni, Yiyang; Zhao, XuEdgeStereo: An Effective Multi-task Learning Network for Stereo Matching and Edge DetectionIJCV 2020[BibTex]Song, Xiao; Zhao, Xu; Fang, Liangji; Hu, Hanwen; Yu, Yizhou

.p-text {text-align: justify;text-indent: 2em;font-size: 12pt;}.image-cell {width: 20%;text-align: center;vertical-align: center;}Video action understanding uses algorithms to analyze the relationship between movingsubjects, people and people, people and objects, and people and environment in the video and understands the actionsand behaviors of people in the video. Additionally, video action understanding detects the events by combiningthe temporal information of the video. We mainly focus on the four sub-tasks: atomic action segmentation, videoaction recognition, temporal action detection and spatio-temporal action detection.1. Video action recognition is a basic task in video action understanding, which can servedownstream temporal action detection and spatiotemporal action detection tasks. Video action recognition is toidentify the action category in a trimmed video and give the confidence of the action category. Its core is videofeature expression, which usually uses optical flow frames and RGB frames to extract motion information andappearance information in videos. We are committed to extracting video features more efficiently and effectively. Weseek to decouple video features at the spatial and temporal levels at the video level to improve theinterpretability of video features. At the same time, we aim at the huge amount of 3D convolution parameters. Thestructure decomposition of the convolution kernel is used to reduce the amount of model parameters and improve thecalculation efficiency of the model.2. Temporal action detection is to locate the boundary of the action in untrimmed videos, andgive the corresponding action category of the action segment. Temporal action detection has important value in videoediting, video recommendation, and intelligent monitoring. We strive to build more accurate descriptions of actionboundaries and generate high-quality action nomination segments. To propose more accurate action segments, wepropose a more flexible temporal action detection framework. In addition, for the ambiguity of the action boundary,the boundary feature is learned in the video feature expression to highlight the difference between the action andbackground of the video.3. Spatio-Temporal Action Detection is an important research direction in thefield of video behavior understanding. Its goal is to locate and identify interesting actions in videossimultaneously in time and space, which is of great significance for understanding the spatiotemporal evolution modeof actions in videos. Compared with simply detecting the start and end time of an action in time, the detection ofthe spatial position of the action subject on each frame provides a broader practical prospect for the practicalapplication of this research direction. On the basis of the existing frame-level and snippet-level spatio-temporalaction detection methods, we focus on improving the multi-task feature learning ability of the model and improvingthe effect of action modeling. At the same time, we also explore the semi-supervision of spatio-temporal actiondetection in data-scarce scenarios.4. Atomic action segmentation is an important step in understanding the temporal relationshipbetween complex actions. Its goal is to divide a complete action into several atomic actions in time withoutsupervision. These atomic actions constitute a compact representation of the original action in time and a completedescription of semantics. Compared with the rough division of actions under an event in temporal actionsegmentation, unsupervised atomic actions have a more fine-grained division of the original action, which willimprove the ability of action modeling in action recognition and action detection. Based on deep learning, we foucson improving the feature representation ability in unsupervised atomic action segmentation and the ability to modelthe temporal relationship between atomic actions. In addition, we also focus on combining atomic action segmentationwith subsequent video understanding tasks.Selected Papers Constructing Semantical Structure by Segmentation Integrated Video Embedding for Temporal Action DetectionIEEE TCSVT 2025[Project][BibTex]Zhao, Zixuan; Liu, Shuming; Zhao, Chengze; Zhao, XuM3Net: Movement Enhancement with Multi-Relation toward Multi-Scale video representation for Temporal Action DetectionPattern Recognition 2024[BibTex]Zhao, Zixuan; Wang, Dongqi; Zhao, XuMovement Enhancement toward Multi-Scale Video Feature Representation for Temporal Action DetectionICCV 2023[Project][BibTex]Zhao, Zixuan; Wang, Dongqi; Zhao, XuTransferable Knowledge-Based Multi-Granularity Fusion Network for Weakly Supervised Temporal Action DetectionIEEE Transactions on Multimedia 2021[BibTex]Su, Haisheng; Zhao, Xu; Lin, Tianwei; Liu, Shuming; Hu, ZhilanBSN: Boundary Sensitive Network for Temporal Action Proposal GenerationECCV 2018[BibTex]Lin, Tianwei; Zhao, Xu; Su, Haisheng; Wang, Chongjing; Yang, MingSingle Shot Temporal Action DetectionACM MM 2017[BibTex]Lin, Tianwei; Zhao, Xu; Shou, Zheng

3D human surface reconstruction aims to estimate human pose and shape, as well as reconstruct surface and clothing details, using computer vision and graphics methods from data collected by vision sensors. This process encompasses two key tasks: human pose and shape estimation, and clothed human surface reconstruction. The former typically involves estimating a parametric human model from images or videos, while the latter seeks to recover the real-world surface geometric details and texture information as accurately as possible, creating a digital twin of the real person. 3D human surface reconstruction has significant applications in mixed reality, virtual meetings, and film production.For a long time, 3D human surface reconstruction has been a challenging research topic in the fields of computer vision and graphics, primarily due to the complexity and variability in human posture, body shape, surface geometry, and texture color. Traditional methods often use volumetric representations or predefined triangular meshes as human models, reconstructing the human body through multi-view stereo algorithms with images captured by multi-view systems. Our research focuses on developing efficient and practical 3D human body representation methods, leveraging these representations to recover human surface shapes from images or video sequences captured by RGB and depth cameras. We explore various possibilities for 3D human representation from different perspectives, such as spatial, temporal, and frequency domains, striving for a balance between reconstruction accuracy and algorithmic efficiency. Additionally, we emphasize the robustness of the reconstruction process and the lightweight nature of the algorithms, with a strong commitment to translating research findings into practical applications.Selected Papers An Embeddable Implicit IUVD Representation for Part-based 3D Human Surface ReconstructionIEEE TIP 2024[Project][BibTex]Li, Baoxing; Deng, Yong; Yang, Yehui; Zhao, XuTemporally Consistent Reconstruction of 3D Clothed Human Surface with Warp FieldIVC 2023[Project][BibTex]Deng, Yong; Li, Baoxing; Yang, Yehui; Zhao, Xu

As a fundamental research topic in computer vision, Human Pose Estimation (HPE) aims to locate key pointson human body on images or in 3D space, which plays an important role in behavior understanding. Compared to 2D poseestimation which locates key points only on images, 3D pose estimation produces 3D key point locations in 3D space,leading to a wider application. So far, 3D Human pose estimation has become a comprehensive research topic based onmachine learning, projective geometry, optimization, etc.We aim to precisely attain 3D coordinates of human body key points from multi-view images. On the basis of commonmethods in visual pose estimation like deep learning and epipolar geometry, we are seeking the practical pathway toaid 3D HPE using anatomical priors, IMU, dynamics, etc. We also pay special attention to pose estimation underbinocular, especially short-baselined camera settings, and the implementations of research finding in the form ofprojects.Related Projects - Parkinson Diagnosis Selected Papers RSB-Pose: Robust Short-Baseline Binocular 3D Human Pose Estimation with Occlusion HandlingIEEE TIP 2024[Project][BibTex]Wan, Xiaoyue; Chen, Zhuo; Zhao, XuDual-Diffusion for Binocular 3D Human Pose EstimationNeurIPS 2024[Project][Code][BibTex]Wan, Xiaoyue; Chen, Zhuo; Duan, Bingzhi; Zhao, XuFSGait: Fine Grained Self-Supervised Gait Abnormality Detection ACCV 2024[BibTex]Duan, Bingzhi; Wan, Xiaoyue; Zhao, XuJoint-Limb Compound Triangulation With Co-Fixing for Stereoscopic Human Pose Estimation IEEE Transactions on Multimedia 2024[BibTex]Chen, Zhuo; Wan, Xiaoyue; Bao, Yiming; Zhao, XuView Consistency Aware Holistic Triangulation for 3D Human Pose Estimation CVIU 2023[Project][BibTex]Wan, Xiaoyue; Chen, Zhuo; Zhao, XuStructural Triangulation: A Closed-Form Solution to Constrained 3D Human Pose EstimationECCV 2022[Project][Code][BibTex]Chen, Zhuo; Zhao, Xu; Wan, XiaoyueFusePose: IMU-Vision Sensor Fusion in Kinematic Space for Parametric Human PoseEstimation IEEE Transactions on Multimedia 2022[BibTex]Bao, Yiming; Zhao, Xu; Qian, Dahong

研究目标“看人看物看世界，知象知义知因果”，让计算机通过视觉去捕捉微妙而复杂的变化，去感知、理解和重构三维世界，是我们长期致力的研究目标。研究方向“以人为中心的视觉计算”，是实验室的主要科研方向，其含义不仅在于视觉的感知对象是人，更重要的是寻求“科技服务于人，改进人类福祉”的可达路径。研究方向细分：人体姿态估计、人体表面重建、视频行为理解、动态环境感知、视觉检测测量、医学影像分析等。科研项目我们实验室致力于前沿计算机视觉技术的研究和应用，相关方法和技术已应用于智能诊疗、智能工厂、自动驾驶等新一代人工智能示范应用场景。科研项目示例：面向帕金森病早期诊断的多模态细粒度智能评估、基于深度学习的视觉检测算法研究、VRU意图识别和预测技术合作项目等。“以人为中心的视觉计算”之三个研究层次：感知 - 测量 - 理解

(2023 普博生)邮箱：preacher@sjtu.edu.cn课题组：动态环境感知研究兴趣：SLAM、三维重建

(2025届 硕士)邮箱：DuanBingzhi@sjtu.edu.cn课题组：人体姿态估计研究兴趣：姿态估计，步态识别，多模态医疗诊断

(2025届 硕士)邮箱：Siru_Liu@sjtu.edu.cn课题组：医学影像分析研究兴趣：医学图像分割

(2025届 硕士)邮箱：chenyuheng@sjtu.edu.cn课题组：动态环境感知研究兴趣：3D 物体检测、多模态融合

(2022 普博生)邮箱：lbx11sjtu@sjtu.edu.cn课题组：人体表面重建研究兴趣：三维人体表面重建的表征研究

(2021 工程师)邮箱：zhuoran-L@sjtu.edu.cn课题组：人体姿态估计研究兴趣：双目视觉、立体匹配