Biography
Hello! This is Jianing “Jed” Yang (杨佳宁). I am a Ph.D. student in Computer Science and Engineering at University of Michigan. I am a member of SLED lab, advised by Joyce Chai and frequently collaborating with David Fouhey. My work focuses on understanding the 3D physical world through vision and language, enabling robots to act in such environments in a generalizable and controllable manner, and using natural language as feedback to teach and improve embodied agents. My dream is to build and deploy household robots to homes around the world to help humans with daily tasks and needs.
Before joining UMich, I obtained my Master’s in Machine Learning from Carnegie Mellon University where I worked with Prof. Louis-Philippe Morency and Prof. Matt Gormley on Multimodal Natural Language Understanding, Dataset Bias Analysis, and Machine Learning. I got my Bachelor’s in Computer Science from Georgia Tech.
Before I delved fully into research, I worked as a Software Development Engineer at Amazon Web Services, intern and full-time.
Outside of work, you will find me skiing/cooking/baking/photographing.
Click here for my full bio and here for my CV.
⭐️ I'm graduating and actively looking for full-time industry or startup research positions in Robotics and/or 3D Computer Vision starting in 2025. Contact me if you are hiring! ⭐️- Embodied AI
- Multimodal Machine Learning
- Natural Language Processing
- Robobtics
- 3D Vision
- Continual Learning
Ph.D. in Computer Science and Engineering, 2025 (expected)
University of Michigan
M.S. in Machine Learning, 2020
Carnegie Mellon University
B.S. in Computer Science, 2018
Georgia Institute of Technology
News
- [Oct. 2024] 🎉 Multi-object hallucination is accepted to NeurIPS and Teachable Reinforcement Learning is accepted to EMNLP. Check them out!
- [June 2024] 🚀 We released 3D-GRAND, the first million-scale densly-grounded 3D-text dataset for 3D-LLMs! Trained with this data, our model obtained stronger 3D grounding capabilities and drastically reduces hallucinations. We also proposed and released 3D-POPE, a benchmark to evaluate 3D-text hallucinations for 3D-LLMs. Try out live demo on our website!
- [Feb. 2024] ⭐️ I will join the Meta Embodied AI team Summer 2024 as a Research Scientist Intern!
- [Jan. 2024] 🎉 LLM-Grounder is accepted to ICRA 2024!
- [Sep. 2023] Preprint of LLM-Grounder, is now available on arXiv and featured in Hugging Face 🤗 Daily Papers! Watch our YouTube video demo, or try it out yourself on our live demo. Chat with an LLM agent to ground 3D objects!
- [June 2023] 🏆 We won 🥇 First Place ($500,000) in the first-ever Amazon Alexa Prize SimBot Challenge! It was an absolute honor to co-lead the amazing Team SEAGULL with Yichi Zhang! Big thank you and congrats to all of our team members! 🎉 Read our technical report here. (Media coverage: U-M, Amazon Science, 机器之心)
- [May 2023] We open-sourced the 📸 Chat-with-NeRF (Twitter: 1, 2) project with a live demo. Try it out - chat with a 3D room and let us know what you think! 🙌
- [Oct. 2022] Our paper DANLI is accepted to EMNLP 2022 (oral)!
- [Aug. 2022] Team SEAGULL advanced to Phase Two of the 2023 Amazon Alexa Prize SimBot Challenge! I will continue to co-lead our team with Yichi to represent UMich! 🎉
Click here for news archive
- [May 2022] I passed my qualification exam and became a Ph.D. candidate 🙌
- [Nov. 2021] We are selected as one of the First Amazon Alexa Prize SimBot Challenge participants! I will be co-leading Team SEAGULL with Yichi to represent UMich!. 🎉
- [Aug. 2021] I started my Ph.D. at University of Michigan!
- [Mar. 2021] 🎉 Our paper MTAG is accepted to NAACL 2021! The code is available here.
- [Dec. 2020] 🎉 I finished my Master’s and graduated from CMU! 🎉 I will continue to stay as a Research Assistant at CMU’s MultiComp lab until my PhD starts in Fall 2021.
- [July 2020] I’m attending ACL 2020 virtually. Check out our video presentation on QA bias analysis!
- [Apr. 2023] Team SEAGULL advanced to one of the five finalists for the SimBot Challenge and was covered by U-M CSE News! 🎉
- [Feb. 2023] Team SEAGULL is selected as one of the 2023 Amazon Alexa Prize SimBot Challenge semi-finalists! 💪
- [July 9th-17th 2022] I will attend NAACL 2022 in-person in Seattle! Hit me up!
- [Nov. 2020] I’m attending EMNLP 2020 virtually. Happy to chat there!
Publications
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use
SEAGULL: An Embodied Agent for Instruction Following through Situated Dialog
What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasets
SUOD: Toward Scalable Unsupervised Outlier Detection
Industry Experience
- FAIR Embodied AI team
Led a load balancing project to decrease system latency from 20 seconds to milliseconds.
Received award for technical soundness and leadership at 2019 Q2 AWS Identity organization meeting.
Blogs
Academia Experience
- Designed a graph neural network (GNN) algorithm for fusion of multimodal temporal data
- Analyzed language artifacts in video QA datasets
- Built pipeline for multimodal question answering about social situations
- Coordinated annotation of dataset
- Advisor: Prof. Louis-Philippe Morency
- Designed new algorithms to improve scheduled sampling training for seq2seq models
- Validated effectiveness of the method on NER, Machine Translation and Text Summarization tasks
- Advisor: Prof. Matt Gormley
- Built cardiac arrest prediction pipeline using multimodal temporal data collected from ICU patients
- Advisor: Prof. Jimeng Sun