Po-Yao (Bernie) Huang

Greetings! I am Po-Yao (Bernie) Huang. I am a research scientist at Facebook AI Research (FAIR) Labs. I obtained my Ph.D. degree from the Language Technologies Institute (LTI) of School of Computer Science (SCS) at Carnegie Mellon University (CMU). My research focus lies at the intersection of multimodal machine learning, encompassing vision, language, and audio, with a particular emphasis on multimodal large language models (MLLMs).

Contact: berniebear_at_gmail.com
Google Scholar: https://scholar.google.com/citations?user=E8K25LIAAAAJ

Work Experience

Education

Publication

• The Llama 3 herd of models (Llama3)
  Llama 3 Team. (Contributor to Llama 3.2 multimodal part)
  

• Chameleon: Mixed-modal early-fusion foundation models
  Chameleon Team. (Joint first author. Contributor to pre-training.)
  

• Cm3: A causal masked multimodal model of the internet
  Armen Aghajanyan, Po-Yao Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer
  

• Dinov2: Learning robust visual features without supervision
  Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski
  JMLR, Jan 2024.

• VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
  Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, David Harwath
  ACL, 2024.

• SeamlessM4T-Massively Multilingual & Multimodal Machine Translation
  Seamless Team.
  Nature, 2024.

• Altogether: Image Captioning via Re-aligning Alt-text
  Hu Xu, Po-Yao Huang, Xiaoqing Ellen Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
  EMNLP, 2024.

• Demystifying clip data (MetaCLIP)
  Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer
  ICLR, 2024.

• MoDE: CLIP Data Experts via Clustering
  Jiawei Ma, Po-Yao Huang, Saining Xie, Shang-Wen Li, Luke Zettlemoyer, Shih-Fu Chang, Wen-Tau Yih, Hu Xu
  CVPR, 2024.

• MAViL: Masked Audio-Video Learners
  Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer
  NeurIPS, 2023.

• Diffusion Models as Masked Autoencoders
  Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, Christoph Feichtenhofer
  ICCV, 2023.

• CiT: Curation in Training for Effective Vision-Language Data
  Hu Xu, Saining Xie, Po-Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer
  ICCV, 2023.

• Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
  Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer
  ICML, 2023.

• STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition
  Xiaoyu Zhu, Po-Yao Huang, Junwei Liang, Alexander G Hauptmann
  CVPR, 2023.

• Masked autoencoders that listen
  Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, Christoph Feichtenhofer
  NeurIPS, 2022.

• Video pivoting unsupervised multi-modal machine translation
  Mingjie Li, Po-Yao Huang, Xiaojun Chang, Junjie Hu, Yi Yang, Alex Hauptmann
  IEEE Transactions on Pattern Analysis and Machine Intelligence, June 2022.

• On adversarial robustness of large-scale audio visual learning (Best student paper)
  Juncheng B Li, Shuhui Qu, Xinjian Li, Po-Yao Bernie Huang, Florian Metze
  ICASSP, 2022.

• Videoclip: Contrastive pre-training for zero-shot video-text understanding
  Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
  EMNLP, 2021.

• Towards Multilingual Vision-Language Models
  Po-Yao Huang
  Ph.D. Thesis, 2021. [slides]

• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
  Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
  ACL-Findings, 2021.

• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
  Po-Yao Huang*, Mandela Patrick*, Junjie Hu, Graham Neubig, Florian Metze, Alexander Hauptmann
  NAACL, 2021.

• Support-set bottlenecks for video-text representation learning
  Mandela Patrick*, Po-Yao Huang*, Yuki Asano*, Florian Metze, Alexander Hauptmann, João Henriques, Andrea Vedaldi
  ICLR, 2021.

• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
  Po-Yao Huang, Junjie Hu, Xiaojun Chang, Alexander Hauptmann
  ACL, 2020.

• Structural Analysis and Optimization of Convolutional Neural Networks with a Small Sample Size
  Rhett N D’souza, Po-Yao Huang, Fang-Cheng Yeh
  Scientific Reports, Jan 2020.

• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
  Po-Yao Huang, Xiaojun Chang, Alexander Hauptmann
  EMNLP, 2019.

• Annotation Efficient Cross-Modal Retrieval with Adversarial Attentive Alignment.
  Po-Yao Huang, Guoliang Kang, Wenhe Liu, Xiaojun Chang, Alexander G Hauptmann
  ACM MM, 2019.

• Improving What Cross-Modal Retrieval Models Learn through Object-Oriented Inter- and Intra-Modal Attention Networks
  Po-Yao Huang, Vaibhav, Xiaojun Chang, Alexander Hauptmann
  ACM ICMR, 2019.

• RCAA: Relational Context-Aware Agents for Person Search
  Xiaojun Chang, Po-Yao Huang, Yi-Dong Shen, Xiaodan Liang, Yi Yang, Alexander Hauptmann
  ECCV, 2018.

• Multimodal Filtering of Social Media for Temporal Monitoring and Event Analysis
  Po-Yao Huang, Junwei Liang, Jean-Baptiste Lamare, Alexander Hauptmann
  ACM ICMR, 2018.

• Attention-based Multimodal Neural Machine Translation
  Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, Chris Dyer
  ACL 2016 First Conference on Machine Translation (WMT16), 2016.

• Entity Hierarchy Embedding
  Zhi-Ting Hu, Po-Yao Huang, Yuntian Deng, Yingkai Gao, and Eric Xing [supp]
  ACL, 2015.

Awards

• 1st Place in the TREC Incident Streams challenge 2019
• 1st Place in the TRECVID ActEV (Activities in Extended Videos) challenge 2019
• 2nd Place in the Kinetics-800 Action Recognition challenge in ActivityNet 2019
• 1st Place in the ActEV challenge in ActivityNet 2019
• 2nd Place in the TRECVID Ad-hoc Video Search challenge 2018
• 2nd Place in the Youtube 8M challenge 2017
• Named Siebel Scholar 2016

---

Scholarship

• ACL, ICMR, AAAI, ACM MM, CVPR travel awards/grants, 2015-2019
• NSF travel awards, 2015-2019
• CMU Research Fellowship 2014-2019
• Taiwan's Study Abroad Scholarship, 2016
• Siebel Scholarship, 2016
• Din-Jing Memorial Scholarship, 2009