Po-Yao (Bernie) Huang

Greetings! I am Po-Yao (Bernie) Huang. I am a research scientist at Facebook AI Research (FAIR) Labs. I obtained my Ph.D. degree from the Language Technologies Institute (LTI) of School of Computer Science (SCS) at Carnegie Mellon University (CMU). My research interest is multimodal machine learning. I am particularly interested in bridging computer vision and natural language processing for the tasks of multimodal machine translation, cross-modal search and retrieval, and large-scale multimodal data mining and analysis.

Contact: berniebear_at_gmail.com
Google Scholar: https://scholar.google.com/citations?user=E8K25LIAAAAJ

Work Experience

Education

Preprints

• MAViL: Masked Audio-Video Learners
  Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer
  

• Dinov2: Learning robust visual features without supervision
  Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski
  

• Diffusion Models as Masked Autoencoders
  Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, Christoph Feichtenhofer
  

• CiT: Curation in Training for Effective Vision-Language Data
  Hu Xu, Saining Xie, Po-Yao Huang, Licheng Yu, Russell Howes, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer
  

• Cm3: A causal masked multimodal model of the internet
  Armen Aghajanyan, Po-Yao Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer
  

---

Publication

• Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
  Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer
  ICML, 2023.

• STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition
  Xiaoyu Zhu, Po-Yao Huang, Junwei Liang, Alexander G Hauptmann
  CVPR, 2023.

• Masked autoencoders that listen
  Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, Christoph Feichtenhofer
  NeurIPS, 2022.

• Video pivoting unsupervised multi-modal machine translation
  Mingjie Li, Po-Yao Huang, Xiaojun Chang, Junjie Hu, Yi Yang, Alex Hauptmann
  IEEE Transactions on Pattern Analysis and Machine Intelligence, June 2022.

• On adversarial robustness of large-scale audio visual learning (Best student paper)
  Juncheng B Li, Shuhui Qu, Xinjian Li, Po-Yao Bernie Huang, Florian Metze
  ICASSP, 2022.

• Videoclip: Contrastive pre-training for zero-shot video-text understanding
  Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
  EMNLP, 2021.

• Towards Multilingual Vision-Language Models
  Po-Yao Huang
  Ph.D. Thesis, 2021. [slides]

• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
  Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
  ACL-Findings, 2021.

• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
  Po-Yao Huang*, Mandela Patrick*, Junjie Hu, Graham Neubig, Florian Metze, Alexander Hauptmann
  NAACL, 2021.

• Support-set bottlenecks for video-text representation learning
  Mandela Patrick*, Po-Yao Huang*, Yuki Asano*, Florian Metze, Alexander Hauptmann, João Henriques, Andrea Vedaldi
  ICLR, 2021.

• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
  Po-Yao Huang, Junjie Hu, Xiaojun Chang, Alexander Hauptmann
  ACL, 2020.

• Structural Analysis and Optimization of Convolutional Neural Networks with a Small Sample Size
  Rhett N D’souza, Po-Yao Huang, Fang-Cheng Yeh
  Scientific Reports, Jan 2020.

• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
  Po-Yao Huang, Xiaojun Chang, Alexander Hauptmann
  EMNLP, 2019.

• Annotation Efficient Cross-Modal Retrieval with Adversarial Attentive Alignment.
  Po-Yao Huang, Guoliang Kang, Wenhe Liu, Xiaojun Chang, Alexander G Hauptmann
  ACM MM, 2019.

• Improving What Cross-Modal Retrieval Models Learn through Object-Oriented Inter- and Intra-Modal Attention Networks
  Po-Yao Huang, Vaibhav, Xiaojun Chang, Alexander Hauptmann
  ACM ICMR, 2019.

• RCAA: Relational Context-Aware Agents for Person Search
  Xiaojun Chang, Po-Yao Huang, Yi-Dong Shen, Xiaodan Liang, Yi Yang, Alexander Hauptmann
  ECCV, 2018.

• Multimodal Filtering of Social Media for Temporal Monitoring and Event Analysis
  Po-Yao Huang, Junwei Liang, Jean-Baptiste Lamare, Alexander Hauptmann
  ACM ICMR, 2018.

• Attention-based Multimodal Neural Machine Translation
  Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, Chris Dyer
  ACL 2016 First Conference on Machine Translation (WMT16), 2016.

• Entity Hierarchy Embedding
  Zhi-Ting Hu, Po-Yao Huang, Yuntian Deng, Yingkai Gao, and Eric Xing [supp]
  ACL, 2015.

Awards

• 1st Place in the TREC Incident Streams challenge 2019
• 1st Place in the TRECVID ActEV (Activities in Extended Videos) challenge 2019
• 2nd Place in the Kinetics-800 Action Recognition challenge in ActivityNet 2019
• 1st Place in the ActEV challenge in ActivityNet 2019
• 2nd Place in the TRECVID Ad-hoc Video Search challenge 2018
• 2nd Place in the Youtube 8M challenge 2017
• Named Siebel Scholar 2016

---

Scholarship

• ACL, ICMR, AAAI, ACM MM, CVPR travel awards/grants, 2015-2019
• NSF travel awards, 2015-2019
• CMU Research Fellowship 2014-2019
• Taiwan's Study Abroad Scholarship, 2016
• Siebel Scholarship, 2016
• Din-Jing Memorial Scholarship, 2009