About
Welcome to my homepage! My full name is Huy Minh Nguyen, and I am a second-year Ph.D student at the Department of Statistics and Data Sciences, University of Texas at Austin where I am fortunate to be advised by Professor Nhat Ho and Professor Alessandro Rinaldo. Before that, I graduated from Ho Chi Minh City University of Science with a Bachelor’s degree in Mathematics (Summa Cum Laude). I am currently working as a research intern at Microsoft AI.
Email: huynm@utexas.edu
Research Interests
My current research focuses on theoretical foundations for Mixture-of-Experts models. In particular, I try to comprehend the effects of various gating functions (namely softmax gate, top-K sparse softmax gate, dense-to-sparse gate, etc) on the convergence of expert estimation under the Mixture-of-Experts models. Based on insights from these results, I aim to design novel gating functions which help improve the performance of Mixture-of-Experts applications, including Large Language Models, Continual Learning and Medical Images. Additionally, I am also interested in Optimal Transport theory.
(*) denotes equal contribution.
Selected Preprints
Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Under review [arXiv]
Statistical Advantages of Perturbing Cosine Router in Sparse Mixture of Experts
Huy Nguyen, Pedram Akbarian*, Trang Pham*, Trang Nguyen*, Shujian Zhang, Nhat Ho
Under review [arXiv]
Mixture of Experts Meets Prompt-Based Continual Learning
Minh Le, An Nguyen*, Huy Nguyen*, Trang Nguyen*, Trang Pham*, Linh Van Ngo, Nhat Ho
Under review [arXiv]
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
Xing Han, Huy Nguyen*, Carl Harris*, Nhat Ho, Suchi Saria
Under review [arXiv]
CompeteSMoE - Effective Training of Sparse Mixture of Experts via Competition
Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, Nhat Ho
Under review [arXiv]
Selected Publications on Mixture of Experts
On Least Square Estimation in Softmax Gating Mixture of Experts
Huy Nguyen, Nhat Ho, Alessandro Rinaldo
Proceedings of the ICML, 2024. [arXiv]
Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
Huy Nguyen, Pedram Akbarian, Nhat Ho
Proceedings of the ICML, 2024. [arXiv]
Demystifying Softmax Gating Function in Gaussian Mixture of Experts
Huy Nguyen, TrungTin Nguyen, Nhat Ho
Advances in NeurIPS, 2023 (Spotlight) . [arXiv] [NeurIPS]
Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
Huy Nguyen, Pedram Akbarian, Fanqi Yan, Nhat Ho
Proceedings of the ICLR, 2024. [arXiv] [ICLR]
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
Huy Nguyen, Pedram Akbarian, TrungTin Nguyen, Nhat Ho
Proceedings of the ICML, 2024. [arXiv]
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
Huy Nguyen*, TrungTin Nguyen*, Khai Nguyen, Nhat Ho
In AISTATS, 2024. [arXiv] [AISTATS]
On Parameter Estimation in Deviated Gaussian Mixture of Experts
Huy Nguyen, Khai Nguyen, Nhat Ho
In AISTATS, 2024. [arXiv]
Selected Publications on Optimal Transport
Entropic Gromov-Wasserstein between Gaussian Distributions
Huy Nguyen*, Khang Le*, Dung Le*, Dat Do, Tung Pham, Nhat Ho
Proceedings of the ICML, 2022. [arXiv] [ICML]
On Multimarginal Partial Optimal Transport: Equivalent Forms and Computational Complexity
Huy Nguyen*, Khang Le*, Khai Nguyen, Tung Pham, Nhat Ho
In AISTATS, 2022. [arXiv] [AISTATS]
On Robust Optimal Transport: Computational Complexity and Barycenter Computation
Huy Nguyen*, Khang Le*, Quang Minh Nguyen, Tung Pham, Hung Bui, Nhat Ho
Advances in NeurIPS, 2021. [arXiv] [NeurIPS]
Recent News
- [May 2024] I start working as a research intern at Microsoft AI.
- [May 2024] Three new papers on Mixture of Experts [1], [2] and [3] are out!
- [May 2024] Three papers on Mixture of Experts, [1], [2] and [3], are accepted to ICML 2024.
- [Apr 2024] I was offered the AISTATS 2024 registration grant. See you in Valencia, Spain this May!
- [Mar 2024] I received the ICLR 2024 Travel Award. See you in Vienna, Austria this May!
- [Feb 2024] Two new papers on the applications of Mixture of Experts in Medical Images [1] and Large Language Models [2] are out!
- [Feb 2024] Two new papers on the theory of Mixture of Experts, [1] and [2], are out!
- [Jan 2024] Two papers on Mixture of Experts, [1] and [2], are accepted to AISTATS 2024.
- [Jan 2024] Our paper “Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts” is accepted to ICLR 2024.
- [Dec 2023] Our paper “Fast Approximation of the Generalized Sliced-Wasserstein Distance” is accepted to ICASSP 2024.
- [Oct 2023] I received the NeurIPS 2023 Scholar Award. See you in New Orleans this December!
- [Oct 2023] Our new paper “A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts” is out.
- [Sep 2023] Our new paper “Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts” is out.
- [Sep 2023] We have two papers accepted to NeurIPS 2023, [1] as spotlight and [2] as poster.
- [Jul 2023] We will present the paper “Fast Approximation of the Generalized Sliced-Wasserstein Distance” at the Frontier4LCD workshop, ICML 2023.
- [May 2023] Three new papers on the Mixture of Experts theory are out! See more at [1], [2] and [3].
- [Feb 2023] Our new paper on Mixture Models theory “Minimax Optimal Rate for Parameter Estimation in Multivariate Deviated Models” is out.
Professional Services
- Conference Reviewer: ICML (2022,2024), NeurIPS (2022-2024), AISTATS (2022-2024) and ICLR (2024).
- Journal Reviewer: Electronic Journal of Statistics
- Workshop Reviewer: Frontier4LCD (ICML 2023).