学术发表

2026

AI for Science, 2026 (Accepted Manuscript)

PhononBench: A Large-Scale Phonon-Based Benchmark for Dynamical Stability in Crystal Generation

Xiao-Qi Han, Ze-Feng Gao^#, Wen-Kao Li, Peng-Jie Guo, Zhong-Yi Lu^#

AI for Science, 2026 (Accepted Manuscript), 2026

PDF arXiv Code Publisher

In this work, we introduce PhononBench, the first large-scale benchmark for dynamical stability in AI-generated crystals. Leveraging the recently developed MatterSim interatomic potential, which achieves DFT-level accuracy in phonon predictions across more than 10,000 materials, PhononBench enables efficient large-scale phonon calculations and dynamical-stability analysis for 108,843 crystal structures generated by six leading crystal generation models.

Arxiv

Screening phonon-mediated superconductors from static orbital Hamiltonians

Jian-Feng Zhang, Ze-Feng Gao, Xiao-Qi Han, Dingshun Lv, Miao Gao, Kai Liu, Xinguo Ren, Zhong-Yi Lu, Tao Xiang

Arxiv, 2026

PDF arXiv

In this work, we develop a low-cost, physically transparent framework that identifies strong electron-phonon coupling materials directly from static orbital-based Hamiltonians without explicit phonon perturbation calculations. Applied to more than 36,000 compounds in the MattKeyBond database, it identifies 34 dynamically stable superconducting candidates with calculated Tc > 10 K after DFPT verification.

Arxiv

Twist-induced magnetic topological phase transition in stacked altermagnetic CrO

Zi-Hao Ding, Ze-Feng Gao, Xiang-Hua Kong, Peng-Jie Guo^#, Zhong-Yi Lu^#

Arxiv, 2026

PDF arXiv

In this work, based on symmetry analysis and first-principles calculations, we show that commensurate twisting drives magnetic topological phase transitions in stacked bilayer CrO, transforming an antiferromagnetic Dirac semimetal into a d-wave altermagnetic bipolarized Weyl semimetal or an unconventional compensated magnetic Weyl semimetal, with Weyl points protected by the spin symmetry at generic k points.

Chinese Physics B, 2026

PhononBench-MP40: a spectrum-resolved benchmark dataset for phonon stability

Wen-Kao Li*, Ze-Feng Gao*^#, Zhong-Yi Lu^#

Chinese Physics B, 2026, 2026

PDF arXiv Publisher

In this work, we present PhononBench-MP40, a spectrum-resolved benchmark dataset of Materials Project-derived crystals for workflow-defined phonon stability. The dataset provides 46,899 completed records with paired stability labels and local phonopy spectra, including 16,683 stable and 30,216 unstable records.

Arxiv

Emergent d-wave altermagnetism in chlorine-adsorbed FeSe monolayer

Zi-Hao Ding, Ze-Feng Gao, Kai Liu, Peng-Jie Guo^#, Zhong-Yi Lu^#

Arxiv, 2026

PDF arXiv

In this work, based on symmetry analysis and first-principles calculations, we propose a realistic route to engineer robust altermagnetism in monolayer FeSe, a prototypical iron-based superconductor, by designing a stoichiometric Fe2Se2Cl structure with gate-tunable hole doping. The resulting altermagnetic state exhibits a giant spin splitting of up to 620 meV and persists even in a 10-layer slab model.

Arxiv

PhononScore: a phonon-aware scoring function for dynamical stability

Xiao-Qi Han, Ze-Feng Gao^#, Zhong-Yi Lu^#

Arxiv, 2026

PDF arXiv Publisher

In this work, we propose PhononScore, a phonon-aware scoring function for crystal generation that directly predicts a unified dynamical-stability score from crystal structures, enabling second-level stability-aware reranking without explicit phonon calculations. The online PhononScore platform is available at http://phononbench.cn/phononscore/.

Arxiv

NQS-Agent: Health-Aware Agentic Hyperparameter Optimization for Neural-Network Quantum States

Jia-Qi Wang, Xiao-Qi Han, Ze-Feng Gao, Rong-Qiang He^#, Zhong-Yi Lu^#

Arxiv, 2026

PDF arXiv

In this work, we develop NQS-Agent, an open-source software framework for health-aware hyperparameter optimization in neural-network quantum state (NQS) calculations. Its workflow monitors energy trajectories, detects destructive optimization events, stops unstable calculations, modifies the learning-rate schedule, and resumes optimization from safe checkpoints.

Physical Review B, 114(5): 055125

Bipolarized Weyl semimetals and quantum crystal valley Hall effect in two-dimensional altermagnetic materials

Chao-Yang Tan, Ze-Feng Gao, Huan-Cheng Yang, Kai Liu, Peng-Jie Guo^#, Zhong-Yi Lu^#

Physical Review B, 114(5): 055125, 2026

PDF arXiv Publisher

In this paper, we predict four ideal two-dimensional type-I altermagnetic bipolarized Weyl semimetals Fe2WTe4 and Fe2MoZ4 (Z=S,Se,Te). More significantly, we introduce the quantum crystal valley Hall effect, a phenomenon achievable in three of these materials namely Fe2WTe4, Fe2MoS4, and Fe2MoTe4, when spin-orbit coupling is considered. Furthermore, these materials have the potential to transition from a quantum crystal valley Hall phase to a Chern insulator phase under strain.

Arxiv

InvDesMobility: a reliability-gated first-principles feedback framework for closed-loop materials discovery

Wen-Kao Li*, Ze-Feng Gao*^#, Peng-Jie Guo, Wei Ji, Zhong-Yi Lu^#

Arxiv, 2026

PDF arXiv Code

In this work, we present InvDesMobility, a reliability-gated first-principles feedback framework that integrates multi-agent automated DFT, evidence stratification, generative structure proposal, acquisition ranking, and auditable release for closed-loop inverse design, screening 2.4 million structures and retaining reliability-gated high-mobility candidates.

Arxiv

LEAP: A closed-loop framework for perovskite precursor additive discovery

Xin-De Wang, Zhi-Rui Chen, Ze-Feng Gao^#, Peng-Jie Guo, Cheng Mu^#, Zhong-Yi Lu^#

Arxiv, 2026

PDF arXiv

In this work, we develop LEAP (LLM-driven Exploration via Active Learning for Perovskites), an expert-in-the-loop closed-loop framework that couples a domain-specialized large language model with active learning for iterative additive prioritization, achieving improved device performance in proof-of-concept perovskite solar cell experiments.

Arxiv

Strategic Over-Parameterization for Generalizable Low-Rank Adaptation

Jing Gao, Zhong-Yi Lu, Pan Zhang, Ze-Feng Gao^#

Arxiv, 2026

PDF arXiv

In this work, we introduce LoRA-Over, a framework that enriches the optimization landscape of low-rank adapters during training and folds the extra capacity back into a standard low-rank structure at inference. Across language understanding, dialogue, arithmetic reasoning, and code generation benchmarks, LoRA-Over consistently outperforms vanilla LoRA while keeping the inference cost identical.

Science Bulletin, 71(9): 2196-2199

Stacking-induced type-II quantum spin Hall insulators with high spin Chern number in unconventional magnetism

Chao-Yang Tan, Panjun Feng, Ze-Feng Gao, Fengjie Ma, Peng-Jie Guo^#, Zhong-Yi Lu^#

Science Bulletin, 71(9): 2196-2199, 2026

PDF arXiv Publisher

In this letter, based on the calculations of lattice model, we demonstrate that stacking two type-II quantum spin Hall insulators does not yield a trivial insulator, but instead forms a quantum spin Hall insulator with high spin Chern number. In this phase, there are two pairs of topological edge states with opposite chirality and polarization coexisting in the boundary.

Arxiv

Bridging Crystal Structure and Material Properties via Bond-Centric Descriptors

Jian-Feng Zhang, Ze-Feng Gao, Xiao-Qi Han, Bo Zhan, Dingshun Lv, Miao Gao, Kai Liu, Xinguo Ren, Zhong-Yi Lu, Tao Xiang

Arxiv, 2026

PDF arXiv

In this paper, we introduce MattKeyBond, a bond-centric materials database that explicitly maps the local electronic landscape and bonding interactions of materials, and build bond-centric descriptors that bridge crystal structure and material properties with improved interpretability and generalizability, particularly when training data are scarce.

Acta Physica Sinica, 75(5): 050801

Applications of multi-agent systems in computational materials science

Ying Wu, Zhong-Yi Lu, Ze-Feng Gao^#

Acta Physica Sinica, 75(5): 050801, 2026

Publisher

This review surveys the use of multi-agent systems in computational materials science, with emphasis on automated end-to-end workflows for structure preparation, parameter configuration, simulation, validation, and result analysis. It discusses VASPilot and PhysAgent as representative systems and outlines how specialized agents can improve the efficiency, reproducibility, and accessibility of materials computation.

Physical Review B, 113(9): 094509

Superconductivity in atom-intercalated quaternary hydrides under ambient pressure

Bo-Wen Yao, Zhenfeng Ouyang, Xiao-Qi Han, Chang-Jiang Wu, Peng-Jie Guo, Ze-Feng Gao^#, Zhong-Yi Lu^#

Physical Review B, 113(9): 094509, 2026

PDF arXiv Publisher

In this work, we used our developed AI search engine (InvDesFlow) to perform extensive investigations regarding ambient stable superconducting hydrides. Several quaternary hydrides with high superconducting temperature are predicted.

ChemSusChem, 19(3): e202502563

Large Language Model-Assisted Additive Selection for Synergistic Defect and Crystallization Control in Efficient Inverted Perovskite Solar Cells

Zhirui Chen, Xinde Wang, Jiaqi Wang, Youcai Hu, Huiji Hu, Junyu Nie, Ziyue Jiao, Yi Wang, Qi Li, Zhihai Cheng, Ze-Feng Gao, Zhong-Yi Lu, Cheng Mu^#

ChemSusChem, 19(3): e202502563, 2026

Publisher

In this work, the Perovskite-R1 large language model is used to identify ethyl 2-aminopropanoate hydrochloride as an effective precursor additive for inverted perovskite solar cells. The additive coordinates with uncoordinated lead and iodide ions to passivate defects and regulate crystallization, enabling a champion power-conversion efficiency of 22.58% together with improved long-term device stability.

Molecules, 31(3): 440

Artificial Intelligence for Perovskite Additive Engineering: From Molecular Screening to Autonomous Discovery

Xin-De Wang, Zhi-Rui Chen, Wen-Kao Li, Peng-Jie Guo, Cheng Mu^#, Ze-Feng Gao^#, Zhong-Yi Lu^#

Molecules, 31(3): 440, 2026

Publisher

In this review, we describe a paradigm shift in additive discovery for perovskite solar cells (PSCs) from trial-and-error methods to AI-driven approaches, covering the physicochemical foundations of additive engineering, machine-learning descriptors, active learning optimization, generative models, and the emerging trend of integrating large language models with autonomous laboratories for closed-loop autonomous discovery.

Communications Materials, 7

Perovskite-R1: A Domain-Specialized LLM for Intelligent Discovery of Precursor Additives and Experimental Design

Xin-De Wang, Zhi-Rui Chen, Peng-Jie Guo, Ze-Feng Gao^#, Cheng Mu^#,Zhong-Yi Lu^#

Communications Materials, 7, 2026

PDF arXiv Code Publisher

In this work, we introduce Perovskite-R1, a specialized large language model (LLM) with advanced reasoning capabilities tailored for the discovery and design of PSC precursor additives. By systematically mining and curating 1,232 high-quality scientific publications and integrating a comprehensive library of 33,269 candidate materials, we constructed a domain-specific instruction-tuning dataset using automated question-answer generation and chain-of-thought reasoning.

Frontiers of Physics, 21(4): 045203

Extremely strong spin-orbit coupling effect in light element altermagnetic materials

Shuai Qu, Zhen-Feng Ouyang, Ze-Feng Gao, Hao Sun, Kai Liu, Peng-Jie Guo^#, Zhong-Yi Lu^#

Frontiers of Physics, 21(4): 045203, 2026

PDF arXiv Publisher

In this paper, we demonstrate that strong spin-orbit coupling effect can be realized in light element altermagnetic materials, and propose a mechanism for realizing the corresponding effective spin-orbit coupling. This mechanism reveals the cooperative effect of crystal symmetry, electron occupation, electronegativity, electron correlation, and intrinsic spin-orbit coupling. Our work not only promotes the understanding of light element compounds with strong spin-orbit coupling effect, but also provides an alternative for realizing light element compounds with an effective strong spin-orbit coupling.

Chinese Physics B, 2026, 35(1):017301.

Type-II Dirac nodal chain semimetal CrB4

Xiao-Yao Hou, Ze-Feng Gao, Peng-Jie Guo^#, Jian-Feng Zhang, Zhong-Yi Lu^#

Chinese Physics B, 2026, 35(1):017301., 2026

PDF arXiv Publisher

In this study, based on symmetry analysis and the first-principles electronic structure calculations, we predict that CrB4 is an ideal type-II Dirac nodal chain semimetal protected by the mirror symmetry. Moreover, there are two nodal rings protected by both space-inversion and time-reversal symmetries in CrB4. More importantly,in CrB4 the topologically protected drumhead surface states span the entire Brillouin zone at the Fermi level.

Nature Computational Science, 6(1): 53-66

Discovering physical laws with parallel symbolic enumeration

Kai Ruan, Yilong Xu, Ze-Feng Gao, Yang Liu, Yike Guo, Ji-Rong Wen, Hao Sun^#

Nature Computational Science, 6(1): 53-66, 2026

PDF arXiv Publisher

In this paper, we introduce a parallelized tree search (PTS) model to efficiently distill generic mathematical expressions from limited data. Through a series of extensive experiments, we demonstrate the superior accuracy and efficiency of PTS for equation discovery, which greatly outperforms the state-of-the-art baseline models on over 80 synthetic and experimental datasets (e.g., lifting its performance by up to 99% accuracy improvement and one-order of magnitude speed up).

2025

Neural Networks, 108467.

Enabling Efficient Low-bit Quantization based on Matrix Product Operators for KV Cache Compression

Jia-Qi Wang, Xiao-Qi Han, Peng-Jie Guo, Rong-Qiang He, Ze-Feng Gao^#, Zhong-Yi Lu^#

Neural Networks, 108467., 2025

PDF Code Publisher

In this work, we introduce MPOQ, a novel data-free quantization technique based on matrix product operators (MPO) to effectively compress the KV cache.

Arxiv

PhysAgent: A Multi-Agent Approach to Automated Discovery of Physical Laws

Xiao-Qi Han, Ze-Feng Gao^#, Peng-Jie Guo, Zhong-Yi Lu^#

Arxiv, 2025

PDF arXiv Code

In this work, we introduce PhysAgent, a novel multi-agent system powered by large language models (LLMs), designed to autonomously execute end-to-end scientific workflows—from hypothesis generation and computational modeling to data analysis and discovery.

Arxiv

Anomalous charge density wave in altermagnetism

Zi-Hao Ding, Lei Wang, Zhen-Feng Ouyang, Jingsi Qiao, Ze-Feng Gao, Wei Ji, Kai Liu, Peng-Jie Guo^#, Zhong-Yi Lu^#

Arxiv, 2025

PDF arXiv

In this letter, based on symmetry analysis and first-principles calculations, we propose for the first time that anomalous charge density wave can be realized in two-dimensional altermagnetic WO. The anomalous charge density wave is characterized by three key features: (i) Unlike conventional charge density wave, whose stabilization is driven by the opening of a gap near the Fermi level, the anomalous charge density wave is stabilized by the occupied states with energies shifting lower far away from the Fermi level; (ii) the anomalous charge density wave increases the density of states near the Fermi level and then enhances-rather than diminishes-the metallicity of materials

Chinese Physics B, 34(10): 100301

HTSC-2025: A Benchmark Dataset of Ambient-Pressure High-Temperature Superconductors for AI-Driven Critical Temperature Prediction

Xiao-Qi Han, Ze-Feng Gao^#, Xin-De Wang, Zhenfeng Ouyang, Peng-Jie Guo, Zhong-Yi Lu

Chinese Physics B, 34(10): 100301, 2025

PDF arXiv Code Publisher

In this work, we present the HTSC-2025, an ambient-pressure high-temperature superconducting benchmark dataset. This comprehensive compilation encompasses theoretically predicted superconducting materials discovered by theoretical physicists from 2023 to 2025 based on BCS superconductivity theory, including the renowned XYH6 system, perovskite MXH3 system, M3XH8 system, cage-like BCN-doped metal atomic systems derived from LaH structural evolution, and two-dimensional honeycomb-structured systems evolving from MgB2.

Chinese Physics Letters, 42(7): 070712

Unconventional compensated magnetic material LaMn2SbO6

Xiao-Yao Hou, Ze-Feng Gao, Huan-Cheng Yang, Peng-Jie Guo, Zhong-Yi Lu

Chinese Physics Letters, 42(7): 070712, 2025

PDF arXiv Publisher

In this study, based on symmetry analysis and the first-principles electronic structure calculations, we predict that LaMn2SbO6 is a unconventional compensated magnetic semiconductor. Given that the Mn ions at opposite spin lattice cannot be connected by any symmetry, the spin splitting in LaMn2SbO6 is isotropic. More importantly, LaMn2SbO6 has already been synthesized experimentally, and its magnetic structure has been confirmed by neutron scattering experiments.

npj Computational Materials

InvDesFlow-AL: Active Learning-based Workflow for Inverse Design of Functional Materials

Xiao-Qi Han, Peng-Jie Guo, Ze-Feng Gao^#, Hao Sun^#,Zhong-Yi Lu^#

npj Computational Materials, 2025

PDF arXiv Code Publisher

In this work, we propose a novel inverse material design generative framework called InvDesFlow-AL, which is based on active learning strategies. This framework can iteratively optimize the material generation process to gradually guide it towards desired performance characteristics. In terms of crystal structure prediction, the InvDesFlow-AL model achieves an RMSE of 0.0423 {\AA}, representing an 32.96% improvement in performance compared to exsisting generative models.

Chinese Physics Letters, 42(4): 047301

InvDesFlow: An AI-driven materials inverse design workflow to explore possible high-temperature superconductors

Xiao-Qi Han, Zhenfeng Ouyang, Peng-Jie Guo, Hao Sun, Ze-Feng Gao^#, Zhong-Yi Lu^#

Chinese Physics Letters, 42(4): 047301, 2025

PDF Code Publisher

We develop InvDesFlow, an artificial intelligence (AI)-driven materials inverse design workflow that integrates deep model pre-training and fine-tuning techniques, diffusion models, and physics-based approaches (e.g., first-principles electronic structure calculation) for the discovery of high-Tc superconductors. Utilizing InvDesFlow, we have obtained 74 thermodynamically stable materials with critical temperatures predicted by the AI model to be Tc ≥ 15 K based on a very small set of samples. Notably, these materials are not contained in any existing dataset.

Physical Review B, 111(9): 094411

Crystal valley Hall effect

Chao-Yang Tan, Ze-Feng Gao, Huan-Cheng Yang, Zheng-Xin Liu, Kai Liu, Peng-Jie Guo^#, Zhong-Yi Lu^#

Physical Review B, 111(9): 094411, 2025

PDF arXiv Publisher

In this paper, based on symmetry analysis and the first-principles electronic structure calculations, we demonstrate that the vally Hall effect without time-reversal symmetry can be realized in two-dimensional altermagnetic materials Fe2WSe4 and Fe2WS4. Due to crystal symmetry required, the vally Hall effect without time-reversal symmetry is called crystal vally Hall effect. In addition, under uniaxial strain, both monolayer Fe2WSe4 and Fe2WS4 can realize piezomagnetic effect.

Chinese Physics Letters, 42(2): 027403

AI-driven inverse design of materials: Past, present, and future

Xiao-Qi Han, Xin-De Wang, Meng-Yuan Xu, Zhen Feng, Bo-Wen Yao, Peng-Jie Guo, Ze-Feng Gao^#, Zhong-Yi Lu^#

Chinese Physics Letters, 42(2): 027403, 2025

PDF arXiv Publisher

In this survey, we look back on the latest advancements in AI-driven inverse design of materials by introducing the background, key findings, and mainstream technological development routes. In addition, we summarize the remaining issues for future directions. This survey provides the latest overview of AI-driven inverse design of materials, which can serve as a useful resource for researchers.

Physical Review B, 112(18): L180405

Luttinger-compensated bipolarized magnetic semiconductor

Peng-Jie Guo^#, Huan-Cheng Yang, Xiao-Yao Hou, Ze-Feng Gao, Wei Ji, Zhong-Yi Lu^#

Physical Review B, 112(18): L180405, 2025

PDF arXiv Publisher

In this study, based on symmetry analysis and the first-principles electronic structure calculations, we predict for the first time two Luttinger compensated bipolarized magnetic semiconductors Mn(CN)2 and Co(CN)2 with isotropic spin splitting as in ferromagnetic materials, providing theoretical guidance for searching Luttinger compensated magnetic materials with distinctive properties.

Physical Review B, 111(14): L140501

High-temperature superconductivity in Li2AuH6 mediated by strong electron-phonon coupling under ambient pressure

Zhenfeng Ouyang, Bo-Wen Yao, Xiao-Qi Han, Peng-Jie Guo, Ze-Feng Gao^#, Zhong-Yi Lu^#

Physical Review B, 111(14): L140501, 2025

PDF arXiv Code Publisher

We used our developed AI search engine~(InvDesFlow) to perform extensive investigations regarding ambient stable superconducting hydrides. A cubic structure Li2AuH6 with Au-H octahedral motifs is identified to be a candidate. After performing thermodynamical analysis, we provide a feasible route to experimentally synthesize this material via the known LiAu and LiH compounds under ambient pressure. The further first-principles calculations suggest that Li2AuH6 shows a high superconducting transition temperature (Tc) ∼ 140 K under ambient pressure.

National Science Review, 12(4): nwaf066

AI-accelerated Discovery of Altermagnetic Materials

Ze-Feng Gao*, Shuai Qu*, Bocheng Zeng*, Yang Liu, Ji-Rong Wen, Hao Sun^#, Peng-Jie Guo^#, Zhong-Yi Lu^#

National Science Review, 12(4): nwaf066, 2025

PDF arXiv Code Publisher

In this paper, we successfully discovered 50 new altermagnetic materials that cover metals, semiconductors, and insulators confirmed by the first-principles electronic structure calculations. The wide range of electronic structural characteristics reveals that various novel physical properties manifest in these newly discovered altermagnetic materials, e.g., anomalous Hall effect, anomalous Kerr effect, and topological property.

2024

38th Conference on Neural Information Processing Systems (NeurIPS 2024)

Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation

Yu-Liang Zhan, Zhong-Yi Lu, Hao Sun^#, Ze-Feng Gao^#

38th Conference on Neural Information Processing Systems (NeurIPS 2024), 2024

PDF arXiv Code Publisher

In this paper, we scale up the parameters of the student model during training, to benefit from over-parameterization without increasing the inference latency. In particular, we propose a tensor decomposition strategy that effectively over-parameterizes the relatively small student model through an efficient and nearly lossless decomposition of its parameter matrices into higher-dimensional tensors.

Arxiv

YuLan: An Open-source Large Language Model

Yutao Zhu, Kun Zhou, Kelong Mao, Wentong Chen, Yiding Sun, Zhipeng Chen, Qian Cao, Yihan Wu, Yushuo Chen, Feng Wang, Lei Zhang, Junyi Li, Xiaolei Wang, Lei Wang, Beichen Zhang, Zican Dong, Xiaoxue Cheng, Yuhan Chen, Xinyu Tang, Yupeng Hou, Qiangqiang Ren, Xincheng Pang, Shufang Xie, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, Yankai Lin, Ruihua Song, Jun Xu, Xu Chen, Rui Yan, Zhewei Wei, Di Hu, Wenbing Huang, Ze-Feng Gao， Yueguo Chen, Weizheng Lu, Ji-Rong Wen

Arxiv, 2024

PDF arXiv

In this paper, we design a three-stage pre-training method to enhance YuLan’s overall capabilities. Subsequent phases of training incorporate instruction-tuning and human alignment, employing a substantial volume of high-quality synthesized data. To facilitate the learning of complex and long-tail knowledge, we devise a curriculum-learning framework throughout across these stages, which helps LLMs learn knowledge in an easy-to-hard manner.

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (COLING 2024)

Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study

Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao^#,Yaliang Li, Bolin Ding, Ji-Rong Wen

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (COLING 2024), 2024

PDF arXiv Code Publisher

This work aims to investigate the impact of quantization on \emph{emergent abilities}, which are important characteristics that distinguish LLMs from small language models. Specially, we examine the abilities of in-context learning, chain-of-thought reasoning, and instruction-following in quantized LLMs.

山东大学学报(理学版)

基于矩阵乘积算符表示的序列化推荐模型

刘沛羽，姚博文, 高泽峰^#, 赵鑫^#

山东大学学报(理学版), 2024

PDF Publisher

推荐系统中的序列化推荐任务面临着高度复杂和多样性大的挑战，基于序列化数据的商品表示学习中广泛采用预训练和微调的方法，现有方法通常忽略了在新领域中模型微调可能会遇到的欠拟合和过拟合问题。为了应对这一问题，构建一种基于矩阵乘积算符(matrix product operator, MPO)表示的神经网络结构，并实现2种灵活的微调策略。

Annual Meeting of the Association for Computational Linguistics (ACL2024)

Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression

Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao^#, Yipeng Ma, Tao Wang, Ji-Rong Wen

Annual Meeting of the Association for Computational Linguistics (ACL2024), 2024

PDF arXiv Publisher

In this paper, we introduce DecoQuant, a novel data-free low-bit quantization technique based on tensor decomposition methods, to effectively compress KV cache. Our core idea is to adjust the outlier distribution of the original matrix by performing tensor decomposition, so that the quantization difficulties are migrated from the matrix to decomposed local tensors. Specially, we find that outliers mainly concentrate on small local tensors, while large tensors tend to have a narrower value range.

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Enhancing Parameter-efficient Fine-tuning with Simple Calibration Based on Stable Rank

Peiyu Liu, Ze-Feng Gao, Xiao Zhang, Wayne Xin Zhao^#, Ji-Rong Wen

Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024

PDF Code Publisher

In this paper, we proposed both theoretical analysesand experimental verification for the proposed calibration strategy. Considering efficiency, we further proposetime-aware and structure-aware strategies to determine the most crucial time to commence the fine-tuning procedureand selectively apply parameter matrices for lightweight fine-tuning, respectively.

2023

Arxiv

Altermagnetic ferroelectric LiFe2F6 and spin-triplet excitonic insulator phase

Peng-Jie Guo, Yuhao Gu, Ze-Feng Gao, Zhong-Yi Lu^#

Arxiv, 2023

PDF arXiv

In this paper, we predict that LiFe2F6 is a d-wave altermagnetic and charge-ordering-mediated ferroelectric material. Moreover, the LiFe2F6 transforms into a ferrimagnetic and ferroelectric phase with strong magnetoelectric coupling under biaxial compressive strain. Interestingly, the spins of the valence band and the conduction band are opposite in ferrimagnetic LiFe2F6, which facilitates a simultaneous spin-triplet excitonic insulator phase.

Future of Information and Communication Conference (FICC2024)

Compression Image Dataset Based on Multiple Matrix Product States

Ze-Feng Gao*, Peiyu Liu*, Wayne Xin Zhao^#, Zhi-Yuan Xie, Ji-Rong Wen and Zhong-Yi Lu

Future of Information and Communication Conference (FICC2024), 2023

PDF Publisher

In this paper, we present an effective dataset compression approach based on the matrix product states (short as MPS) andknowledge distillation. MPS can decompose image samples into a sequential product of tensors to achieve task-agnostic image compression by preserving the low-rank information of the images. Based on this property, we use multiple MPS to represent the image datasets samples. Meanwhile, we also designed a task-related component based on knowledge distillation to enhance the generality of the compressed dataset.

Annual Meeting of the Association for Computational Linguistics (ACL2023), Oral (Nominated for Best Paper Reward)

Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization

Ze-Feng Gao, Kun Zhou,Peiyu Liu, Wayne Xin Zhao^#, Ji-Rong Wen

Annual Meeting of the Association for Computational Linguistics (ACL2023), Oral (Nominated for Best Paper Reward), 2023

PDF Code Publisher

In this paper, we focus on just scaling up the parameters of PLMs during fine-tuning, to benefit from the over-parameterization but not increasing the inference latency. Extensive experiments have demonstrated that our approach can significantly boost the fine-tuning performance of small PLMs and even help small PLMs outperform 3x parameterized larger ones.

Association for Computational Linguistics: EMNLP 2023

Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing

Peiyu Liu*, Ze-Feng Gao*, Yushuo Chen, Wayne Xin Zhao^#, Ji-Rong Wen

Association for Computational Linguistics: EMNLP 2023, 2023

PDF arXiv Code Publisher

In this paper, we propose a parameter-efficient pre-training approach that utilizes matrix decomposition and parameter-sharing strategies to scale PLMs. Extensive experiments have demonstrated the effectiveness of our proposed model in reducing the model size and achieving highly competitive performance (i.e. with fewer parameters than BERT-base, we successfully scale the model depth by a factor of 4x and even achieve 0.1 points higher than BERT-large for GLUE score).

Arxiv

Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture

Peiyu Liu*, Ze-Feng Gao*, Yushuo Chen, Wayne Xin Zhao^#, Ji-Rong Wen

Arxiv, 2023

PDF arXiv

In this paper, we propose a highly parameter-efficient approach to scaling pre-trained language models (PLMs) to a deeper model depth based on matrix product operator (MPO) decomposition, which shares the central tensor across all layers to reduce model size while keeping layer-specific auxiliary tensors and adapters for flexible adaptation.

2022

International Conference on Computational Linguistic (COLING2022), Oral Presentation

Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models

Ze-Feng Gao*, Peiyu Liu*, Wayne Xin Zhao#, Zhong-Yi Lu, Ji-Rong Wen

International Conference on Computational Linguistic (COLING2022), Oral Presentation, 2022

PDF arXiv Code

In this paper, we can reduce the parameters of the original MoE architecture by sharing a global central tensor across experts and keeping expert-specific auxiliary tensors. We also design the gradient mask strategy for the tensor structure of MPO to alleviate the overfitting problem.

天津师范大学学报(自然科学版)

利用自注意力机制优化费米网络的数值研究

王佳奇, 高泽峰^#, 李永峰，王璐^#

天津师范大学学报(自然科学版), 2022

PDF arXiv

为了探索不使用特定形式的试探态研究多电子系统基态性质的方法，以至多约10个原子的小分子为例，利用神经网络的方法对多电子系统进行求解.此外，利用包含自注意力机制的Transformer结构对费米网络（FermiNet）进行改进, 结果表明：Transformer-FermiNet能够在保证原费米网络结果精度的同时将网络参数的规模缩减为原来的3/4.

2021

Annual Meeting of the Association for Computational Linguistics (ACL2021), Poster

Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Peiyu Liu*, Ze-Feng Gao*, Wayne Xin Zhao^#, Z.Y. Xie, Zhong-Yi Lu^#, Ji-Rong Wen

Annual Meeting of the Association for Computational Linguistics (ACL2021), Poster, 2021

PDF arXiv Code Slides Publisher

This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics.

2020

Arxiv

Compressing LSTM Networks by Matrix Product Operators

Ze-Feng Gao*, Xingwei Sun*, Lan Gao, Junfeng Li^#, Zhong-Yi Lu^#

Arxiv, 2020

PDF arXiv

We propose an alternative LSTM model to reduce the number of parameters significantly by representing the weight parameters based on matrix product operators (MPO), which are used to characterize the local correlation in quantum states in physics.

IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, 2837-2847

A Model Compression Method With Matrix Product Operators for Speech Enhancement

Xingwei Sun*, Ze-Feng Gao*, Zhong-Yi Lu^#, Junfeng Li^#, Yonghong Yan

IEEE/ACM Transactions on Audio, Speech, and Language Processing 28, 2837-2847, 2020

PDF arXiv Publisher

In this paper, we propose a model compression method based on matrix product operators (MPO) to substantially reduce the number of parameters in DNN models for speech enhancement.

Physical Review Research 2 (2), 023300

Compressing deep neural networks by matrix product operators

Ze-Feng Gao*,Song Cheng*, Rong-Qiang He, Zhi-Yuan Xie^#, Hui-Hai Zhao^#, Zhong-Yi Lu^#, Tao Xiang^#

Physical Review Research 2 (2), 023300, 2020

PDF arXiv Code Publisher

In this paper, we show that neural network can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states.

* 共同第一作者　^# 通讯作者