Training Large-Vocabulary Neural Language Model by Private Federated Learning for Resource-Constrained Devices

*= Equal Contributors

Federated Studying (FL) is a method to coach fashions utilizing knowledge distributed throughout units. Differential Privateness (DP) supplies a proper privateness assure for delicate knowledge. Our aim is to coach a big neural community language mannequin (NNLM) on compute-constrained units whereas preserving privateness utilizing FL and DP. Nevertheless, the DP-noise launched to the mannequin will increase because the mannequin dimension grows, which frequently prevents convergence. We suggest Partial Embedding Updates (PEU), a novel approach to lower noise by reducing payload dimension. Moreover, we undertake Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to scale back the reminiscence calls for of huge fashions on compute-constrained units. This mix of strategies makes it doable to coach large-vocabulary language fashions whereas preserving accuracy and privateness.