We consider alternating gradient descent (AGD) with fixed step size $\eta > 0$, applied to the asymmetric matrix factorization objective. We show that, for a rank-$r$ matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$, $T = \left( \left(\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})}\right)^2 \log(1/\epsilon)\right)$ iterations of alternating gradient descent suffice to reach an $\epsilon$-optimal factorization $| \mathbf{A} - \mathbf{X}_T^{\vphantom{\intercal}} \mathbf{Y}_T^{\intercal} |_{\rm F}^2 \leq \epsilon | \mathbf{A} |_{\rm F}^2$ with high probability starting from an atypical random initialization. The factors have rank $d>r$ so that $\mathbf{X}_T\in\mathbb{R}^{m \times d}$ and $\mathbf{Y}_T \in\mathbb{R}^{n \times d}$. Experiments suggest that our proposed initialization is not merely of theoretical benefit, but rather significantly improves convergence of gradient descent in practice. Our proof is conceptually simple: a uniform PL-inequality and uniform Lipschitz smoothness constant are guaranteed for a sufficient number of iterations, starting from our random initialization. Our proof method should be useful for extending and simplifying convergence analyses for a broader class of nonconvex low-rank factorization problems.

Publication

In *Advances in Neural Information Processing Systems 36 (NeurIPS 2023)*

Date

Dec 2023

Tags

Links

Citation

R. Ward, T. G. Kolda.
**Convergence of Alternating Gradient Descent for Matrix Factorization**.
In *Advances in Neural Information Processing Systems 36 (NeurIPS 2023)*,
2023.
http://arxiv.org/abs/2305.06927

Machine Learning (cs.LG), Optimization and Control (math.OC), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Mathematics

```
@inproceedings{WaKo23,
author = {Ward, Rachel and Kolda, Tamara G.},
title = {Convergence of Alternating Gradient Descent for Matrix Factorization},
booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},
month = {December},
year = {2023},
eprint = {2305.06927},
}
```