Lecture Notes: Optimization for Machine Learning

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Proche Pas pertinent

   Lecture Notes: Convex Optimization

   Andreas Habring · 2026 · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Lecture notes for a course on convex optimization taught by Andreas Habring at Graz University of Technology in 2026. The notes cover mathematical preliminaries, fundamental results about existence and of solutions for minimization problems, projected subgradient descent, proximal-gradient methods, heavy ball gradient descent, Nesterov accelerated gradient descent and FISTA, primal-dual methods, and a short intro to optimal transport.

2. Proche* Pas pertinent

   Optimization for machine learning

   Suvrit Sra, S. Nowozin, Stephen J. Wright · 2010 · cité 738 fois · Pas encore résumé
3. Proche* Pas pertinent

   Advanced Lectures on Machine Learning

   O. Bousquet, U. von Luxburg, G. Rätsch · 2004 · Lecture Notes in Computer Science · cité 304 fois · Pas encore résumé
4. Proche Pas pertinent

   Introduction to Machine Learning: Class Notes 67577

   Amnon Shashua · 2009 · Machine Learning · preprint, non relu par les pairs · cité 52 fois · Résumé IA disponible

   Résumé scientifique

   Introduction to Machine learning covering Statistical Inference (Bayes, EM, ML/MaxEnt duality), algebraic and spectral methods (PCA, LDA, CCA, Clustering), and PAC learning (the Formal model, VC dimension, Double Sampling theorem).

5. Proche Pas pertinent

   Lecture Notes: Neural Network Architectures

   Evelyn Herberg · 2023 · Machine Learning · preprint, non relu par les pairs · cité 4 fois · Pas encore résumé

   Résumé scientifique

   These lecture notes provide an overview of Neural Network architectures from a mathematical point of view. Especially, Machine Learning with Neural Networks is seen as an optimization problem. Covered are an introduction to Neural Networks and the following architectures: Feedforward Neural Network, Convolutional Neural Network, ResNet, and Recurrent Neural Network.

6. Apparenté* Pas pertinent

   Theory of Convex Optimization for Machine Learning

   Sébastien Bubeck · 2014 · arXiv.org · preprint, non relu par les pairs · cité 119 fois · Pas encore résumé
7. Apparenté Pas pertinent

   Elementos da teoria de aprendizagem de máquina supervisionada

   Vladimir G. Pestov · 2019 · Machine Learning · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

   Résumé scientifique

   This is a set of lecture notes for an introductory course (advanced undergaduates or the 1st graduate course) on foundations of supervised machine learning (in Portuguese). The topics include: the geometry of the Hamming cube, concentration of measure, shattering and VC dimension, Glivenko-Cantelli classes, PAC learnability, universal consistency and the k-NN classifier in metric spaces, dimensionality reduction, universal approximation, sample compression. There are appendices on metric and normed spaces, measure theory, etc., making the notes self-contained. Este é um conjunto de notas de aula para um curso introdutório (curso de graduação avançado ou o 1o curso de pós) sobre fundamentos d…

8. Apparenté Cité par cet article Une méthode qu’il utilise Pas pertinent

   Practical Bayesian Optimization of Machine Learning Algorithms

   Jasper Snoek, H. Larochelle, Ryan P. Adams · 2012 · Neural Information Processing Systems · cité 10 374 fois · Pas encore résumé

   Résumé scientifique

   The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of…

Les travaux antérieurs sur lesquels il s’appuie

1. Apparenté Une méthode qu’il utilise Pas pertinent

   Input Warping for Bayesian Optimization of Non-Stationary Functions

   Jasper Snoek, Kevin Swersky, R. Zemel et al. · 2014 · International Conference on Machine Learning · cité 346 fois · Pas encore résumé

   Résumé scientifique

   Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in "log-space," to mitigate …

2. Apparenté Pas pertinent

   Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

   Noam Shazeer, Mitchell Stern · 2018 · International Conference on Machine Learning · cité 1 412 fois · Pas encore résumé

   Résumé scientifique

   In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay …

3. Apparenté Pas pertinent

   AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization

   Rachel A. Ward, Xiaoxia Wu, L. Bottou · 2018 · International Conference on Machine Learning · cité 472 fois · Pas encore résumé

   Résumé scientifique

   Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule. Yet, the theoretical guarantees to date for AdaGrad are for online and convex optimization. We bridge this gap by providing theoretical guarantees for the convergence of AdaGrad for smooth, nonconvex functions. We show that the norm version of AdaGrad (AdaGrad-Norm) converges to a stationary point at the $\mathcal{O}(\log(N)/\sqrt{N})$ rate in the stochastic settin…

4. Apparenté* Pas pertinent

   Computing Machinery and Intelligence

   A. Turing · 1950 · The Philosophy of Artificial Intelligence · cité 12 401 fois · Pas encore résumé

Les articles qui le citent

1. Apparenté Pas pertinent

   Adaptive Gradient Methods with Local Guarantees

   Zhou Lu, Wen-Han Xia, Sanjeev Arora et al. · 2022 · arXiv.org · preprint, non relu par les pairs · cité 13 fois · Pas encore résumé

   Résumé scientifique

   Adaptive gradient methods are the method of choice for optimization in machine learning and used to train the largest deep models. In this paper we study the problem of learning a local preconditioner, that can change as the data is changing along the optimization trajectory. We propose an adaptive gradient method that has provable adaptive regret guarantees vs. the best local preconditioner. To derive this guarantee, we prove a new adaptive regret bound in online learning that improves upon previous adaptive online learning methods. We demonstrate the robustness of our method in automatically choosing the optimal learning rate schedule for popular benchmarking tasks in vision and language d…

2. Apparenté Utilise cet article en profondeur Pas pertinent

   Symmetry Teleportation for Accelerated Optimization

   Bohong Zhao, Nima Dehmamy, R. Walters et al. · 2022 · Neural Information Processing Systems · cité 34 fois · Pas encore résumé

   Résumé scientifique

   Existing gradient-based optimization methods update parameters locally, in a direction that minimizes the loss function. We study a different approach, symmetry teleportation, that allows parameters to travel a large distance on the loss level set, in order to improve the convergence speed in subsequent steps. Teleportation exploits symmetries in the loss landscape of optimization problems. We derive loss-invariant group actions for test functions in optimization and multi-layer neural networks, and prove a necessary condition for teleportation to improve convergence rate. We also show that our algorithm is closely related to second order methods. Experimentally, we show that teleportation i…

3. Apparenté Pas pertinent

   Accelerated gradient sliding and variance reduction.

   D. Dvinskikh, S. Omelchenko, A. Tyurin et al. · 2019 · arXiv: Optimization and Control · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   We consider sum-type strongly convex optimization problem (first term) with smooth convex not proximal friendly composite (second term). We show that the complexity of this problem can be split into optimal number of incremental oracle calls for the first (sum-type) term and optimal number of oracle calls for the second (composite) term. Here under `optimal number' we mean estimate that corresponds to the well known lower bound in the absence of another term.

4. Apparenté Pas pertinent

   StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models

   Ding-Zhi Yu, Rui Pan, Yuxing Liu et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 5 fois · Pas encore résumé

   Résumé scientifique

   Sign-based optimization algorithms, such as SignSGD, have garnered attention for their performance in distributed learning and training large foundation models. Despite their empirical superiority, SignSGD is known to diverge on non-smooth objectives, which are ubiquitous due to ReLUs, max-pools, and mixture-of-experts. To overcome this limitation, we propose StoSignSGD, an algorithm that injects structural stochasticity into the sign operator while maintaining an unbiased update step. In the regime of (online) convex optimization, StoSignSGD rigorously resolves the non-convergence issues of SignSGD, achieving a sharp convergence rate matching the lower bound. For the more challenging non-co…

5. Apparenté Pas pertinent

   Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions

   Vladimir Feinberg, Xin-Yi Chen, Y. Sun et al. · 2023 · Neural Information Processing Systems · cité 23 fois · Pas encore résumé

   Résumé scientifique

   Adaptive regularization methods that exploit more than the diagonal entries exhibit state of the art performance for many tasks, but can be prohibitive in terms of memory and running time. We find the spectra of the Kronecker-factored gradient covariance matrix in deep learning (DL) training tasks are concentrated on a small leading eigenspace that changes throughout training, motivating a low-rank sketching approach. We describe a generic method for reducing memory and compute requirements of maintaining a matrix preconditioner using the Frequent Directions (FD) sketch. While previous approaches have explored applying FD for second-order optimization, we present a novel analysis which allow…