EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   On the Efficiency of Convolutional Neural Networks

   Andrew Lavin · 2024 · Machine Learning · AI summary available

   Abstract

   Since the breakthrough performance of AlexNet in 2012, convolutional neural networks (convnets) have grown into extremely powerful vision models. Deep learning researchers have used convnets to produce accurate results that were unachievable a decade ago. Yet computer scientists make computational efficiency their primary objective. Accuracy with exorbitant cost is not acceptable; an algorithm must also minimize its computational requirements. Confronted with the daunting computation that convnets use, deep learning researchers also became interested in efficiency. Researchers applied tremendous effort to find the convnet architectures that have the greatest efficiency. However, skepticism g…

2. Close Not relevant

   Efficiently Scaling Transformer Inference

   Reiner Pope, Sholto Douglas, Aakanksha Chowdhery et al. · 2022 · Machine Learning · AI summary available

   Abstract

   We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Better understanding of the engineering tradeoffs for inference for large Transformer-based models is important as use cases of these models are growing rapidly throughout application areas. We develop a simple analytical model for inference efficiency to select the best multi-dimensional partitioning techniques optimized for TPU v4 slices based on the application requirements. We combine these with a suite of low-level optimizations to achieve a new Pareto frontier on the latency and model FLOPS utili…

3. Close Not relevant

   SIFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

   Shreyas Saxena, Vithursan Thangarasa, Abhay Gupta et al. · 2023 · Machine Learning · AI summary available

   Abstract

   Recent works have explored the use of weight sparsity to improve the training efficiency (test accuracy w.r.t training FLOPs) of deep neural networks (DNNs). These works aim to reduce training FLOPs but training with sparse weights often leads to accuracy loss or requires longer train schedules, making the resulting training efficiency less clear. In contrast, we focus on using sparsity to increase accuracy while using the same FLOPS as the dense model and show training efficiency gains through higher accuracy. In this work, we introduce SIFT, a family of Sparse Iso-FLOP Transformations which are used as drop-in replacements for dense layers to improve their representational capacity and FLO…

4. Close Not relevant

   Auto-scaling Vision Transformers without Training

   Wuyang Chen, Wei Huang, Xianzhi Du et al. · 2022 · Machine Learning · AI summary available

   Abstract

   This work targets automated designing and scaling of Vision Transformers (ViTs). The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart. To tackle these issues, we propose As-ViT, an auto-scaling framework for ViTs without training, which automatically discovers and scales up ViTs in an efficient and principled manner. Specifically, we first design a "seed" ViT topology by leveraging a training-free search process. This extremely fast search is fulfilled by a comprehensive study of ViT's network complexity, yielding a st…

5. Close Not relevant

   Compute Better Spent: Replacing Dense Layers with Structured Matrices

   Shikai Qiu, Andres Potapczynski, Marc Finzi et al. · 2024 · Machine Learning · AI summary available

   Abstract

   Dense linear layers are the dominant computational bottleneck in foundation models. Identifying more efficient alternatives to dense matrices has enormous potential for building more compute-efficient models, as exemplified by the success of convolutional networks in the image domain. In this work, we systematically explore structured matrices as replacements for dense matrices. We show that different structures often require drastically different initialization scales and learning rates, which are crucial to performance, especially as models scale. Using insights from the Maximal Update Parameterization, we determine the optimal scaling for initialization and learning rates of these unconve…

6. Close Not relevant

   Towards Sustainable Learning: Coresets for Data-efficient Deep Learning

   Yu Yang, Hao Kang, Baharan Mirzasoleiman · 2023 · Machine Learning · AI summary available

   Abstract

   To improve the efficiency and sustainability of learning deep models, we propose CREST, the first scalable framework with rigorous theoretical guarantees to identify the most valuable examples for training non-convex models, particularly deep networks. To guarantee convergence to a stationary point of a non-convex function, CREST models the non-convex loss as a series of quadratic functions and extracts a coreset for each quadratic sub-region. In addition, to ensure faster convergence of stochastic gradient methods such as (mini-batch) SGD, CREST iteratively extracts multiple mini-batch coresets from larger random subsets of training data, to ensure nearly-unbiased gradients with small varia…

7. Close Not relevant

   Scaling Exponents Across Parameterizations and Optimizers

   Katie Everett, Lechao Xiao, Mitchell Wortsman et al. · 2024 · Machine Learning · AI summary available

   Abstract

   Robust and effective scaling of models from small to large width typically requires the precise adjustment of many algorithmic and architectural details, such as parameterization and optimizer choices. In this work, we propose a new perspective on parameterization by investigating a key assumption in prior work about the alignment between parameters and data and derive new theoretical results under weaker assumptions and a broader set of optimizers. Our extensive empirical investigation includes tens of thousands of models trained with all combinations of three optimizers, four parameterizations, several alignment assumptions, more than a dozen learning rates, and fourteen model sizes up to …

8. Close Not relevant

   From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph

   Junfeng Gong, Zhiyi Wei, Junying Chen et al. · 2025 · Machine Learning · AI summary available

   Abstract

   Despite significant evolution of CUDA programming and domain-specific libraries, effectively utilizing GPUs with massively parallel engines remains difficult. Large language models (LLMs) show strong potential in generating optimized CUDA code from sequential code. However, using LLMs in practice faces two major challenges: cloud-based APIs pose risks of code leakage, and local deployment is often computationally expensive and inefficient. These drawbacks have spurred interest in small language models (SLMs), which are more lightweight and privacy-friendly. Encouragingly, recent studies show that SLMs can achieve performance comparable to LLMs on specific tasks. While SLMs can match LLMs on …