Compute Trends Across Three Eras of Machine Learning
AI-generated Key Points
- Compute, data, and algorithmic advances are the three fundamental factors that guide progress in Machine Learning (ML).
- A recent study focused on trends in compute to better understand how it has evolved over time.
- Before 2010, training compute grew in line with Moore's law, doubling roughly every 20 months.
- Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated significantly, doubling approximately every 6 months.
- In late 2015, a new trend emerged as firms developed large-scale ML models with requirements for training compute that were 10 to 100 times larger than previous models.
- The history of compute in ML can be split into three eras: Pre-Deep Learning Era, Deep Learning Era and Large-Scale Era.
- There is a fast-growing demand for advanced ML systems that require more and more computing power.
- Large-scale models are a separate trend from traditional deep learning models due to their unique requirements.
- Possible causes for a potential slowdown in this trend are discussed in Appendix G.
- Record-setting models before and after September 2015 show no significant difference in trends.
- Paying attention to the most compute-intensive models overall is likely to advance the frontier of ML research.
- There is an increasing demand for even larger-scale ML models requiring massive amounts of training compute power beyond what was previously observed.
Authors: Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos
Abstract: Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.