Compute Trends Across Three Eras of Machine Learning

AI-generated keywords: Compute Data Algorithmic Machine Learning Moore's Law

AI-generated Key Points

  • Compute, data, and algorithmic advances are the three fundamental factors that guide progress in Machine Learning (ML).
  • A recent study focused on trends in compute to better understand how it has evolved over time.
  • Before 2010, training compute grew in line with Moore's law, doubling roughly every 20 months.
  • Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated significantly, doubling approximately every 6 months.
  • In late 2015, a new trend emerged as firms developed large-scale ML models with requirements for training compute that were 10 to 100 times larger than previous models.
  • The history of compute in ML can be split into three eras: Pre-Deep Learning Era, Deep Learning Era and Large-Scale Era.
  • There is a fast-growing demand for advanced ML systems that require more and more computing power.
  • Large-scale models are a separate trend from traditional deep learning models due to their unique requirements.
  • Possible causes for a potential slowdown in this trend are discussed in Appendix G.
  • Record-setting models before and after September 2015 show no significant difference in trends.
  • Paying attention to the most compute-intensive models overall is likely to advance the frontier of ML research.
  • There is an increasing demand for even larger-scale ML models requiring massive amounts of training compute power beyond what was previously observed.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos

License: CC BY 4.0

Abstract: Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

Submitted to arXiv on 11 Feb. 2022

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2202.05924v1

In the field of Machine Learning (ML), compute, data, and algorithmic advances are the three fundamental factors that guide progress. A recent study focused on trends in the most readily quantified factor - compute - to better understand how it has evolved over time. The study revealed that before 2010, training compute grew in line with Moore's law, doubling roughly every 20 months. However, since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated significantly, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with requirements for training compute that were 10 to 100 times larger than previous models. This observation led researchers to split the history of compute in ML into three eras: the Pre-Deep Learning Era, the Deep Learning Era and the Large-Scale Era. The study highlights that there is a fast-growing demand for advanced ML systems that require more and more computing power. Additionally, it suggests that large-scale models are a separate trend from traditional deep learning models due to their unique requirements. Further analysis conducted by researchers found some possible causes for a potential slowdown in this trend which are discussed in Appendix G. They also showed that if we look only at record-setting models before and after September 2015, there is no significant difference in trends. The study emphasizes paying attention to the most compute-intensive models overall as they are likely to advance the frontier of ML research. Researchers looked at trends in record-setting models and found results consistent with those presented earlier. Finally, our data suggests that around present times there is an increasing demand for even larger-scale ML models requiring massive amounts of training compute power beyond what was previously observed. This highlights an ongoing need for continued advancements in computing technology to support future developments in Machine Learning research.
Created on 17 Apr. 2023

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.