Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Authors: Fei Xia, Aakanksha Chowdhery, Tianhe Yu, Jeffrey Zhao, Dian Yu, Alexander Pritzel, David Silver, Fan Yang, Basil Mustafa, Neil Houlsby, Sholto Douglas, Jacob Devlin, Anselm Levskaya, Kefan Xiao, Timothy Lillicrap, David Reitter, Behnam Neyshabur, Shuo-yiin Chang, Chung-Cheng Chiu, Yonghui Wu, Yuanzhong Xu, Mostafa Dehghani, Elena Buchatskaya, Chih-Kuan Yeh, Benjamin Lee, Yao Zhao, Mohammad Saleh, George Tucker, Julian Schrittwieser, Lev Proleev, Diana Mincu, Le Hou, Sebastian Borgeaud, Eliza Rutherford, Lisa Anne Hendricks, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Jack W. Rae, Pranav Shyam, Henryk Michalewski, Jacob Austin, Fabian Mentzer, Biao Zhang, Justin Gilmer, Dong Li, Jean-Baptiste Lespiau, Roman Ring, Loren Maggiore, Albin Cassirer, Michela Paganini, Kelvin Xu, Siamak Shakeri, Tianqi Liu, Ryan Burnell, Dmitry Lepikhin, Maxim Krikun, Orhan Firat, Xi Chen, Josip Djolonga, Mario Lučić, Marco Tagliasacchi, Christian Frank, Sebastian Riedel, Cheng Li, Yingjie Miao, Srivatsan Srinivasan, Tom Le Paine, Richard Powell, Petko Georgiev, Xuehan Xiong, Zhen Yang, Maja Trebacz, Mia Glaese, Will Hawkins, Harsh Mehta, Machel Reid, Disha Shrivastava, Kate Baumli, Michael Chang, Natalie Clay, Sheleem Kashem, Sarah York, Denis Teplyashin, Junhyuk Oh, Fangyu Liu, Rohan Anil, Melvin Johnson, Emanuel Taropa, Laurent El Shafey, Erica Moreira, Mark Omernick, Yujing Zhang, Junwhan Ahn, Paul Barham, Siddhartha Brahma, Yong Cheng, Ethan Dyer, Fangxiaoyu Feng, Lucas Gonzalez, Steven Hand, Michael Isard, Wenhao Jia, YaGuang Li, Hyeontaek Lim, Marcello Maggioni, Vedant Misra, Zachary Nado, Martin Polacek, Alex Polozov, Siyuan Qiao, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Ambrose Slone, Daniel Sohn, Simon Tokumine, Kiran Vodrahalli, Yunhan Xu, Qiao Zhang, Ce Zheng, Parker Schuh, Thanumalayan Sankaranarayana Pillai, Albert Webson, Jilin Chen, James Qin, Ankur Bapna, Nanxin Chen, Tara Sainath, Johan Schalkwyk, Arthur Guez, Elena Gribovskaya, Angeliki Lazaridou, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Yujia Li, Kareem Ayoub, Ioannis Antonoglou, Aditya Siddhant, Chenjie Gu, Andrew Brock, Ankesh Anand, Jaehoon Lee, Bernd Bohnet, Kevin Swersky, Antoine Yang, Antoine Miech, Anja Hauth, Yelin Kim, Dan Hurt, Andy Swing, Radu Soricut, James Molloy, Fabio Viola, Malcolm Reynolds, Eli Collins, Clemens Meyer, Megha Goel, Enrique Piqueras, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Misha Khalman, Alexandre Frechette, Charlotte Smith, James Lottes, Nathan Schucher, Alban Rrustemi, Phil Crone, Tomas Kocisky, Fred Alcober, Megan Barnes, Shantanu Thakoor, Rishabh Joshi, Rahma Chaabouni, Arun Ahuja, Ruibo Liu, Jeremy Chen, Jordan Grimstad, Ale Jakse Hartman, Martin Chadwick, Evan Senter, Michael Laskin, Lorenzo Blanco, Jenny Brennan, Sergey Brin, Gabriela Surita, Jane Labanowski, Stephanie Winkler, Emilio Parisotto, Kate Olszewska, Ravi Addanki, Geoff Brown, Nithya Attaluri, Zoe Ashwood, Anton Briukhov, Sanjay Ganapathy, Axel Stjerngren, Pedram Pejman, Juliette Love, Dawn Bloxwich, Peter Humphreys, Sina Samangooei, Richard Tanburn, Sarah Hodkinson, Minh Giang, Zaheer Abbas, Dominika Rogozińska, Vitaly Nikolaev, Lukas Zilka, Luheng He, Josh Newlan, Dawei Jia, Raoul de Liedekerke, Anirudh Baddepudi, Alex Goldin, Craig Swanson, Jessica Landon, Ruizhe Zhao, James Keeling, Boxi Wu, Salem Haykal, Abhanshu Sharma, Paul Komarek, Ross McIlroy, Michael Sharman, Paul Natsev, Paul Michel, Kris Cao, Christina Butterfield, Paul Kishan Rubenstein, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Priya Jhakra, Taylor Tobin, Nimesh Ghelani, Lora Aroyo, Jonas Adler, Mateo Wirth, Lisa Lee, Thais Kagohara, Sophie Bridgers, Anna Bortsova, Zafarali Ahmed, Vijay Bolina, Mariko Iinuma, James Besley, Da-Woon Chung, Ramona Comanescu, Xiance Si, Raphaël Lopez Kaufman, Mohamed Elhawaty, Helen Miller, Shereen Ashraf, Ada Ma, Rory Blevins, Oscar Chang, Carrie Muir, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Mingqiu Wang, Sophia Austin, Jiepu Jiang, Justin Chiu, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Vittorio Selo, Rhys May, Michael B. Chang, Adrià Recasens, Ben Caine, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Alex Tomala, Yunhao Tang, Elspeth White, Sharad Vikram, Sergi Caelles, Ross Hemsley, Gregory Thornton, Wojciech Stokowiec, Phoebe Thacker, James Svensson, Katerina Tsihlas, Arpi Vezer, Keran Rong, Mina Khan, Vladimir Feinberg, James Cobon-Kerr, Sayed Hadi Hashemi, Richard Ives, Nathan Byrd, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Matthew Tung, Hannah Sheahan, Sid Mittal, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Gamaleldin Elsayed, Romina Datta, Oskar Bunyan, Mahmoud Alnahlawi, Ivy Zheng, David Steiner, Michael Azzam, Abhijit Karmarkar, Johnson Jia, Zhenkai Zhu, Alberto Magni, Javier Snaider, Yana Kulizhskaya, Chimezie Iwuanyanwu, Priya Ponnapalli, Anitha Vijayakumar, Somer Greene, Colin Gaffney, Ken Franko, Christina Sorokin, Rohan Jain, James Lee-Thorp, Christopher Yew, Libin Bai, Alex Korchemniy, Nejc Trdin, Oliver Woodman, Rupert Kemp, Rishika Sinha, Nir Levine, Sonam Goenka, Robin Strudel, Balaji Lakshminarayanan, Mukarram Tariq, Yanhua Sun, Anmol Gulati, Nikhil Sethi, Joe Stanton, Mani Varadarajan, Brona Robenek, Dario de Cesare, Tom Hudson, Lexi Walker, Alex Morris, Ivo Penchev, Matthew Mauger, Alexey Guseynov, Seth Odoom, Anastasia Petrushkina, Chetan Tekur, Ruoxin Sang, Stephan Lee, Evgenii Eltyshev, Christof Angermueller, Maria Georgaki, Dinghua Li, Andrew Dai, Aishwarya Kamath, Jean-baptiste Alayrac, Stephen Spencer, Luke Vilnis, Nobuyuki Morioka, Peter Choy, Isabel Gao, Santiago Ontañón, Mario Pinto, Yifan Ding, Marvin Ritter, Guangda Lai, Bibo Xu, Antoine He, Skye Giordano, Danilo Martins, Sean Sechrist, Chris Larkin, Kevin Hui, David Barker, Justin Mao-Jones, Matthew Wiethoff, Beer Changpinyo, Paul Voigtlaender, Wiktor Gworek, Séb Arnold, Kashyap Krishnakumar, Elahe Dabir, Arnar Mar Hrafnkelsson, Ian Mackinnon, Josh Lipschultz, Mihaela Rosca, Cicero Nogueira dos Santos, Chulayuth Asawaroengchai, Carrie Grimes Bostock, Jonathan Caton, Vivek Sharma, Tim Blyth, Carlos Araya, Rakesh Shivanna, Ali Khodaei, Garrett Tanzer, Zhufeng Pan, Khuslen Baatarsukh, Xerxes Dotiwalla, Vincent Hellendoorn, Gabe Barth-Maron, Alek Andreev, Renshen Wang, Dave Lacey, Anastasija Ilić, Sadegh Jazayeri, Kingshuk Dasgupta, Shourya Sarcar, Tina Ornduff, Lucy Kim, Li Lao, Haibin Zhang, David Engel, Jessica Austin, Lilly Taylor, Dangyi Liu, Irene Cai, Harry Richardson, Bill Rosgen
Abstract: In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks across modalities, improves the state-of-the-art in long-document QA, long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 2.1 (200k) and GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Assess the quality of the AI-generated content by voting
Score: 0
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Some bits of the article are not summarized yet, you can re-run the summarizing process by clicking on the Run button below.
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.