Metadata Caching in Presto: Towards Fast Data Processing

Presto is an open-source distributed SQL query engine that aims to provide "SQL on everything" for OLAP (Online Analytical Processing). Since its release in 2013, Presto has gained significant popularity in the field of large-scale data analytics and has been adopted by numerous enterprises. However, during the development and operation of Presto, it was observed that parsing column-oriented data files in Presto worker nodes consumed a considerable amount of CPU resources. This issue hindered companies from scaling up their analytical data volumes. To address this problem, the authors propose a solution in the form of a metadata caching layer. This caching layer is built on top of the Alluxio SDK cache and is integrated into each Presto worker node. Its purpose is to cache intermediate results during file parsing. The metadata cache offers two methods of caching: caching decompressed metadata bytes from raw data files and caching deserialized metadata objects. In order to evaluate the effectiveness of their approach, the authors conducted experiments using the TPC-DS benchmark on Presto. The results demonstrate that when the cache is warm (i.e., already populated with relevant data), the first method can reduce CPU consumption during queries by 10%-20%. On the other hand, utilizing the second method can minimize CPU usage by an even greater extent, achieving reductions of 20%-40%. Overall, this paper presents a valuable contribution to improving data processing efficiency in Presto through the implementation of a metadata caching layer. By reducing CPU consumption during file parsing, this approach enables companies to handle larger volumes of analytical data effectively.

Improving Data Processing Efficiency in Presto with a Metadata Caching Layer

Presto is an open-source distributed SQL query engine that enables users to perform “SQL on everything” for online analytical processing (OLAP). Since its release in 2013, Presto has become increasingly popular among enterprises for large-scale data analytics. However, during the development and operation of Presto, it was observed that parsing column-oriented data files in Presto worker nodes consumed a considerable amount of CPU resources. This issue hindered companies from scaling up their analytical data volumes. To address this problem, the authors propose a solution in the form of a metadata caching layer.

The Metadata Caching Layer

The metadata caching layer is built on top of the Alluxio SDK cache and integrated into each Presto worker node. Its purpose is to cache intermediate results during file parsing. The metadata cache offers two methods of caching:

• Caching decompressed metadata bytes from raw data files
• Caching deserialized metadata objects.

Evaluation Results

In order to evaluate the effectiveness of their approach, the authors conducted experiments using the TPC-DS benchmark on Presto. The results demonstrate that when the cache is warm (i.e., already populated with relevant data), utilizing either method can reduce CPU consumption during queries by 10%-20%. On one hand, employing the first method can minimize CPU usage by 10%-20%. On the other hand, utilizing the second method can achieve reductions of 20%-40%.

Conclusion

Overall, this paper presents a valuable contribution to improving data processing efficiency in Presto through implementation of a metadata caching layer. By reducing CPU consumption during file parsing, this approach enables companies to handle larger volumes of analytical data effectively and efficiently scale up their operations without any hindrance due to resource constraints.