You are here

Design and Evaluation of Parallel Hashing over Large-scale Data


Long Cheng, Spyros Kotoulas, Tomas Ward

Publication Type: 
Refereed Conference Meeting Proceeding
High-performance analytical data processing sys- tems often run on servers with large amounts of memory. A common data structure used in such environment is the hash tables. This paper focuses on investigating efficient parallel hash algorithms for processing large-scale data. Currently, hash tables on distributed architectures are accessed one key at a time by local or remote threads while shared-memory ap- proaches focus on accessing a single table with multiple threads. A relatively straightforward “bulk-operation” approach seems to have been neglected by researchers. In this work, using such a method, we propose a high-level parallel hashing framework, Structured Parallel Hashing, targeting efficiently processing massive data on distributed memory. We present a theoretical analysis of the proposed method and describe the design of our hashing implementations. The evaluation reveals a very interesting result - the proposed straightforward method can vastly outperform distributed hashing methods and can even offer performance comparable with approaches based on shared memory supercomputers which use specialized hardware predicates. Moreover, we char- acterize the performance of our hash implementations through extensive experiments, thereby allowing system developers to make a more informed choice for their high-performance applications.
Conference Name: 
Proceedings of the 21st IEEE International Conference on High Performance Computing, HiPC
Proceedings of the 21st IEEE International Conference on High Performance Computing, HiPC
Digital Object Identifer (DOI): 
Publication Date: 
Conference Location: 
United States of America
Research Group: 
Open access repository: