Item

Abstract

High performance applications involving large data sets require the efficient and flexible use of multiple disks. In an external memory machine with $D$ parallel, independent disks, only one block can be accessed on each disk in one I/O step. This restriction leads to a load balancing problem that is perhaps the main inhibitor for adapting single-disk external memory algorithms to multiple disks. This paper shows that this problem can be solved efficiently using a combination of randomized placement, redundancy and an optimal scheduling algorithm. A buffer of $\Ohh{D}$ blocks suffices to support efficient writing of arbitrary blocks if blocks are distributed uniformly at random to the disks (e.g., by hashing). If two randomly allocated copies of each block exist, $N$ arbitrary blocks can be read within $\ceil{N/D}+1$ I/O steps with high probability. In addition, the redundancy can be reduced from $2$ to $1+1/r$ for any integer $r$. %Using appropriate codes even multiple disk %failures can be tolerated with low redundancy. These results can be used to emulate the simple and powerful ``single-disk multi-head'' model of external computing \cite{AggVit88} on the physically more realistic independent disk model \cite{VitShr94} with small constant overhead. This is faster than a lower bound for deterministic emulation \cite{Arm96}.