Obscenely amount of data
-
How would you sort an obscenely amount of data? Say 500 gigabytes or more? Any brilliant ideas?
-
How would you sort an obscenely amount of data? Say 500 gigabytes or more? Any brilliant ideas?
Divide the data into files small enough to sort by normal operations, sort each file, and merge the sorted files. You could also import the data into a database engine and have it do the ordering.
-
Divide the data into files small enough to sort by normal operations, sort each file, and merge the sorted files. You could also import the data into a database engine and have it do the ordering.
-
Brady Kelly wrote:
You could also import the data into a database engine and have it do the ordering.
Which database engine you think could handle such huge datasets?
In SQL Server 2005 the number of rows per table is only constrained by storage space. and the maximum file sizes are 16 terabytes for the data file and 2 terabytes for the log file.
-
How would you sort an obscenely amount of data? Say 500 gigabytes or more? Any brilliant ideas?
Hi, I would try to (partially) sort the data as it is generated, not afterwards. If applicable. For instance for words, keep 26 or 26^n collections. :)
Luc Pattyn
try { [Search CP Articles] [Search CP Forums] [Forum Guidelines] [My Articles] } catch { [Google] }