Here is a comparison of two separate workflows and what they do.
You have a file on disk
The way that you would create this, if you were to do archiving and compression separately, would be:
Then you would do
Now, let’s decompress!
Way 1
You have
You decide to run:
You run:
The total data we WROTE to disk in this process was 2GB (for gunzip) + 2GB (for tar) + a few bytes for metadata = about 4GB.
Way 2
You have
You decide to run:
The total data we WROTE to disk in this process was 2GB of uncompressed data + a few bytes for metadata = about 2GB.
If you notice, the amount of disk I/O in Way 2 is identical to the disk I/O performed by, say, the
And if compression ratio is your concern, use the
You have a file on disk
blah.tar.gz which is, say, 1 GB of gzip-compressed data which, when uncompressed, occupies 2 GB (so a compression ratio of 50%).The way that you would create this, if you were to do archiving and compression separately, would be:
tar cf blah.tar files ...
This would result in blah.tar which is a mere aggregation of the files ... in uncompressed form.Then you would do
gzip blah.tar
This would read the contents of blah.tar from disk, compress them through the gzip compression algorithm, write the contents to blah.tar.gz, then unlink (delete) the file blah.tar.Now, let’s decompress!
Way 1
You have
blah.tar.gz, one way or another.You decide to run:
gunzip blah.tar.gz
This will- READ the 1GB compressed data contents of
blah.tar.gz. - PROCESS the compressed data through the
gzipdecompressor in memory. - As the memory buffer fills up with “a block” worth of data, WRITE the uncompressed data into the file
blah.taron disk and repeat until all the compressed data is read. - Unlink (delete) the file
blah.tar.gz.
blah.tar on disk, which is uncompressed
but contains one or more files within it, with very low data structure
overhead. The file size is probably a couple bytes larger than the sum of all the file data would be.You run:
tar xvf blah.tar
This will- READ the 2GB of uncompressed data contents of
blah.tarand thetarfile format’s data structures, including information about file permissions, file names, directories, etc. - WRITE to disk the 2GB of data plus the metadata. This involves: translating the data structure / metadata information into creating new files and directories on disk as appropriate, or rewriting existing files and directories with new data contents.
The total data we WROTE to disk in this process was 2GB (for gunzip) + 2GB (for tar) + a few bytes for metadata = about 4GB.
Way 2
You have
blah.tar.gz, one way or another.You decide to run:
tar xvzf blah.tar.gz
This will- READ the 1GB compressed data contents of
blah.tar.gz, a block at a time, into memory. - PROCESS the compressed data through the
gzipdecompressor in memory. - As the memory buffer fills up, it will pipe that data, in memory, through to the
tarfile format parser, which will read the information about metadata, etc. and the uncompressed file data. - As the memory buffer fills up in the
tarfile parser, it will WRITE the uncompressed data to disk, by creating files and directories and filling them up with the uncompressed contents.
The total data we WROTE to disk in this process was 2GB of uncompressed data + a few bytes for metadata = about 2GB.
If you notice, the amount of disk I/O in Way 2 is identical to the disk I/O performed by, say, the
Zip or7-Zip programs, adjusting for any differences in compression ratio.And if compression ratio is your concern, use the
Xz compressor to encapsulate tar, and you have LZMA2′ed TAR archive, which is just as efficient as the most advanced algorithm available to 7-Zip :-)
No comments:
Post a Comment