other-licenses/snappy/src/README@97036ab72558 (annotated)
other-licenses/snappy/src/README
Tue, 06 Jan 2015 21:39:09 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Tue, 06 Jan 2015 21:39:09 +0100
- branch
- TOR_BUG_9701
- changeset 8
- 97036ab72558
- permissions
- -rw-r--r--
Conditionally force memory storage according to privacy.thirdparty.isolate;
This solves Tor bug #9701, complying with disk avoidance documented in
https://www.torproject.org/projects/torbrowser/design/#disk-avoidance.
| michael@0 | 1 | Snappy, a fast compressor/decompressor. |
| michael@0 | 2 | |
| michael@0 | 3 | |
| michael@0 | 4 | Introduction |
| michael@0 | 5 | ============ |
| michael@0 | 6 | |
| michael@0 | 7 | Snappy is a compression/decompression library. It does not aim for maximum |
| michael@0 | 8 | compression, or compatibility with any other compression library; instead, |
| michael@0 | 9 | it aims for very high speeds and reasonable compression. For instance, |
| michael@0 | 10 | compared to the fastest mode of zlib, Snappy is an order of magnitude faster |
| michael@0 | 11 | for most inputs, but the resulting compressed files are anywhere from 20% to |
| michael@0 | 12 | 100% bigger. (For more information, see "Performance", below.) |
| michael@0 | 13 | |
| michael@0 | 14 | Snappy has the following properties: |
| michael@0 | 15 | |
| michael@0 | 16 | * Fast: Compression speeds at 250 MB/sec and beyond, with no assembler code. |
| michael@0 | 17 | See "Performance" below. |
| michael@0 | 18 | * Stable: Over the last few years, Snappy has compressed and decompressed |
| michael@0 | 19 | petabytes of data in Google's production environment. The Snappy bitstream |
| michael@0 | 20 | format is stable and will not change between versions. |
| michael@0 | 21 | * Robust: The Snappy decompressor is designed not to crash in the face of |
| michael@0 | 22 | corrupted or malicious input. |
| michael@0 | 23 | * Free and open source software: Snappy is licensed under a BSD-type license. |
| michael@0 | 24 | For more information, see the included COPYING file. |
| michael@0 | 25 | |
| michael@0 | 26 | Snappy has previously been called "Zippy" in some Google presentations |
| michael@0 | 27 | and the like. |
| michael@0 | 28 | |
| michael@0 | 29 | |
| michael@0 | 30 | Performance |
| michael@0 | 31 | =========== |
| michael@0 | 32 | |
| michael@0 | 33 | Snappy is intended to be fast. On a single core of a Core i7 processor |
| michael@0 | 34 | in 64-bit mode, it compresses at about 250 MB/sec or more and decompresses at |
| michael@0 | 35 | about 500 MB/sec or more. (These numbers are for the slowest inputs in our |
| michael@0 | 36 | benchmark suite; others are much faster.) In our tests, Snappy usually |
| michael@0 | 37 | is faster than algorithms in the same class (e.g. LZO, LZF, FastLZ, QuickLZ, |
| michael@0 | 38 | etc.) while achieving comparable compression ratios. |
| michael@0 | 39 | |
| michael@0 | 40 | Typical compression ratios (based on the benchmark suite) are about 1.5-1.7x |
| michael@0 | 41 | for plain text, about 2-4x for HTML, and of course 1.0x for JPEGs, PNGs and |
| michael@0 | 42 | other already-compressed data. Similar numbers for zlib in its fastest mode |
| michael@0 | 43 | are 2.6-2.8x, 3-7x and 1.0x, respectively. More sophisticated algorithms are |
| michael@0 | 44 | capable of achieving yet higher compression rates, although usually at the |
| michael@0 | 45 | expense of speed. Of course, compression ratio will vary significantly with |
| michael@0 | 46 | the input. |
| michael@0 | 47 | |
| michael@0 | 48 | Although Snappy should be fairly portable, it is primarily optimized |
| michael@0 | 49 | for 64-bit x86-compatible processors, and may run slower in other environments. |
| michael@0 | 50 | In particular: |
| michael@0 | 51 | |
| michael@0 | 52 | - Snappy uses 64-bit operations in several places to process more data at |
| michael@0 | 53 | once than would otherwise be possible. |
| michael@0 | 54 | - Snappy assumes unaligned 32- and 64-bit loads and stores are cheap. |
| michael@0 | 55 | On some platforms, these must be emulated with single-byte loads |
| michael@0 | 56 | and stores, which is much slower. |
| michael@0 | 57 | - Snappy assumes little-endian throughout, and needs to byte-swap data in |
| michael@0 | 58 | several places if running on a big-endian platform. |
| michael@0 | 59 | |
| michael@0 | 60 | Experience has shown that even heavily tuned code can be improved. |
| michael@0 | 61 | Performance optimizations, whether for 64-bit x86 or other platforms, |
| michael@0 | 62 | are of course most welcome; see "Contact", below. |
| michael@0 | 63 | |
| michael@0 | 64 | |
| michael@0 | 65 | Usage |
| michael@0 | 66 | ===== |
| michael@0 | 67 | |
| michael@0 | 68 | Note that Snappy, both the implementation and the main interface, |
| michael@0 | 69 | is written in C++. However, several third-party bindings to other languages |
| michael@0 | 70 | are available; see the Google Code page at http://code.google.com/p/snappy/ |
| michael@0 | 71 | for more information. Also, if you want to use Snappy from C code, you can |
| michael@0 | 72 | use the included C bindings in snappy-c.h. |
| michael@0 | 73 | |
| michael@0 | 74 | To use Snappy from your own C++ program, include the file "snappy.h" from |
| michael@0 | 75 | your calling file, and link against the compiled library. |
| michael@0 | 76 | |
| michael@0 | 77 | There are many ways to call Snappy, but the simplest possible is |
| michael@0 | 78 | |
| michael@0 | 79 | snappy::Compress(input.data(), input.size(), &output); |
| michael@0 | 80 | |
| michael@0 | 81 | and similarly |
| michael@0 | 82 | |
| michael@0 | 83 | snappy::Uncompress(input.data(), input.size(), &output); |
| michael@0 | 84 | |
| michael@0 | 85 | where "input" and "output" are both instances of std::string. |
| michael@0 | 86 | |
| michael@0 | 87 | There are other interfaces that are more flexible in various ways, including |
| michael@0 | 88 | support for custom (non-array) input sources. See the header file for more |
| michael@0 | 89 | information. |
| michael@0 | 90 | |
| michael@0 | 91 | |
| michael@0 | 92 | Tests and benchmarks |
| michael@0 | 93 | ==================== |
| michael@0 | 94 | |
| michael@0 | 95 | When you compile Snappy, snappy_unittest is compiled in addition to the |
| michael@0 | 96 | library itself. You do not need it to use the compressor from your own library, |
| michael@0 | 97 | but it contains several useful components for Snappy development. |
| michael@0 | 98 | |
| michael@0 | 99 | First of all, it contains unit tests, verifying correctness on your machine in |
| michael@0 | 100 | various scenarios. If you want to change or optimize Snappy, please run the |
| michael@0 | 101 | tests to verify you have not broken anything. Note that if you have the |
| michael@0 | 102 | Google Test library installed, unit test behavior (especially failures) will be |
| michael@0 | 103 | significantly more user-friendly. You can find Google Test at |
| michael@0 | 104 | |
| michael@0 | 105 | http://code.google.com/p/googletest/ |
| michael@0 | 106 | |
| michael@0 | 107 | You probably also want the gflags library for handling of command-line flags; |
| michael@0 | 108 | you can find it at |
| michael@0 | 109 | |
| michael@0 | 110 | http://code.google.com/p/google-gflags/ |
| michael@0 | 111 | |
| michael@0 | 112 | In addition to the unit tests, snappy contains microbenchmarks used to |
| michael@0 | 113 | tune compression and decompression performance. These are automatically run |
| michael@0 | 114 | before the unit tests, but you can disable them using the flag |
| michael@0 | 115 | --run_microbenchmarks=false if you have gflags installed (otherwise you will |
| michael@0 | 116 | need to edit the source). |
| michael@0 | 117 | |
| michael@0 | 118 | Finally, snappy can benchmark Snappy against a few other compression libraries |
| michael@0 | 119 | (zlib, LZO, LZF, FastLZ and QuickLZ), if they were detected at configure time. |
| michael@0 | 120 | To benchmark using a given file, give the compression algorithm you want to test |
| michael@0 | 121 | Snappy against (e.g. --zlib) and then a list of one or more file names on the |
| michael@0 | 122 | command line. The testdata/ directory contains the files used by the |
| michael@0 | 123 | microbenchmark, which should provide a reasonably balanced starting point for |
| michael@0 | 124 | benchmarking. (Note that baddata[1-3].snappy are not intended as benchmarks; they |
| michael@0 | 125 | are used to verify correctness in the presence of corrupted data in the unit |
| michael@0 | 126 | test.) |
| michael@0 | 127 | |
| michael@0 | 128 | |
| michael@0 | 129 | Contact |
| michael@0 | 130 | ======= |
| michael@0 | 131 | |
| michael@0 | 132 | Snappy is distributed through Google Code. For the latest version, a bug tracker, |
| michael@0 | 133 | and other information, see |
| michael@0 | 134 | |
| michael@0 | 135 | http://code.google.com/p/snappy/ |
