The Tor Browser / file comparison

comparison: other-licenses/7zstub/src/DOC/lzma.txt

other-licenses/7zstub/src/DOC/lzma.txt

branch
TOR_BUG_9701
changeset 14
925c144e1f1f
equal deleted inserted replaced
-1:000000000000 0:295eac11a018
1 LZMA SDK 4.40
2 -------------
3
4 LZMA SDK Copyright (C) 1999-2006 Igor Pavlov
5
6 LZMA SDK provides the documentation, samples, header files, libraries,
7 and tools you need to develop applications that use LZMA compression.
8
9 LZMA is default and general compression method of 7z format
10 in 7-Zip compression program (www.7-zip.org). LZMA provides high
11 compression ratio and very fast decompression.
12
13 LZMA is an improved version of famous LZ77 compression algorithm.
14 It was improved in way of maximum increasing of compression ratio,
15 keeping high decompression speed and low memory requirements for
16 decompressing.
17
18
19
20 LICENSE
21 -------
22
23 LZMA SDK is available under any of the following licenses:
24
25 1) GNU Lesser General Public License (GNU LGPL)
26 2) Common Public License (CPL)
27 3) Simplified license for unmodified code (read SPECIAL EXCEPTION)
28 4) Proprietary license
29
30 It means that you can select one of these four options and follow rules of that license.
31
32
33 1,2) GNU LGPL and CPL licenses are pretty similar and both these
34 licenses are classified as
35 - "Free software licenses" at http://www.gnu.org/
36 - "OSI-approved" at http://www.opensource.org/
37
38
39 3) SPECIAL EXCEPTION
40
41 Igor Pavlov, as the author of this code, expressly permits you
42 to statically or dynamically link your code (or bind by name)
43 to the files from LZMA SDK without subjecting your linked
44 code to the terms of the CPL or GNU LGPL.
45 Any modifications or additions to files from LZMA SDK, however,
46 are subject to the GNU LGPL or CPL terms.
47
48 SPECIAL EXCEPTION allows you to use LZMA SDK in applications with closed code,
49 while you keep LZMA SDK code unmodified.
50
51
52 SPECIAL EXCEPTION #2: Igor Pavlov, as the author of this code, expressly permits
53 you to use this code under the same terms and conditions contained in the License
54 Agreement you have for any previous version of LZMA SDK developed by Igor Pavlov.
55
56 SPECIAL EXCEPTION #2 allows owners of proprietary licenses to use latest version
57 of LZMA SDK as update for previous versions.
58
59
60 SPECIAL EXCEPTION #3: Igor Pavlov, as the author of this code, expressly permits
61 you to use code of the following files:
62 BranchTypes.h, LzmaTypes.h, LzmaTest.c, LzmaStateTest.c, LzmaAlone.cpp,
63 LzmaAlone.cs, LzmaAlone.java
64 as public domain code.
65
66
67 4) Proprietary license
68
69 LZMA SDK also can be available under a proprietary license which
70 can include:
71
72 1) Right to modify code without subjecting modified code to the
73 terms of the CPL or GNU LGPL
74 2) Technical support for code
75
76 To request such proprietary license or any additional consultations,
77 send email message from that page:
78 http://www.7-zip.org/support.html
79
80
81 You should have received a copy of the GNU Lesser General Public
82 License along with this library; if not, write to the Free Software
83 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
84
85 You should have received a copy of the Common Public License
86 along with this library.
87
88
89 LZMA SDK Contents
90 -----------------
91
92 LZMA SDK includes:
93
94 - C++ source code of LZMA compressing and decompressing
95 - ANSI-C compatible source code for LZMA decompressing
96 - C# source code for LZMA compressing and decompressing
97 - Java source code for LZMA compressing and decompressing
98 - Compiled file->file LZMA compressing/decompressing program for Windows system
99
100 ANSI-C LZMA decompression code was ported from original C++ sources to C.
101 Also it was simplified and optimized for code size.
102 But it is fully compatible with LZMA from 7-Zip.
103
104
105 UNIX/Linux version
106 ------------------
107 To compile C++ version of file->file LZMA, go to directory
108 C/7zip/Compress/LZMA_Alone
109 and type "make" or "make clean all" to recompile all.
110
111 In some UNIX/Linux versions you must compile LZMA with static libraries.
112 To compile with static libraries, change string in makefile
113 LIB = -lm
114 to string
115 LIB = -lm -static
116
117
118 Files
119 ---------------------
120 C - C / CPP source code
121 CS - C# source code
122 Java - Java source code
123 lzma.txt - LZMA SDK description (this file)
124 7zFormat.txt - 7z Format description
125 7zC.txt - 7z ANSI-C Decoder description (this file)
126 methods.txt - Compression method IDs for .7z
127 LGPL.txt - GNU Lesser General Public License
128 CPL.html - Common Public License
129 lzma.exe - Compiled file->file LZMA encoder/decoder for Windows
130 history.txt - history of the LZMA SDK
131
132
133 Source code structure
134 ---------------------
135
136 C - C / CPP files
137 Common - common files for C++ projects
138 Windows - common files for Windows related code
139 7zip - files related to 7-Zip Project
140 Common - common files for 7-Zip
141 Compress - files related to compression/decompression
142 LZ - files related to LZ (Lempel-Ziv) compression algorithm
143 BinTree - Binary Tree Match Finder for LZ algorithm
144 HashChain - Hash Chain Match Finder for LZ algorithm
145 Patricia - Patricia Match Finder for LZ algorithm
146 RangeCoder - Range Coder (special code of compression/decompression)
147 LZMA - LZMA compression/decompression on C++
148 LZMA_Alone - file->file LZMA compression/decompression
149 LZMA_C - ANSI-C compatible LZMA decompressor
150 LzmaDecode.h - interface for LZMA decoding on ANSI-C
151 LzmaDecode.c - LZMA decoding on ANSI-C (new fastest version)
152 LzmaDecodeSize.c - LZMA decoding on ANSI-C (old size-optimized version)
153 LzmaTest.c - test application that decodes LZMA encoded file
154 LzmaTypes.h - basic types for LZMA Decoder
155 LzmaStateDecode.h - interface for LZMA decoding (State version)
156 LzmaStateDecode.c - LZMA decoding on ANSI-C (State version)
157 LzmaStateTest.c - test application (State version)
158 Branch - Filters for x86, IA-64, ARM, ARM-Thumb, PowerPC and SPARC code
159 Archive - files related to archiving
160 7z_C - 7z ANSI-C Decoder
161
162 CS - C# files
163 7zip
164 Common - some common files for 7-Zip
165 Compress - files related to compression/decompression
166 LZ - files related to LZ (Lempel-Ziv) compression algorithm
167 LZMA - LZMA compression/decompression
168 LzmaAlone - file->file LZMA compression/decompression
169 RangeCoder - Range Coder (special code of compression/decompression)
170
171 Java - Java files
172 SevenZip
173 Compression - files related to compression/decompression
174 LZ - files related to LZ (Lempel-Ziv) compression algorithm
175 LZMA - LZMA compression/decompression
176 RangeCoder - Range Coder (special code of compression/decompression)
177
178 C/C++ source code of LZMA SDK is part of 7-Zip project.
179
180 You can find ANSI-C LZMA decompressing code at folder
181 C/7zip/Compress/LZMA_C
182 7-Zip doesn't use that ANSI-C LZMA code and that code was developed
183 specially for this SDK. And files from LZMA_C do not need files from
184 other directories of SDK for compiling.
185
186 7-Zip source code can be downloaded from 7-Zip's SourceForge page:
187
188 http://sourceforge.net/projects/sevenzip/
189
190
191 LZMA features
192 -------------
193 - Variable dictionary size (up to 1 GB)
194 - Estimated compressing speed: about 1 MB/s on 1 GHz CPU
195 - Estimated decompressing speed:
196 - 8-12 MB/s on 1 GHz Intel Pentium 3 or AMD Athlon
197 - 500-1000 KB/s on 100 MHz ARM, MIPS, PowerPC or other simple RISC
198 - Small memory requirements for decompressing (8-32 KB + DictionarySize)
199 - Small code size for decompressing: 2-8 KB (depending from
200 speed optimizations)
201
202 LZMA decoder uses only integer operations and can be
203 implemented in any modern 32-bit CPU (or on 16-bit CPU with some conditions).
204
205 Some critical operations that affect to speed of LZMA decompression:
206 1) 32*16 bit integer multiply
207 2) Misspredicted branches (penalty mostly depends from pipeline length)
208 3) 32-bit shift and arithmetic operations
209
210 Speed of LZMA decompressing mostly depends from CPU speed.
211 Memory speed has no big meaning. But if your CPU has small data cache,
212 overall weight of memory speed will slightly increase.
213
214
215 How To Use
216 ----------
217
218 Using LZMA encoder/decoder executable
219 --------------------------------------
220
221 Usage: LZMA <e|d> inputFile outputFile [<switches>...]
222
223 e: encode file
224
225 d: decode file
226
227 b: Benchmark. There are two tests: compressing and decompressing
228 with LZMA method. Benchmark shows rating in MIPS (million
229 instructions per second). Rating value is calculated from
230 measured speed and it is normalized with AMD Athlon 64 X2 CPU
231 results. Also Benchmark checks possible hardware errors (RAM
232 errors in most cases). Benchmark uses these settings:
233 (-a1, -d21, -fb32, -mfbt4). You can change only -d. Also you
234 can change number of iterations. Example for 30 iterations:
235 LZMA b 30
236 Default number of iterations is 10.
237
238 <Switches>
239
240
241 -a{N}: set compression mode 0 = fast, 1 = normal
242 default: 1 (normal)
243
244 d{N}: Sets Dictionary size - [0, 30], default: 23 (8MB)
245 The maximum value for dictionary size is 1 GB = 2^30 bytes.
246 Dictionary size is calculated as DictionarySize = 2^N bytes.
247 For decompressing file compressed by LZMA method with dictionary
248 size D = 2^N you need about D bytes of memory (RAM).
249
250 -fb{N}: set number of fast bytes - [5, 273], default: 128
251 Usually big number gives a little bit better compression ratio
252 and slower compression process.
253
254 -lc{N}: set number of literal context bits - [0, 8], default: 3
255 Sometimes lc=4 gives gain for big files.
256
257 -lp{N}: set number of literal pos bits - [0, 4], default: 0
258 lp switch is intended for periodical data when period is
259 equal 2^N. For example, for 32-bit (4 bytes)
260 periodical data you can use lp=2. Often it's better to set lc0,
261 if you change lp switch.
262
263 -pb{N}: set number of pos bits - [0, 4], default: 2
264 pb switch is intended for periodical data
265 when period is equal 2^N.
266
267 -mf{MF_ID}: set Match Finder. Default: bt4.
268 Algorithms from hc* group doesn't provide good compression
269 ratio, but they often works pretty fast in combination with
270 fast mode (-a0).
271
272 Memory requirements depend from dictionary size
273 (parameter "d" in table below).
274
275 MF_ID Memory Description
276
277 bt2 d * 9.5 + 4MB Binary Tree with 2 bytes hashing.
278 bt3 d * 11.5 + 4MB Binary Tree with 3 bytes hashing.
279 bt4 d * 11.5 + 4MB Binary Tree with 4 bytes hashing.
280 hc4 d * 7.5 + 4MB Hash Chain with 4 bytes hashing.
281
282 -eos: write End Of Stream marker. By default LZMA doesn't write
283 eos marker, since LZMA decoder knows uncompressed size
284 stored in .lzma file header.
285
286 -si: Read data from stdin (it will write End Of Stream marker).
287 -so: Write data to stdout
288
289
290 Examples:
291
292 1) LZMA e file.bin file.lzma -d16 -lc0
293
294 compresses file.bin to file.lzma with 64 KB dictionary (2^16=64K)
295 and 0 literal context bits. -lc0 allows to reduce memory requirements
296 for decompression.
297
298
299 2) LZMA e file.bin file.lzma -lc0 -lp2
300
301 compresses file.bin to file.lzma with settings suitable
302 for 32-bit periodical data (for example, ARM or MIPS code).
303
304 3) LZMA d file.lzma file.bin
305
306 decompresses file.lzma to file.bin.
307
308
309 Compression ratio hints
310 -----------------------
311
312 Recommendations
313 ---------------
314
315 To increase compression ratio for LZMA compressing it's desirable
316 to have aligned data (if it's possible) and also it's desirable to locate
317 data in such order, where code is grouped in one place and data is
318 grouped in other place (it's better than such mixing: code, data, code,
319 data, ...).
320
321
322 Using Filters
323 -------------
324 You can increase compression ratio for some data types, using
325 special filters before compressing. For example, it's possible to
326 increase compression ratio on 5-10% for code for those CPU ISAs:
327 x86, IA-64, ARM, ARM-Thumb, PowerPC, SPARC.
328
329 You can find C/C++ source code of such filters in folder "7zip/Compress/Branch"
330
331 You can check compression ratio gain of these filters with such
332 7-Zip commands (example for ARM code):
333 No filter:
334 7z a a1.7z a.bin -m0=lzma
335
336 With filter for little-endian ARM code:
337 7z a a2.7z a.bin -m0=bc_arm -m1=lzma
338
339 With filter for big-endian ARM code (using additional Swap4 filter):
340 7z a a3.7z a.bin -m0=swap4 -m1=bc_arm -m2=lzma
341
342 It works in such manner:
343 Compressing = Filter_encoding + LZMA_encoding
344 Decompressing = LZMA_decoding + Filter_decoding
345
346 Compressing and decompressing speed of such filters is very high,
347 so it will not increase decompressing time too much.
348 Moreover, it reduces decompression time for LZMA_decoding,
349 since compression ratio with filtering is higher.
350
351 These filters convert CALL (calling procedure) instructions
352 from relative offsets to absolute addresses, so such data becomes more
353 compressible. Source code of these CALL filters is pretty simple
354 (about 20 lines of C++), so you can convert it from C++ version yourself.
355
356 For some ISAs (for example, for MIPS) it's impossible to get gain from such filter.
357
358
359 LZMA compressed file format
360 ---------------------------
361 Offset Size Description
362 0 1 Special LZMA properties for compressed data
363 1 4 Dictionary size (little endian)
364 5 8 Uncompressed size (little endian). -1 means unknown size
365 13 Compressed data
366
367
368 ANSI-C LZMA Decoder
369 ~~~~~~~~~~~~~~~~~~~
370
371 To compile ANSI-C LZMA Decoder you can use one of the following files sets:
372 1) LzmaDecode.h + LzmaDecode.c + LzmaTest.c (fastest version)
373 2) LzmaDecode.h + LzmaDecodeSize.c + LzmaTest.c (old size-optimized version)
374 3) LzmaStateDecode.h + LzmaStateDecode.c + LzmaStateTest.c (zlib-like interface)
375
376
377 Memory requirements for LZMA decoding
378 -------------------------------------
379
380 LZMA decoder doesn't allocate memory itself, so you must
381 allocate memory and send it to LZMA.
382
383 Stack usage of LZMA decoding function for local variables is not
384 larger than 200 bytes.
385
386 How To decompress data
387 ----------------------
388
389 LZMA Decoder (ANSI-C version) now supports 5 interfaces:
390 1) Single-call Decompressing
391 2) Single-call Decompressing with input stream callback
392 3) Multi-call Decompressing with output buffer
393 4) Multi-call Decompressing with input callback and output buffer
394 5) Multi-call State Decompressing (zlib-like interface)
395
396 Variant-5 is similar to Variant-4, but Variant-5 doesn't use callback functions.
397
398 Decompressing steps
399 -------------------
400
401 1) read LZMA properties (5 bytes):
402 unsigned char properties[LZMA_PROPERTIES_SIZE];
403
404 2) read uncompressed size (8 bytes, little-endian)
405
406 3) Decode properties:
407
408 CLzmaDecoderState state; /* it's 24-140 bytes structure, if int is 32-bit */
409
410 if (LzmaDecodeProperties(&state.Properties, properties, LZMA_PROPERTIES_SIZE) != LZMA_RESULT_OK)
411 return PrintError(rs, "Incorrect stream properties");
412
413 4) Allocate memory block for internal Structures:
414
415 state.Probs = (CProb *)malloc(LzmaGetNumProbs(&state.Properties) * sizeof(CProb));
416 if (state.Probs == 0)
417 return PrintError(rs, kCantAllocateMessage);
418
419 LZMA decoder uses array of CProb variables as internal structure.
420 By default, CProb is unsigned_short. But you can define _LZMA_PROB32 to make
421 it unsigned_int. It can increase speed on some 32-bit CPUs, but memory
422 usage will be doubled in that case.
423
424
425 5) Main Decompressing
426
427 You must use one of the following interfaces:
428
429 5.1 Single-call Decompressing
430 -----------------------------
431 When to use: RAM->RAM decompressing
432 Compile files: LzmaDecode.h, LzmaDecode.c
433 Compile defines: no defines
434 Memory Requirements:
435 - Input buffer: compressed size
436 - Output buffer: uncompressed size
437 - LZMA Internal Structures (~16 KB for default settings)
438
439 Interface:
440 int res = LzmaDecode(&state,
441 inStream, compressedSize, &inProcessed,
442 outStream, outSize, &outProcessed);
443
444
445 5.2 Single-call Decompressing with input stream callback
446 --------------------------------------------------------
447 When to use: File->RAM or Flash->RAM decompressing.
448 Compile files: LzmaDecode.h, LzmaDecode.c
449 Compile defines: _LZMA_IN_CB
450 Memory Requirements:
451 - Buffer for input stream: any size (for example, 16 KB)
452 - Output buffer: uncompressed size
453 - LZMA Internal Structures (~16 KB for default settings)
454
455 Interface:
456 typedef struct _CBuffer
457 {
458 ILzmaInCallback InCallback;
459 FILE *File;
460 unsigned char Buffer[kInBufferSize];
461 } CBuffer;
462
463 int LzmaReadCompressed(void *object, const unsigned char **buffer, SizeT *size)
464 {
465 CBuffer *bo = (CBuffer *)object;
466 *buffer = bo->Buffer;
467 *size = MyReadFile(bo->File, bo->Buffer, kInBufferSize);
468 return LZMA_RESULT_OK;
469 }
470
471 CBuffer g_InBuffer;
472
473 g_InBuffer.File = inFile;
474 g_InBuffer.InCallback.Read = LzmaReadCompressed;
475 int res = LzmaDecode(&state,
476 &g_InBuffer.InCallback,
477 outStream, outSize, &outProcessed);
478
479
480 5.3 Multi-call decompressing with output buffer
481 -----------------------------------------------
482 When to use: RAM->File decompressing
483 Compile files: LzmaDecode.h, LzmaDecode.c
484 Compile defines: _LZMA_OUT_READ
485 Memory Requirements:
486 - Input buffer: compressed size
487 - Buffer for output stream: any size (for example, 16 KB)
488 - LZMA Internal Structures (~16 KB for default settings)
489 - LZMA dictionary (dictionary size is encoded in stream properties)
490
491 Interface:
492
493 state.Dictionary = (unsigned char *)malloc(state.Properties.DictionarySize);
494
495 LzmaDecoderInit(&state);
496 do
497 {
498 LzmaDecode(&state,
499 inBuffer, inAvail, &inProcessed,
500 g_OutBuffer, outAvail, &outProcessed);
501 inAvail -= inProcessed;
502 inBuffer += inProcessed;
503 }
504 while you need more bytes
505
506 see LzmaTest.c for more details.
507
508
509 5.4 Multi-call decompressing with input callback and output buffer
510 ------------------------------------------------------------------
511 When to use: File->File decompressing
512 Compile files: LzmaDecode.h, LzmaDecode.c
513 Compile defines: _LZMA_IN_CB, _LZMA_OUT_READ
514 Memory Requirements:
515 - Buffer for input stream: any size (for example, 16 KB)
516 - Buffer for output stream: any size (for example, 16 KB)
517 - LZMA Internal Structures (~16 KB for default settings)
518 - LZMA dictionary (dictionary size is encoded in stream properties)
519
520 Interface:
521
522 state.Dictionary = (unsigned char *)malloc(state.Properties.DictionarySize);
523
524 LzmaDecoderInit(&state);
525 do
526 {
527 LzmaDecode(&state,
528 &bo.InCallback,
529 g_OutBuffer, outAvail, &outProcessed);
530 }
531 while you need more bytes
532
533 see LzmaTest.c for more details:
534
535
536 5.5 Multi-call State Decompressing (zlib-like interface)
537 ------------------------------------------------------------------
538 When to use: file->file decompressing
539 Compile files: LzmaStateDecode.h, LzmaStateDecode.c
540 Compile defines:
541 Memory Requirements:
542 - Buffer for input stream: any size (for example, 16 KB)
543 - Buffer for output stream: any size (for example, 16 KB)
544 - LZMA Internal Structures (~16 KB for default settings)
545 - LZMA dictionary (dictionary size is encoded in stream properties)
546
547 Interface:
548
549 state.Dictionary = (unsigned char *)malloc(state.Properties.DictionarySize);
550
551
552 LzmaDecoderInit(&state);
553 do
554 {
555 res = LzmaDecode(&state,
556 inBuffer, inAvail, &inProcessed,
557 g_OutBuffer, outAvail, &outProcessed,
558 finishDecoding);
559 inAvail -= inProcessed;
560 inBuffer += inProcessed;
561 }
562 while you need more bytes
563
564 see LzmaStateTest.c for more details:
565
566
567 6) Free all allocated blocks
568
569
570 Note
571 ----
572 LzmaDecodeSize.c is size-optimized version of LzmaDecode.c.
573 But compiled code of LzmaDecodeSize.c can be larger than
574 compiled code of LzmaDecode.c. So it's better to use
575 LzmaDecode.c in most cases.
576
577
578 EXIT codes
579 -----------
580
581 LZMA decoder can return one of the following codes:
582
583 #define LZMA_RESULT_OK 0
584 #define LZMA_RESULT_DATA_ERROR 1
585
586 If you use callback function for input data and you return some
587 error code, LZMA Decoder also returns that code.
588
589
590
591 LZMA Defines
592 ------------
593
594 _LZMA_IN_CB - Use callback for input data
595
596 _LZMA_OUT_READ - Use read function for output data
597
598 _LZMA_LOC_OPT - Enable local speed optimizations inside code.
599 _LZMA_LOC_OPT is only for LzmaDecodeSize.c (size-optimized version).
600 _LZMA_LOC_OPT doesn't affect LzmaDecode.c (speed-optimized version)
601 and LzmaStateDecode.c
602
603 _LZMA_PROB32 - It can increase speed on some 32-bit CPUs,
604 but memory usage will be doubled in that case
605
606 _LZMA_UINT32_IS_ULONG - Define it if int is 16-bit on your compiler
607 and long is 32-bit.
608
609 _LZMA_SYSTEM_SIZE_T - Define it if you want to use system's size_t.
610 You can use it to enable 64-bit sizes supporting
611
612
613
614 C++ LZMA Encoder/Decoder
615 ~~~~~~~~~~~~~~~~~~~~~~~~
616 C++ LZMA code use COM-like interfaces. So if you want to use it,
617 you can study basics of COM/OLE.
618
619 By default, LZMA Encoder contains all Match Finders.
620 But for compressing it's enough to have just one of them.
621 So for reducing size of compressing code you can define:
622 #define COMPRESS_MF_BT
623 #define COMPRESS_MF_BT4
624 and it will use only bt4 match finder.
625
626
627 ---
628
629 http://www.7-zip.org
630 http://www.7-zip.org/support.html

Mercurial Repository: The Tor Browser

Europalab SCM Repository Server

mercurial