The Tor Browser / file revision

 /*

 *******************************************************************************

 * Copyright (c) 1996-2010, International Business Machines Corporation

 *               and others. All Rights Reserved.

 *******************************************************************************

 * File unorm.h

 *

 * Created by: Vladimir Weinstein 12052000

 *

 * Modification history :

 *

 * Date        Name        Description

 * 02/01/01    synwee      Added normalization quickcheck enum and method.

 */

 #ifndef UNORM_H

 #define UNORM_H

 #include "unicode/utypes.h"

 #if !UCONFIG_NO_NORMALIZATION

 #include "unicode/uiter.h"

 #include "unicode/unorm2.h"

 /**

  * \file

  * \brief C API: Unicode Normalization 

  *

  * <h2>Unicode normalization API</h2>

  *

  * Note: This API has been replaced by the unorm2.h API and is only available

  * for backward compatibility. The functions here simply delegate to the

  * unorm2.h functions, for example unorm2_getInstance() and unorm2_normalize().

  * There is one exception: The new API does not provide a replacement for unorm_compare().

  *

  * <code>unorm_normalize</code> transforms Unicode text into an equivalent composed or

  * decomposed form, allowing for easier sorting and searching of text.

  * <code>unorm_normalize</code> supports the standard normalization forms described in

  * <a href="http://www.unicode.org/unicode/reports/tr15/" target="unicode">

  * Unicode Standard Annex #15: Unicode Normalization Forms</a>.

  *

  * Characters with accents or other adornments can be encoded in

  * several different ways in Unicode.  For example, take the character A-acute.

  * In Unicode, this can be encoded as a single character (the

  * "composed" form):

  *

  * \code

  *      00C1    LATIN CAPITAL LETTER A WITH ACUTE

  * \endcode

  *

  * or as two separate characters (the "decomposed" form):

  *

  * \code

  *      0041    LATIN CAPITAL LETTER A

  *      0301    COMBINING ACUTE ACCENT

  * \endcode

  *

  * To a user of your program, however, both of these sequences should be

  * treated as the same "user-level" character "A with acute accent".  When you are searching or

  * comparing text, you must ensure that these two sequences are treated 

  * equivalently.  In addition, you must handle characters with more than one

  * accent.  Sometimes the order of a character's combining accents is

  * significant, while in other cases accent sequences in different orders are

  * really equivalent.

  *

  * Similarly, the string "ffi" can be encoded as three separate letters:

  *

  * \code

  *      0066    LATIN SMALL LETTER F

  *      0066    LATIN SMALL LETTER F

  *      0069    LATIN SMALL LETTER I

  * \endcode

  *

  * or as the single character

  *

  * \code

  *      FB03    LATIN SMALL LIGATURE FFI

  * \endcode

  *

  * The ffi ligature is not a distinct semantic character, and strictly speaking

  * it shouldn't be in Unicode at all, but it was included for compatibility

  * with existing character sets that already provided it.  The Unicode standard

  * identifies such characters by giving them "compatibility" decompositions

  * into the corresponding semantic characters.  When sorting and searching, you

  * will often want to use these mappings.

  *

  * <code>unorm_normalize</code> helps solve these problems by transforming text into the

  * canonical composed and decomposed forms as shown in the first example above.  

  * In addition, you can have it perform compatibility decompositions so that 

  * you can treat compatibility characters the same as their equivalents.

  * Finally, <code>unorm_normalize</code> rearranges accents into the proper canonical

  * order, so that you do not have to worry about accent rearrangement on your

  * own.

  *

  * Form FCD, "Fast C or D", is also designed for collation.

  * It allows to work on strings that are not necessarily normalized

  * with an algorithm (like in collation) that works under "canonical closure", i.e., it treats precomposed

  * characters and their decomposed equivalents the same.

  *

  * It is not a normalization form because it does not provide for uniqueness of representation. Multiple strings

  * may be canonically equivalent (their NFDs are identical) and may all conform to FCD without being identical

  * themselves.

  *

  * The form is defined such that the "raw decomposition", the recursive canonical decomposition of each character,

  * results in a string that is canonically ordered. This means that precomposed characters are allowed for as long

  * as their decompositions do not need canonical reordering.

  *

  * Its advantage for a process like collation is that all NFD and most NFC texts - and many unnormalized texts -

  * already conform to FCD and do not need to be normalized (NFD) for such a process. The FCD quick check will

  * return UNORM_YES for most strings in practice.

  *

  * unorm_normalize(UNORM_FCD) may be implemented with UNORM_NFD.

  *

  * For more details on FCD see the collation design document:

  * http://source.icu-project.org/repos/icu/icuhtml/trunk/design/collation/ICU_collation_design.htm

  *

  * ICU collation performs either NFD or FCD normalization automatically if normalization

  * is turned on for the collator object.

  * Beyond collation and string search, normalized strings may be useful for string equivalence comparisons,

  * transliteration/transcription, unique representations, etc.

  *

  * The W3C generally recommends to exchange texts in NFC.

  * Note also that most legacy character encodings use only precomposed forms and often do not

  * encode any combining marks by themselves. For conversion to such character encodings the

  * Unicode text needs to be normalized to NFC.

  * For more usage examples, see the Unicode Standard Annex.

  */

 /**

  * Constants for normalization modes.

  * @stable ICU 2.0

  */

 typedef enum {

   /** No decomposition/composition. @stable ICU 2.0 */

   UNORM_NONE = 1, 

   /** Canonical decomposition. @stable ICU 2.0 */

   UNORM_NFD = 2,

   /** Compatibility decomposition. @stable ICU 2.0 */

   UNORM_NFKD = 3,

   /** Canonical decomposition followed by canonical composition. @stable ICU 2.0 */

   UNORM_NFC = 4,

   /** Default normalization. @stable ICU 2.0 */

   UNORM_DEFAULT = UNORM_NFC, 

   /** Compatibility decomposition followed by canonical composition. @stable ICU 2.0 */

   UNORM_NFKC =5,

   /** "Fast C or D" form. @stable ICU 2.0 */

   UNORM_FCD = 6,

   /** One more than the highest normalization mode constant. @stable ICU 2.0 */

   UNORM_MODE_COUNT

 } UNormalizationMode;

 /**

  * Constants for options flags for normalization.

  * Use 0 for default options,

  * including normalization according to the Unicode version

  * that is currently supported by ICU (see u_getUnicodeVersion).

  * @stable ICU 2.6

  */

 enum {

     /**

      * Options bit set value to select Unicode 3.2 normalization

      * (except NormalizationCorrections).

      * At most one Unicode version can be selected at a time.

      * @stable ICU 2.6

      */

     UNORM_UNICODE_3_2=0x20

 };

 /**

  * Lowest-order bit number of unorm_compare() options bits corresponding to

  * normalization options bits.

  *

  * The options parameter for unorm_compare() uses most bits for

  * itself and for various comparison and folding flags.

  * The most significant bits, however, are shifted down and passed on

  * to the normalization implementation.

  * (That is, from unorm_compare(..., options, ...),

  * options>>UNORM_COMPARE_NORM_OPTIONS_SHIFT will be passed on to the

  * internal normalization functions.)

  *

  * @see unorm_compare

  * @stable ICU 2.6

  */

 #define UNORM_COMPARE_NORM_OPTIONS_SHIFT 20

 /**

  * Normalize a string.

  * The string will be normalized according the specified normalization mode

  * and options.

  * The source and result buffers must not be the same, nor overlap.

  *

  * @param source The string to normalize.

  * @param sourceLength The length of source, or -1 if NUL-terminated.

  * @param mode The normalization mode; one of UNORM_NONE, 

  *             UNORM_NFD, UNORM_NFC, UNORM_NFKC, UNORM_NFKD, UNORM_DEFAULT.

  * @param options The normalization options, ORed together (0 for no options).

  * @param result A pointer to a buffer to receive the result string.

  *               The result string is NUL-terminated if possible.

  * @param resultLength The maximum size of result.

  * @param status A pointer to a UErrorCode to receive any errors.

  * @return The total buffer size needed; if greater than resultLength,

  *         the output was truncated, and the error code is set to U_BUFFER_OVERFLOW_ERROR.

  * @stable ICU 2.0

  */

 U_STABLE int32_t U_EXPORT2 

 unorm_normalize(const UChar *source, int32_t sourceLength,

                 UNormalizationMode mode, int32_t options,

                 UChar *result, int32_t resultLength,

                 UErrorCode *status);

 /**

  * Performing quick check on a string, to quickly determine if the string is 

  * in a particular normalization format.

  * Three types of result can be returned UNORM_YES, UNORM_NO or

  * UNORM_MAYBE. Result UNORM_YES indicates that the argument

  * string is in the desired normalized format, UNORM_NO determines that

  * argument string is not in the desired normalized format. A 

  * UNORM_MAYBE result indicates that a more thorough check is required, 

  * the user may have to put the string in its normalized form and compare the 

  * results.

  *

  * @param source       string for determining if it is in a normalized format

  * @param sourcelength length of source to test, or -1 if NUL-terminated

  * @param mode         which normalization form to test for

  * @param status       a pointer to a UErrorCode to receive any errors

  * @return UNORM_YES, UNORM_NO or UNORM_MAYBE

  *

  * @see unorm_isNormalized

  * @stable ICU 2.0

  */

 U_STABLE UNormalizationCheckResult U_EXPORT2

 unorm_quickCheck(const UChar *source, int32_t sourcelength,

                  UNormalizationMode mode,

                  UErrorCode *status);

 /**

  * Performing quick check on a string; same as unorm_quickCheck but

  * takes an extra options parameter like most normalization functions.

  *

  * @param src        String that is to be tested if it is in a normalization format.

  * @param srcLength  Length of source to test, or -1 if NUL-terminated.

  * @param mode       Which normalization form to test for.

  * @param options    The normalization options, ORed together (0 for no options).

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return UNORM_YES, UNORM_NO or UNORM_MAYBE

  *

  * @see unorm_quickCheck

  * @see unorm_isNormalized

  * @stable ICU 2.6

  */

 U_STABLE UNormalizationCheckResult U_EXPORT2

 unorm_quickCheckWithOptions(const UChar *src, int32_t srcLength, 

                             UNormalizationMode mode, int32_t options,

                             UErrorCode *pErrorCode);

 /**

  * Test if a string is in a given normalization form.

  * This is semantically equivalent to source.equals(normalize(source, mode)) .

  *

  * Unlike unorm_quickCheck(), this function returns a definitive result,

  * never a "maybe".

  * For NFD, NFKD, and FCD, both functions work exactly the same.

  * For NFC and NFKC where quickCheck may return "maybe", this function will

  * perform further tests to arrive at a TRUE/FALSE result.

  *

  * @param src        String that is to be tested if it is in a normalization format.

  * @param srcLength  Length of source to test, or -1 if NUL-terminated.

  * @param mode       Which normalization form to test for.

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return Boolean value indicating whether the source string is in the

  *         "mode" normalization form.

  *

  * @see unorm_quickCheck

  * @stable ICU 2.2

  */

 U_STABLE UBool U_EXPORT2

 unorm_isNormalized(const UChar *src, int32_t srcLength,

                    UNormalizationMode mode,

                    UErrorCode *pErrorCode);

 /**

  * Test if a string is in a given normalization form; same as unorm_isNormalized but

  * takes an extra options parameter like most normalization functions.

  *

  * @param src        String that is to be tested if it is in a normalization format.

  * @param srcLength  Length of source to test, or -1 if NUL-terminated.

  * @param mode       Which normalization form to test for.

  * @param options    The normalization options, ORed together (0 for no options).

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return Boolean value indicating whether the source string is in the

  *         "mode/options" normalization form.

  *

  * @see unorm_quickCheck

  * @see unorm_isNormalized

  * @stable ICU 2.6

  */

 U_STABLE UBool U_EXPORT2

 unorm_isNormalizedWithOptions(const UChar *src, int32_t srcLength,

                               UNormalizationMode mode, int32_t options,

                               UErrorCode *pErrorCode);

 /**

  * Iterative normalization forward.

  * This function (together with unorm_previous) is somewhat

  * similar to the C++ Normalizer class (see its non-static functions).

  *

  * Iterative normalization is useful when only a small portion of a longer

  * string/text needs to be processed.

  *

  * For example, the likelihood may be high that processing the first 10% of some

  * text will be sufficient to find certain data.

  * Another example: When one wants to concatenate two normalized strings and get a

  * normalized result, it is much more efficient to normalize just a small part of

  * the result around the concatenation place instead of re-normalizing everything.

  *

  * The input text is an instance of the C character iteration API UCharIterator.

  * It may wrap around a simple string, a CharacterIterator, a Replaceable, or any

  * other kind of text object.

  *

  * If a buffer overflow occurs, then the caller needs to reset the iterator to the

  * old index and call the function again with a larger buffer - if the caller cares

  * for the actual output.

  * Regardless of the output buffer, the iterator will always be moved to the next

  * normalization boundary.

  *

  * This function (like unorm_previous) serves two purposes:

  *

  * 1) To find the next boundary so that the normalization of the part of the text

  * from the current position to that boundary does not affect and is not affected

  * by the part of the text beyond that boundary.

  *

  * 2) To normalize the text up to the boundary.

  *

  * The second step is optional, per the doNormalize parameter.

  * It is omitted for operations like string concatenation, where the two adjacent

  * string ends need to be normalized together.

  * In such a case, the output buffer will just contain a copy of the text up to the

  * boundary.

  *

  * pNeededToNormalize is an output-only parameter. Its output value is only defined

  * if normalization was requested (doNormalize) and successful (especially, no

  * buffer overflow).

  * It is useful for operations like a normalizing transliterator, where one would

  * not want to replace a piece of text if it is not modified.

  *

  * If doNormalize==TRUE and pNeededToNormalize!=NULL then *pNeeded... is set TRUE

  * if the normalization was necessary.

  *

  * If doNormalize==FALSE then *pNeededToNormalize will be set to FALSE.

  *

  * If the buffer overflows, then *pNeededToNormalize will be undefined;

  * essentially, whenever U_FAILURE is true (like in buffer overflows), this result

  * will be undefined.

  *

  * @param src The input text in the form of a C character iterator.

  * @param dest The output buffer; can be NULL if destCapacity==0 for pure preflighting.

  * @param destCapacity The number of UChars that fit into dest.

  * @param mode The normalization mode.

  * @param options The normalization options, ORed together (0 for no options).

  * @param doNormalize Indicates if the source text up to the next boundary

  *                    is to be normalized (TRUE) or just copied (FALSE).

  * @param pNeededToNormalize Output flag indicating if the normalization resulted in

  *                           different text from the input.

  *                           Not defined if an error occurs including buffer overflow.

  *                           Always FALSE if !doNormalize.

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return Length of output (number of UChars) when successful or buffer overflow.

  *

  * @see unorm_previous

  * @see unorm_normalize

  *

  * @stable ICU 2.1

  */

 U_STABLE int32_t U_EXPORT2

 unorm_next(UCharIterator *src,

            UChar *dest, int32_t destCapacity,

            UNormalizationMode mode, int32_t options,

            UBool doNormalize, UBool *pNeededToNormalize,

            UErrorCode *pErrorCode);

 /**

  * Iterative normalization backward.

  * This function (together with unorm_next) is somewhat

  * similar to the C++ Normalizer class (see its non-static functions).

  * For all details see unorm_next.

  *

  * @param src The input text in the form of a C character iterator.

  * @param dest The output buffer; can be NULL if destCapacity==0 for pure preflighting.

  * @param destCapacity The number of UChars that fit into dest.

  * @param mode The normalization mode.

  * @param options The normalization options, ORed together (0 for no options).

  * @param doNormalize Indicates if the source text up to the next boundary

  *                    is to be normalized (TRUE) or just copied (FALSE).

  * @param pNeededToNormalize Output flag indicating if the normalization resulted in

  *                           different text from the input.

  *                           Not defined if an error occurs including buffer overflow.

  *                           Always FALSE if !doNormalize.

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return Length of output (number of UChars) when successful or buffer overflow.

  *

  * @see unorm_next

  * @see unorm_normalize

  *

  * @stable ICU 2.1

  */

 U_STABLE int32_t U_EXPORT2

 unorm_previous(UCharIterator *src,

                UChar *dest, int32_t destCapacity,

                UNormalizationMode mode, int32_t options,

                UBool doNormalize, UBool *pNeededToNormalize,

                UErrorCode *pErrorCode);

 /**

  * Concatenate normalized strings, making sure that the result is normalized as well.

  *

  * If both the left and the right strings are in

  * the normalization form according to "mode/options",

  * then the result will be

  *

  * \code

  *     dest=normalize(left+right, mode, options)

  * \endcode

  *

  * With the input strings already being normalized,

  * this function will use unorm_next() and unorm_previous()

  * to find the adjacent end pieces of the input strings.

  * Only the concatenation of these end pieces will be normalized and

  * then concatenated with the remaining parts of the input strings.

  *

  * It is allowed to have dest==left to avoid copying the entire left string.

  *

  * @param left Left source string, may be same as dest.

  * @param leftLength Length of left source string, or -1 if NUL-terminated.

  * @param right Right source string. Must not be the same as dest, nor overlap.

  * @param rightLength Length of right source string, or -1 if NUL-terminated.

  * @param dest The output buffer; can be NULL if destCapacity==0 for pure preflighting.

  * @param destCapacity The number of UChars that fit into dest.

  * @param mode The normalization mode.

  * @param options The normalization options, ORed together (0 for no options).

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return Length of output (number of UChars) when successful or buffer overflow.

  *

  * @see unorm_normalize

  * @see unorm_next

  * @see unorm_previous

  *

  * @stable ICU 2.1

  */

 U_STABLE int32_t U_EXPORT2

 unorm_concatenate(const UChar *left, int32_t leftLength,

                   const UChar *right, int32_t rightLength,

                   UChar *dest, int32_t destCapacity,

                   UNormalizationMode mode, int32_t options,

                   UErrorCode *pErrorCode);

 /**

  * Option bit for unorm_compare:

  * Both input strings are assumed to fulfill FCD conditions.

  * @stable ICU 2.2

  */

 #define UNORM_INPUT_IS_FCD          0x20000

 /**

  * Option bit for unorm_compare:

  * Perform case-insensitive comparison.

  * @stable ICU 2.2

  */

 #define U_COMPARE_IGNORE_CASE       0x10000

 #ifndef U_COMPARE_CODE_POINT_ORDER

 /* see also unistr.h and ustring.h */

 /**

  * Option bit for u_strCaseCompare, u_strcasecmp, unorm_compare, etc:

  * Compare strings in code point order instead of code unit order.

  * @stable ICU 2.2

  */

 #define U_COMPARE_CODE_POINT_ORDER  0x8000

 #endif

 /**

  * Compare two strings for canonical equivalence.

  * Further options include case-insensitive comparison and

  * code point order (as opposed to code unit order).

  *

  * Canonical equivalence between two strings is defined as their normalized

  * forms (NFD or NFC) being identical.

  * This function compares strings incrementally instead of normalizing

  * (and optionally case-folding) both strings entirely,

  * improving performance significantly.

  *

  * Bulk normalization is only necessary if the strings do not fulfill the FCD

  * conditions. Only in this case, and only if the strings are relatively long,

  * is memory allocated temporarily.

  * For FCD strings and short non-FCD strings there is no memory allocation.

  *

  * Semantically, this is equivalent to

  *   strcmp[CodePointOrder](NFD(foldCase(NFD(s1))), NFD(foldCase(NFD(s2))))

  * where code point order and foldCase are all optional.

  *

  * UAX 21 2.5 Caseless Matching specifies that for a canonical caseless match

  * the case folding must be performed first, then the normalization.

  *

  * @param s1 First source string.

  * @param length1 Length of first source string, or -1 if NUL-terminated.

  *

  * @param s2 Second source string.

  * @param length2 Length of second source string, or -1 if NUL-terminated.

  *

  * @param options A bit set of options:

  *   - U_FOLD_CASE_DEFAULT or 0 is used for default options:

  *     Case-sensitive comparison in code unit order, and the input strings

  *     are quick-checked for FCD.

  *

  *   - UNORM_INPUT_IS_FCD

  *     Set if the caller knows that both s1 and s2 fulfill the FCD conditions.

  *     If not set, the function will quickCheck for FCD

  *     and normalize if necessary.

  *

  *   - U_COMPARE_CODE_POINT_ORDER

  *     Set to choose code point order instead of code unit order

  *     (see u_strCompare for details).

  *

  *   - U_COMPARE_IGNORE_CASE

  *     Set to compare strings case-insensitively using case folding,

  *     instead of case-sensitively.

  *     If set, then the following case folding options are used.

  *

  *   - Options as used with case-insensitive comparisons, currently:

  *

  *   - U_FOLD_CASE_EXCLUDE_SPECIAL_I

  *    (see u_strCaseCompare for details)

  *

  *   - regular normalization options shifted left by UNORM_COMPARE_NORM_OPTIONS_SHIFT

  *

  * @param pErrorCode ICU error code in/out parameter.

  *                   Must fulfill U_SUCCESS before the function call.

  * @return <0 or 0 or >0 as usual for string comparisons

  *

  * @see unorm_normalize

  * @see UNORM_FCD

  * @see u_strCompare

  * @see u_strCaseCompare

  *

  * @stable ICU 2.2

  */

 U_STABLE int32_t U_EXPORT2

 unorm_compare(const UChar *s1, int32_t length1,

               const UChar *s2, int32_t length2,

               uint32_t options,

               UErrorCode *pErrorCode);

 #endif /* #if !UCONFIG_NO_NORMALIZATION */

 #endif