1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/intl/icu/source/data/cldr-icu-readme.txt Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,263 @@
1.4 +# Copyright (C) 2010-2013, International Business Machines Corporation and others.
1.5 +# All Rights Reserved.
1.6 +#
1.7 +# Commands for regenerating ICU4C locale data (.txt files) from CLDR.
1.8 +#
1.9 +# The process requires local copies of
1.10 +# - CLDR (the source of most of the data, and some Java tools)
1.11 +# - ICU4J (used only for checking the converted data)
1.12 +# - ICU4C (the destination for the new data, and the source for some of it)
1.13 +# (Either check out ICU4C from Subversion, or download the additional
1.14 +# icu4c-*-data.zip file so that the icu/source/data/ directory is fully
1.15 +# populated.)
1.16 +#
1.17 +# For an official CLDR data integration into ICU, these should be clean, freshly
1.18 +# checked-out. For released CLDR sources, an alternative to checking out sources
1.19 +# for a given version is downloading the zipped sources for the common (core.zip)
1.20 +# and tools (tools.zip) directory subtrees from the Data column in
1.21 +# [http://cldr.unicode.org/index/downloads].
1.22 +#
1.23 +# The versions of each of these must match. Included with the release notes for
1.24 +# ICU is the version number and/or a CLDR svn tag name for the revision of CLDR
1.25 +# that was the source of the data for that release of ICU.
1.26 +#
1.27 +# Note: Some versions of the OpenJDK will not build the CLDR java utilities.
1.28 +# If you see compilation errors complaining about type incompatibilities with
1.29 +# functions on generic classes, try switching to the Sun JDK.
1.30 +#
1.31 +# Besides a standard JDK, the process also requires ant
1.32 +# (http://ant.apache.org/),
1.33 +# plus the xml-apis.jar from the Apache xalan package
1.34 +# (http://xml.apache.org/xalan-j/downloads.html).
1.35 +#
1.36 +# Note: Enough things can (and will) fail in this process that it is best to
1.37 +# run the commands separately from an interactive shell. They should all
1.38 +# copy and paste without problems.
1.39 +#
1.40 +# It is often useful to save logs of the output of many of the steps in this
1.41 +# process. The commands below put log files in /tmp; you may want to put them
1.42 +# somewhere else.
1.43 +#
1.44 +#----
1.45 +#
1.46 +# There are several environment variables that need to be defined.
1.47 +#
1.48 +# a) Java- and ant-related variables
1.49 +#
1.50 +# JAVA_HOME: Path to JDK (a directory, containing e.g. bin/java, bin/javac,
1.51 +# etc.); on many systems this can be set using
1.52 +# `/usr/libexec/java_home`.
1.53 +#
1.54 +# ANT_OPTS: You may want to set:
1.55 +#
1.56 +# -Xmx1024m, to give Java more memory; otherwise it may run out
1.57 +# of heap.
1.58 +#
1.59 +# b) CLDR-related variables
1.60 +#
1.61 +# CLDR_DIR: Path to root of CLDR sources, below which are the common and
1.62 +# tools directories.
1.63 +# CLDR_CLASSES: Defined relative to CLDR_DIR. It only needs to be set if you
1.64 +# are not running ant jar for CLDR and have a non-default output
1.65 +# folder for cldr-tools classes.
1.66 +#
1.67 +# c) ICU-related variables
1.68 +# These variables only need to be set if you're directly reusing the
1.69 +# commands below.
1.70 +#
1.71 +# ICU4C_DIR: Path to root of ICU4C sources, below which is the source dir.
1.72 +#
1.73 +# ICU4J_ROOT: Path to root of ICU4J sources, below which is the main dir.
1.74 +#
1.75 +#----
1.76 +#
1.77 +# If you are adding or removing locales, or specific kinds of locale data,
1.78 +# there are some xml files in the ICU sources that need to be updated (these xml
1.79 +# files are used in addition to the CLDR files as inputs to the CLDR data build
1.80 +# process for ICU):
1.81 +#
1.82 +# icu/trunk/source/data/icu-config.xml - Update <locales> to add or remove
1.83 +# CLDR locales for inclusion in ICU. Update <paths> to prefer
1.84 +# alt forms for certain paths, or to exclude certain paths; note
1.85 +# that <paths> items can only have draft or alt attributes.
1.86 +#
1.87 +# Note that if a language-only locale (e.g. "de") is included in
1.88 +# <locales>, then all region sublocales for that language that
1.89 +# are present in CLDR data (e.g. "de_AT", "de_BE", "de_CH", etc.)
1.90 +# should also be included in <locales>, per PMC policy decision
1.91 +# 2012-05-02 (see http://bugs.icu-project.org/trac/ticket/9298).
1.92 +#
1.93 +# icu/trunk/source/data/build.xml - If you are adding or removing break
1.94 +# iterators, you need to update <fileset id="brkitr" ...> under
1.95 +# <target name="clean" ...> to clean the correct set of files.
1.96 +#
1.97 +# icu/trunk/source/data/xml/ - If you are adding a new locale, break
1.98 +# iterator, collation tailoring, or rule-based number formatter,
1.99 +# you may need to add a corresponding xml file in (respectively)
1.100 +# the main/, brkitr/, collation/, or rbnf/ subdirectory here.
1.101 +#
1.102 +#----
1.103 +#
1.104 +# For an official CLDR data integration into ICU, there are some additional
1.105 +# considerations:
1.106 +#
1.107 +# a) Don't commit anything in ICU sources (and possibly any changes in CLDR
1.108 +# sources, depending on their nature) until you have finished testing and
1.109 +# resolving build issues and test failures for both ICU4C and ICU4J.
1.110 +#
1.111 +# b) There are version numbers that may need manual updating in CLDR (other
1.112 +# version numbers get updated automatically, based on these):
1.113 +#
1.114 +# common/dtd/ldml.dtd - update cldrVersion
1.115 +# common/dtd/ldmlBCP47.dtd - update cldrVersion
1.116 +# common/dtd/ldmlSupplemental.dtd - update cldrVersion
1.117 +# tools/java/org/unicode/cldr/util/CLDRFile.java - update GEN_VERSION
1.118 +#
1.119 +# c) After everything is committed, you will need to tag the CLDR, ICU4J, and
1.120 +# ICU4C sources that ended up being used for the integration; see step 17
1.121 +# below.
1.122 +#
1.123 +################################################################################
1.124 +
1.125 +# 1a. Java and ant variables, adjust for your system
1.126 +
1.127 +export JAVA_HOME=`/usr/libexec/java_home`
1.128 +export ANT_OPTS="-Xmx1024m
1.129 +
1.130 +# 1b. CLDR variables, adjust for your setup; with cygwin it might be e.g.
1.131 +# CLDR_DIR=`cygpath -wp /build/cldr`
1.132 +
1.133 +export CLDR_DIR=$HOME/cldr/trunk
1.134 +#export CLDR_CLASSES=$CLDR_DIR/tools/java/classes
1.135 +
1.136 +# 1c. ICU variables
1.137 +
1.138 +export ICU4C_DIR=$HOME/icu/icu/trunk
1.139 +export ICU4J_ROOT=$HOME/icu/icu4j/trunk
1.140 +
1.141 +# 2. Build the CLDR Java tools
1.142 +
1.143 +cd $CLDR_DIR/tools/java
1.144 +#cd $CLDR_DIR/cldr-tools
1.145 +ant jar
1.146 +
1.147 +# 3. Configure ICU4C, build and test without new data first, to verify that
1.148 +# there are no pre-existing errors (configure shown here for MacOSX, adjust
1.149 +# for your platform).
1.150 +
1.151 +cd $ICU4C_DIR/source
1.152 +./runConfigureICU MacOSX
1.153 +make all 2>&1 | tee /tmp/icu4c-oldData-makeAll.txt
1.154 +make check 2>&1 | tee /tmp/icu4c-oldData-makeCheck.txt
1.155 +
1.156 +# 4. Build the new ICU4C data files; these include .txt files and .mk files.
1.157 +# These new files will replace whatever was already present in the ICU4C sources.
1.158 +# This process uses ant with ICU's data/build.xml and data/icu-config.xml to
1.159 +# operate (via CLDR's ant/CLDRConverterTool.java and ant/CLDRBuild.java) the
1.160 +# necessary CLDR tools including LDML2ICUConverter, ConvertTransforms, etc.
1.161 +# This process will take several minutes.
1.162 +# Keep a log so you can investigate anything that looks suspicious.
1.163 +
1.164 +cd $ICU4C_DIR/source/data
1.165 +ant clean
1.166 +ant all 2>&1 | tee /tmp/cldrNN-buildLog.txt
1.167 +
1.168 +# 5. Check which data files have modifications, which have been added or removed
1.169 +# (if there are no changes, you may not need to proceed further). Make sure the
1.170 +# list seems reasonable.
1.171 +
1.172 +svn status
1.173 +
1.174 +# 6. Fix any errors, investigate any warnings. Some warnings are expected,
1.175 +# including warnings for missing versions in locale names which specify some
1.176 +# collationvariants, e.g.
1.177 +# [cldr-build] WARNING (ja_JP_TRADITIONAL): No version #??
1.178 +# [cldr-build] WARNING (zh_TW_STROKE): No version #??
1.179 +# and warnings for some empty collation bundles, e.g.
1.180 +# [cldr-build] WARNING (en): warning: No collations found. Bundle will ...
1.181 +# [cldr-build] WARNING (to): warning: No collations found. Bundle will ...
1.182 +#
1.183 +# Fixing may entail modifying CLDR source data or tools - for example,
1.184 +# updating the validSubLocales for collation data (file a bug if appropriate).
1.185 +# Repeat steps 4-5 until there are no build errors and no unexpected
1.186 +# warnings.
1.187 +
1.188 +# 7. Now rebuild ICU4C with the new data and run make check tests.
1.189 +# Again, keep a log so you can investigate the errors.
1.190 +
1.191 +cd $ICU4C_DIR/source
1.192 +make check 2>&1 | tee /tmp/icu4c-newData-makeCheck.txt
1.193 +
1.194 +# 8. Investigate each test case failure. The first run processing new CLDR data
1.195 +# from the Survey Tool can result in thousands of failures (in many cases, one
1.196 +# CLDR data fix can resolve hundreds of test failures). If the error is caused
1.197 +# by bad CLDR data, then file a CLDR bug, fix the data, and regenerate from
1.198 +# step 4. If the data is OK but the testcase needs to be updated because the
1.199 +# data has legitimately changed, then update the testcase. You will check in
1.200 +# the updated testcases along with the new ICU data at the end of this process.
1.201 +# Note that if the new data has any differences in structure, you will have to
1.202 +# update test/testdata/structLocale.txt or /tsutil/cldrtest/TestLocaleStructure
1.203 +# may fail.
1.204 +# Repeat steps 4-7 until there are no errors.
1.205 +
1.206 +# 9. Now run the make check tests in exhaustive mode:
1.207 +
1.208 +cd $ICU4C_DIR/source
1.209 +export INTLTEST_OPTS="-e"
1.210 +export CINTLTST_OPTS="-e"
1.211 +make check 2>&1 | tee /tmp/icu4c-newData-makeCheckEx.txt
1.212 +
1.213 +# 10. Again, investigate each failure, fixing CLDR data or ICU test cases as
1.214 +# appropriate, and repeating steps 4-7 and 9 until there are no errors.
1.215 +
1.216 +# 11. Now with ICU4J, build and test without new data first, to verify that
1.217 +# there are no pre-existing errors (or at least to have the pre-existing errors
1.218 +# as a base for comparison):
1.219 +
1.220 +cd $ICU4J_ROOT
1.221 +ant all 2>&1 | tee /tmp/icu4j-oldData-antAll.txt
1.222 +ant check 2>&1 | tee /tmp/icu4j-oldData-antCheck.txt
1.223 +
1.224 +# 12. Now build the new data for ICU4J
1.225 +
1.226 +cd $ICU4C_DIR/source/data
1.227 +make icu4j-data-install
1.228 +
1.229 +# 13. Now rebuild ICU4J with the new data and run tests:
1.230 +# Keep a log so you can investigate the errors.
1.231 +
1.232 +cd $ICU4J_ROOT
1.233 +ant check 2>&1 | tee /tmp/icu4j-newData-antCheck.txt
1.234 +
1.235 +# 14. Investigate test case failures; fix test cases and repeat from step 12,
1.236 +# or fix CLDR data and repeat from step 4, as appropriate, until; there are no
1.237 +# more failures in ICU4C or ICU4J (except failures that were present before you
1.238 +# began testing the new CLDR data).
1.239 +
1.240 +# 15. Check the file changes; then svn add or svn remove as necessary, and
1.241 +# commit the changes.
1.242 +
1.243 +cd $ICU4C_DIR/source
1.244 +svn status
1.245 +# add or remove as necessary, then commit
1.246 +
1.247 +cd $ICU4J_ROOT
1.248 +svn status
1.249 +# add or remove as necessary, then commit
1.250 +
1.251 +# 16. For an official CLDR data integration into ICU, now tag the CLDR, ICU4J,
1.252 +# and ICU4C sources with an appropriate CLDR milestone (you can check previous
1.253 +# tags for format), e.g.:
1.254 +
1.255 +svn copy svn+ssh://unicode.org/repos/cldr/trunk \
1.256 +svn+ssh://unicode.org/repos/cldr/tags/release-NNN \
1.257 +--parents -m "cldrbug nnnn: tag cldr sources for NNN"
1.258 +
1.259 +svn copy svn+ssh://source.icu-project.org/repos/icu/icu4j/trunk \
1.260 +svn+ssh://source.icu-project.org/repos/icu/icu4j/tags/cldr-NNN \
1.261 +--parents -m 'ticket:mmmm: tag the version used for integrating CLDR NNN'
1.262 +
1.263 +svn copy svn+ssh://source.icu-project.org/repos/icu/icu/trunk \
1.264 +svn+ssh://source.icu-project.org/repos/icu/icu/tags/cldr-NNN \
1.265 +--parents -m 'ticket:mmmm: tag the version used for integrating CLDR NNN'
1.266 +
