Sunday, May 26, 2013

Cross-building ICU for Applications on Embedded Devices

There are some pretty common libraries that come very handy in some situations when developing for common embedded systems like Android and iOS. One of this is the ICU library, which is useful when you need to work on code page conversion, collation, transliteration etc...

ICU is available as library for some systems, but it is quite large (libicudata for instance is more than 23MB alone). It is possible to reduce the size considerably reading this, and rebuilding.

ICU is perfectly portable on Linux, MacOS, Android, iOS, Linux Embedded etc... The process of cross-building is very simple, but still it took me a couple of hours to build for Linux, MacOS, iOS device and simulator and Android. So, I found some scripts around and fixed those bit (maybe the originals were a little outdated). I therefore write here a couple of notes on how to do it quickly (tested on 51.1).

Download the sources

I commonly download the sources from the repo directly:

export MY_DIR=some building directory
cd $MY_DIR
svn export http://source.icu-project.org/repos/icu/icu/tags/release-51-2 icu-51.2


You might want now to modify uconfig.h or data to avoid including data which is useless for your application: http://userguide.icu-project.org/icudata.

Build for Android

Cross-building ICU requires to build it first for the system where the cross-build is run, then for the target system. So, if we're using Linux when building for Android, let's first build ICU for Linux:

cd $MY_DIR
mkdir build_icu_linux
cd build_icu_linux


and use this script to build from there (change the variables and build options according to your needs):

export ICU_SOURCES=$MY_DIR/icu-51.2
export CPPFLAGS="-O3 -fno-short-wchar -DU_USING_ICU_NAMESPACE=1 -fno-short-enums \
-DU_HAVE_NL_LANGINFO_CODESET=0 -D__STDC_INT64__ -DU_TIMEZONE=0 \
-DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_BREAK_ITERATION=1 \
-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 -DUCONFIG_NO_TRANSLITERATION=0 \
-DUCONFIG_NO_REGULAR_EXPRESSIONS=1"

sh $ICU_SOURCES/source/runConfigureICU Linux --prefix=$PWD/icu_build --enable-extras=no \
--enable-strict=no -enable-static --enable-shared=no --enable-tests=no \
--enable-samples=no --enable-dyload=no
make -j4
make install


Now you can build for Android:

cd $MY_DIR
mkdir build_icu_android
cd build_icu_android


and use this script:

export ICU_SOURCES=$MY_DIR/icu-51.2
export ANDROIDVER=8
export AR=/usr/bin/ar
export BASE=$MY_DIR
export HOST_ICU=$BASE/build_icu_android
export ICU_CROSS_BUILD=$BASE/build_icu_linux
export NDK_STANDARD_ROOT=your toolchain root
export CPPFLAGS="-I$NDK_STANDARD_ROOT/sysroot/usr/include/ \
-O3 -fno-short-wchar -DU_USING_ICU_NAMESPACE=1 -fno-short-enums \
-DU_HAVE_NL_LANGINFO_CODESET=0 -D__STDC_INT64__ -DU_TIMEZONE=0 \
-DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_BREAK_ITERATION=1 \
-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 -DUCONFIG_NO_TRANSLITERATION=0 \
-DUCONFIG_NO_REGULAR_EXPRESSIONS=1"
export LDFLAGS="-lc -lstdc++ -Wl,-rpath-link=$NDK_STANDARD_ROOT/sysroot/usr/lib/"

export PATH=$PATH:$NDK_STANDARD_ROOT/bin

$ICU_SOURCES/source/configure --with-cross-build=$ICU_CROSS_BUILD \
--enable-extras=no --enable-strict=no -enable-static --enable-shared=no \
--enable-tests=no --enable-samples=no --enable-dyload=no \
--host=arm-linux-androideabi --prefix=$PWD/icu_build
make -j4
make install


In the icu_build directory you should have all you need to build your new application.
Note that I didn't use the NDK here, but the standard toolchain that results from the make-standalone-toolchain.sh script in the NDK.
Also note that part of ICU is already in /system/lib in some Android devices but I don't think there is any guarantee that it will be in every device (not part of the standard Android interface) and don't know exactly what is included inside that build.

Building for iOS

The same approach can be applied to cross-build for iOS. First, I built for MacOS in this case, and then for iOS device and simulator.

cd $MY_DIR
mkdir build_icu_mac
cd build_icu_mac


The script is similar to the Linux one:

ICUSRC_PATH=$MY_DIR/icu-51.2
export CPPFLAGS="-O3 -DU_USING_ICU_NAMESPACE=1 -fno-short-enums \
-DU_HAVE_NL_LANGINFO_CODESET=0 -D__STDC_INT64__ -DU_TIMEZONE=0 \
-DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_BREAK_ITERATION=1 \
-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 -DUCONFIG_NO_TRANSLITERATION=0 \
-DUCONFIG_NO_REGULAR_EXPRESSIONS=1"

sh $ICUSRC_PATH/source/runConfigureICU MacOSX --prefix=$PWD/icu_build --enable-extras=no \
--enable-strict=no -enable-static --enable-shared=no --enable-tests=no \
--enable-samples=no --enable-dyload=no
make -j4
make install


Then I built for the iOS simulator in:

cd $MY_DIR
mkdir build_icu_simulator
cd build_icu_simulator


with this script:

DEVROOT=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer
SDKROOT=$DEVROOT/SDKs/iPhoneSimulator6.1.sdk
SYSROOT=$SDKROOT

ICU_SOURCES=$MY_DIR/icu-51.2
ICU_FLAGS="-I$ICU_PATH/source/common/ -I$ICU_PATH/source/tools/tzcode/ -O3 \
-DU_USING_ICU_NAMESPACE=1 -fno-short-enums -DU_HAVE_NL_LANGINFO_CODESET=0 \
-D__STDC_INT64__ -DU_TIMEZONE=0 -DUCONFIG_NO_LEGACY_CONVERSION=1 \
-DUCONFIG_NO_BREAK_ITERATION=1 -DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 \
-DUCONFIG_NO_TRANSLITERATION=0 -DUCONFIG_NO_REGULAR_EXPRESSIONS=1"

export CPPFLAGS="-I$SDKROOT/usr/include/ -I$SDKROOT/usr/include/ -I./include/ \
-miphoneos-version-min=2.2 $ICU_FLAGS -pipe -arch i386 -no-cpp-precomp \
-isysroot $SDKROOT"

export LDFLAGS="-arch i386 -L$SDKROOT/usr/lib/ -isysroot $SDKROOT \
-Wl,-dead_strip -miphoneos-version-min=2.0"

sh $ICU_PATH/source/configure --host=i686-apple-darwin11 --enable-static --disable-shared \
-with-cross-build=/Users/luca/tmp/icu_build_mac --prefix=$PWD/icu_build
make -j4
make install


It works similarly for the arm device itself:

cd $MY_DIR
mkdir build_icu_device
cd build_icu_device


and this is the script to build:

DEVROOT=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer
SDKROOT=$DEVROOT/SDKs/iPhoneOS6.1.sdk
SYSROOT=$SDKROOT

ICU_PATH=$MY_DIR/icu-51.2
ICU_FLAGS="-I$ICU_PATH/source/common/ -I$ICU_PATH/source/tools/tzcode/ -O3 \
-fno-short-wchar -DU_USING_ICU_NAMESPACE=1 -fno-short-enums \
-DU_HAVE_NL_LANGINFO_CODESET=0 -D__STDC_INT64__ -DU_TIMEZONE=0 \
-DUCONFIG_NO_LEGACY_CONVERSION=1 -DUCONFIG_NO_BREAK_ITERATION=1 \
-DUCONFIG_NO_COLLATION=1 -DUCONFIG_NO_FORMATTING=1 -DUCONFIG_NO_TRANSLITERATION=0 \
-DUCONFIG_NO_REGULAR_EXPRESSIONS=1"

export CPPFLAGS="-I$SDKROOT/usr/include/ -I$SDKROOT/usr/include/ -I./include/ \
-miphoneos-version-min=2.2 $ICU_FLAGS -pipe -no-cpp-precomp -isysroot $SDKROOT"

export CC="$DEVROOT/usr/llvm-gcc-4.2/bin/arm-apple-darwin10-llvm-gcc-4.2"
export CXX="$DEVROOT/usr/llvm-gcc-4.2/bin/arm-apple-darwin10-llvm-g++-4.2"

export LDFLAGS="-L$SDKROOT/usr/lib/ -isysroot $SDKROOT -Wl,-dead_strip \
-miphoneos-version-min=2.0"

sh $ICU_PATH/source/configure --host=arm-apple-darwin --enable-static \
--disable-shared -with-cross-build=$MY_DIR/icu_build_mac --prefix=$PWD/icu_build
make -j4
make install


Post a comment if you find something wrong!
Of course the same approach might work similarly for other embedded devices with a proper toolchain :-)
Not too difficult, but still might speed up your work! ;-)

9 comments:

  1. Thank you so much for the post! I've been trying to get ICU to compile for Android for over a week with no success until now.

    However, I still have one problem. When I try to link to the ICU libs in my NDK project, I get the following linking error:

    error: undefined reference to 'icu_51::BreakIterator::createLineInstance(icu_51::Locale const&, UErrorCode&)'
    /Developer/Android-NDK/toolchains/arm-linux-androideabi-4.6/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld: jni/libicuuc.a(udata.ao): in function openCommonData(char const*, int, UErrorCode*):udata.cpp(.text._ZL14openCommonDataPKciP10UErrorCode+0x224): error: undefined reference to 'icudt51_dat'
    /Developer/Android-NDK/toolchains/arm-linux-androideabi-4.6/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld: jni/libicuuc.a(udata.ao): in function openCommonData(char const*, int, UErrorCode*):udata.cpp(.text._ZL14openCommonDataPKciP10UErrorCode+0x22c): error: undefined reference to 'icudt51_dat'

    Any idea what's going on? This happens even without changing uconfig.h or data.

    ReplyDelete
    Replies
    1. I did some research, and it seems that normally ICU Data is kept in a shared library, so now it it can't find it.
      Now what?

      Delete
    2. Did you build statically? How are you linking to icu in your makefile?

      Delete
    3. I built it statically, like your instructions. I am linking to it with the NDK. Here's my Android.mk file: http://pastebin.com/evrSqnBn

      Delete
    4. Also, if I try to do this:
      // Test to make sure ICU is working
      string testing;
      UnicodeString testStr = UNICODE_STRING_SIMPLE("Testing with ICU!");
      testStr.toUTF8String(testing);

      I get this:
      error: 'class icu_51::UnicodeString' has no member named 'toUTF8String'

      Strange...

      Delete
    5. Did you have a look at the sources? I'm no icu expert, I just built it for a project I worked on.

      Anyway, in unistr.h I see this:

      #if U_HAVE_STD_STRING

      /**
      * Convert the UnicodeString to UTF-8 and append the result
      * to a standard string.
      * Unpaired surrogates are replaced with U+FFFD.
      * Calls toUTF8().
      *
      * @param result A standard string (or a compatible object)
      * to which the UTF-8 version of the string is appended.
      * @return The string object.
      * @stable ICU 4.2
      * @see toUTF8
      */
      template
      StringClass &toUTF8String(StringClass &result) const {
      StringByteSink sbs(&result);
      toUTF8(sbs);
      return result;
      }

      #endif

      and in platform.h I see this:

      /**
      * \def U_HAVE_STD_STRING
      * Defines whether the standard C++ (STL) <string> header is available.
      * @internal
      */
      #ifdef U_HAVE_STD_STRING
      /* Use the predefined value. */
      #elif U_PLATFORM == U_PF_ANDROID
      # define U_HAVE_STD_STRING 0
      #else
      # define U_HAVE_STD_STRING 1
      #endif

      So, did you try to add U_HAVE_STD_STRING=1 to the definitions already? I use that method myself, I don't know why my scripts do not define it... but have you tried?

      Delete
    6. Tried it, same.

      ICU is becoming a real pain... I wish they officially supported mobile platforms. Do you know of any other framework that lets you develop libraries for Android and iOS with Unicode support?

      Delete
    7. I had the same error (undefined reference to 'icudt51_dat') while linking with static ICU on Android. Solved it by placing icu-data prebuilt library at the end of LOCAL_STATIC_LIBRARIES list in my Android.mk, like this:
      LOCAL_STATIC_LIBRARIES := icu-i18n icu-io icu-le icu-lx icu-tu icu-uc icu-data
      This way it works.

      Delete
  2. This post was really helpful, but I got stuck building the android version until I changed the LDFLAGS to:

    export LDFLAGS="-lc -lstdc++ --sysroot=$NDK_STANDARD_ROOT/sysroot/"

    In the config.log file, it was saying that the compiler tests were failing because it could not find the crtbegin_dynamic.o file.

    ReplyDelete