CI build mobile libs #21

Robert-Steiner · 2021-03-09T10:15:18Z

Goal:

We want to test our android/ios libraries on different cpu architectures
in order to identify errors early in development.

What we have achieved:

We can compile and test our android libraries on different cpu architectures by using an emulator.
However, we can only compile the iOS libraries because there is no emulator for it.
We could run our tests on the iOS simulator, but it uses an architecture (x86_64)
which is not used in any iOS device. So it is not worth it.

Implementation details

Use a build matrix for testing/building android/ios libraries on different cpu architectures

There are several advantages to using a build matrix instead of one job that builds/tests all of the libraries one at a time:

each job can be executed in parallel
- this speeds up the overall execution of the ci
if one job fails, the others continue to run
- this makes debugging easier because we can immediately see if an error occurs only in a specific cpu architecture
each job has its own build cache
- this also speeds up the execution of the ci, since unpacking a small cache is faster than a large one

Optimised android test

We noticed that the execution time of the tests on the emulator is significantly reduced
if we compile our tests in release mode. However, in release mode the debug assertions are removed
which means that we lose some checks. So the idea was to compile it in debug mode (to keep the
debug assertions) and only set opt-level to 3 (to turn on all optimisations).
However, this turned out to be more difficult than expected. In the end we decided to use
RUSTFLAGS="-C opt-level=3 -C debug-assertions=yes". More details at the end of the description.

test-android-libs without caching / debug mode

Job	Build	Test rubert	Test rubert_tokenizer	Test xayn_ai	Test xayn_ai_ffi
aarch64-linux-android	3m 30s	144.56s	0.33s	0.12s	445.52s
armv7-linux-androideabi	4m 35s	152.85s	0.33s	0.13s	499.53s
i686-linux-android	3m 27s	151.51s	0.34s	0.14s	380.79s
x86_64-linux-android	3m 42s	146.17s	0.28s	0.12s	404.44s

Total duration: 17m 25s
Billable time: 59m 5s

test-android-libs without caching / release mode

Job	Build	Test rubert	Test rubert_tokenizer	Test xayn_ai	Test xayn_ai_ffi
aarch64-linux-android	7m 46s	25.27s	0.10s	0.05s	76.03s
armv7-linux-androideabi	7m 07s	13.64s	0.07s	0.04s	38.79s
i686-linux-android	7m 02s	8.62s	0.07s	0.03s	12.18s
x86_64-linux-android	7m 01s	8.81s	0.08s	0.03s	13.64s

Total duration: 11m 32s
Billable time: 37m 44s

test-android-libs with caching / release mode

Job	Build	Test rubert	Test rubert_tokenizer	Test xayn_ai	Test xayn_ai_ffi
aarch64-linux-android	1m 01s	23.76s	0.10s	0.04s	69.13s
armv7-linux-androideabi	1m 03s	12.16s	0.07s	0.03s	34.73s
i686-linux-android	1m 17s	10.12s	0.08s	0.03s	15.21s
x86_64-linux-android	55.27s	9.26s	0.07s	0.03s	13.79s

Total duration: 4m 38s
Billable time: 13m 7s

We have activated the option -- -Z unstable-options --report-time so that if a test suddenly becomes slow again, we can recognize it immediately.

Using `cross` for testing

cross makes it easy to test our code on different cpu architectures. Under the hood it runs a docker container that runs an emulator (qemu).

Features/Limitations of cross

builds iOS on macOS
- in the README they say that it is also possible to run the tests in iOS, however, i don't know how. I created an issue for that Running tests on *-apple-ios targets cross-rs/cross#535
builds android on linux & macOS
- you will need Rust < 1.50.0. 1.50.0 does not work Building for x86_64-unknown-linux-gnu on Windows host fails on Rust 1.50 cross-rs/cross#527 (at least on macOS)
since cross mounts the repo into the container, you cannot execute it in a folder belonging to a member who depends on another member

For example:

cd xayn_ai/xayn-ai
cross test --target aarch64-linux-android

does not work. It cannot find the directory of rubert, since it is not mounted into the container. So you have to run

cd xayn_ai
cross test --target aarch64-linux-android -p xayn-ai

Using `cargo-ndk` for building the release binaries

To build the final android libraries we will keep using cargo-ndk because it has some advantages over cross:

in ndk we can set the ndk api level (in cross it is hardcoded in the dockerfile)
ndk automatically strips the binary (with the right strip binary) + moves it into the jniLib folder structure

RUST_LOG=debug cargo ndk -t armeabi-v7a --platform 21 -o ./jniLibs build --release
[2021-03-18T10:03:09Z INFO  cargo_ndk::cli] Using NDK at path: /Users/robert/Library/Android/sdk/ndk/22.0.7026061
[2021-03-18T10:03:09Z INFO  cargo_ndk::cli] NDK API level: 21
[2021-03-18T10:03:09Z INFO  cargo_ndk::cli] Building targets: armeabi-v7a
[2021-03-18T10:03:09Z INFO  cargo_ndk::cli] Building armeabi-v7a (armv7-linux-androideabi)
[2021-03-18T10:03:09Z DEBUG cargo_ndk::cargo] ar: /Users/robert/Library/Android/sdk/ndk/22.0.7026061/toolchains/llvm/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ar
[2021-03-18T10:03:09Z DEBUG cargo_ndk::cargo] linker: /Users/robert/Library/Android/sdk/ndk/22.0.7026061/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang
[2021-03-18T10:03:09Z DEBUG cargo_ndk::cargo] cargo: /Users/robert/.rustup/toolchains/1.49.0-x86_64-apple-darwin/bin/cargo
[2021-03-18T10:03:09Z DEBUG cargo_ndk::cargo] Working directory does not match manifest-path
    Finished release [optimized] target(s) in 0.16s
[2021-03-18T10:03:09Z INFO  cargo_ndk::cli] Copying libraries to ./jniLibs...
[2021-03-18T10:03:09Z INFO  cargo_ndk::cli] /Users/robert/projects/xayn-ai/target/armv7-linux-androideabi/release/libxayn_ai_ffi.so -> ./jniLibs/armeabi-v7a/libxayn_ai_ffi.so
[2021-03-18T10:03:09Z DEBUG cargo_ndk::cargo] strip: /Users/robert/Library/Android/sdk/ndk/22.0.7026061/toolchains/llvm/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-strip

We will do the integration of cargo-ndk in a separate pr.

Why we chose `RUSTFLAGS` instead of `[profile.test]`

TL;DR:

The test profile only applies to our library but not on the dependencies.

cargo book

Note that when using the cargo test and cargo bench commands, the test/bench profiles only apply to the final test executable.
Dependencies will continue to use the dev/release profiles. Also note that when a library is built for unit tests,
then the library is built with the test profile. However, when building an integration test target, the library target
is built with the dev profile and linked into the integration test executable.

cargo test

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
    -C embed-bitcode=no
    -C debuginfo=2
    -C metadata=79293ec08c2c0f19
    -C extra-filename=-79293ec08c2c0f19
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps

If opt-level is missing the compiler will by default use -C opt-level=0
which implies -C debug-assertions=on

Our Library

rustc
    --crate-name mem_leak
    --edition=2018 mem_leak/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi
    --emit=dep-info,link
    -C embed-bitcode=no
    -C debuginfo=2
    --test
    -C metadata=ad90509da42dfc40
    -C extra-filename=-ad90509da42dfc40
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
    --extern a_dep=/Users/robert/projects/mem_leak_lib/target/debug/deps/liba_dep-79293ec08c2c0f19.rlib

default: -C opt-level=0 -> -C debug-assertions=on

dependencies debug-assertions turned on
library debug-assertions turned on
dependencies optimised
library optimised

cargo test &

[profile.dev]
opt-level = 3

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
-C opt-level=3
    -C embed-bitcode=no
    -C debuginfo=2
-C debug-assertions=on
    -C metadata=e55d6ac30c7cc4bc
    -C extra-filename=-e55d6ac30c7cc4bc
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps

-C opt-level=3 implies debug-assertions=off but cargo correctly uses the default value of
debug-assertions=on in profile.dev.

Our Library

rustc
    --crate-name mem_leak
    --edition=2018 mem_leak/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi
    --emit=dep-info,link
    -C embed-bitcode=no
    -C debuginfo=2
    --test
    -C metadata=06e2c8a5b5ed242d
    -C extra-filename=-06e2c8a5b5ed242d
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
    --extern a_dep=/Users/robert/projects/mem_leak_lib/target/debug/deps/liba_dep-e55d6ac30c7cc4bc.rlib

dependencies debug-assertions turned on
library debug-assertions turned on
dependencies optimised
library optimised

cargo test &

[profile.dev]
opt-level = 3

[profile.test]
opt-level = 3

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
-C opt-level=3
    -C embed-bitcode=no
    -C debuginfo=2
-C debug-assertions=on
    -C metadata=e55d6ac30c7cc4bc
    -C extra-filename=-e55d6ac30c7cc4bc
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps

Our Library

rustc
    --crate-name mem_leak
    --edition=2018 mem_leak/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi
    --emit=dep-info,link
-C opt-level=3
    -C embed-bitcode=no
    -C debuginfo=2
-C debug-assertions=on
    --test
    -C metadata=b8ff0b3fe541eacb
    -C extra-filename=-b8ff0b3fe541eacb
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
    --extern a_dep=/Users/robert/projects/mem_leak_lib/target/debug/deps/liba_dep-e55d6ac30c7cc4bc.rlib

dependencies debug-assertions turned on
library debug-assertions turned on
dependencies optimised
library optimised

Cool, we've reached our goal, haven't we? Not really.
If we add opt-level=3 to profile.dev, the complication times during local
development will be longer.

An alternative would be to use RUSTFLAGS. Let's take a look.

RUSTFLAGS="-C opt-level=3" cargo test

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
    -C embed-bitcode=no
    -C debuginfo=2
    -C metadata=79293ec08c2c0f19
    -C extra-filename=-79293ec08c2c0f19
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
-C opt-level=3

Our Library

rustc
    --crate-name mem_leak
    --edition=2018 mem_leak/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi
    --emit=dep-info,link
    -C embed-bitcode=no
    -C debuginfo=2
    --test
    -C metadata=ad90509da42dfc40
    -C extra-filename=-ad90509da42dfc40
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
    --extern a_dep=/Users/robert/projects/mem_leak_lib/target/debug/deps/liba_dep-79293ec08c2c0f19.rlib
-C opt-level=3

dependencies debug-assertions turned on
library debug-assertions turned on
dependencies optimised
library optimised

Hmmm cargo does not apply the default values of the profiles anymore.

What happens if we and profile.dev again?

RUSTFLAGS="-C opt-level=3" cargo test &

[profile.dev]
opt-level = 3

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
-C opt-level=3
    -C embed-bitcode=no
    -C debuginfo=2
-C debug-assertions=on
    -C metadata=e55d6ac30c7cc4bc
    -C extra-filename=-e55d6ac30c7cc4bc
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
-C opt-level=3

debug-assertions=on is back and we have opt-level=3 twice.
Which of the two opt-level=3 will it actually choose?

Our Library

rustc
    --crate-name mem_leak
    --edition=2018 mem_leak/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi
    --emit=dep-info,link
    -C embed-bitcode=no
    -C debuginfo=2
    --test
    -C metadata=06e2c8a5b5ed242d
    -C extra-filename=-06e2c8a5b5ed242d
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
    --extern a_dep=/Users/robert/projects/mem_leak_lib/target/debug/deps/liba_dep-e55d6ac30c7cc4bc.rlib
-C opt-level=3

dependencies debug-assertions turned on
library debug-assertions turned on
dependencies optimised
library optimised

Which of the two opt-level=3 will it actually choose?

Let's set opt-level to 3 in the Cargo.toml and call cargo with
RUSTFLAGS="-C debug-assertions=no".

RUSTFLAGS="-C debug-assertions=no" cargo test &

[profile.dev]
opt-level = 3

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
-C opt-level=3
    -C embed-bitcode=no
    -C debuginfo=2
-C debug-assertions=on
    -C metadata=e55d6ac30c7cc4bc
    -C extra-filename=-e55d6ac30c7cc4bc
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
-C debug-assertions=no

The test did not fail, so the debug-assertions are turned off which means that the last
debug-assertions flag overwrites the previous flag.

In order to have both opt-level=3 and debug-assertions=yes we need to run

RUSTFLAGS="-C opt-level=3 -C debug-assertions=yes" cargo test

Dependencies

rustc
    --crate-name a_dep
    --edition=2018 a-dep/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi,artifacts
    --crate-type lib
    --emit=dep-info,metadata,link
    -C embed-bitcode=no
    -C debuginfo=2
    -C metadata=e55d6ac30c7cc4bc
    -C extra-filename=-e55d6ac30c7cc4bc
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
-C opt-level=3
-C debug-assertions=yes

Our Library

rustc
    --crate-name mem_leak
    --edition=2018 mem_leak/src/lib.rs
    --error-format=json
    --json=diagnostic-rendered-ansi
    --emit=dep-info,link
    -C embed-bitcode=no
    -C debuginfo=2
    --test
    -C metadata=ad90509da42dfc40
    -C extra-filename=-ad90509da42dfc40
    --out-dir /Users/robert/projects/mem_leak_lib/target/debug/deps
    -C incremental=/Users/robert/projects/mem_leak_lib/target/debug/incremental
    -L dependency=/Users/robert/projects/mem_leak_lib/target/debug/deps
    --extern a_dep=/Users/robert/projects/mem_leak_lib/target/debug/deps/liba_dep-79293ec08c2c0f19.rlib
-C opt-level=3
-C debug-assertions=yes

.github/workflows/ci.yml

janpetschexain · 2021-03-25T17:18:56Z

your pr summary is awesome 👍 we should put a link to this readme/pr directly in the ci yaml file.

Robert-Steiner · 2021-03-26T08:28:14Z

your pr summary is awesome 👍 we should put a link to this readme/pr directly in the ci yaml file.

Thanks 😊. I can do that!

janpetschexain

lgtm 👍

.github/workflows/ci.yml

.github/workflows/audit.yml

.github/workflows/ci.yml

Co-authored-by: janpetschexain <58227040+janpetschexain@users.noreply.github.com>

Robert-Steiner changed the title ~~wip~~ CI build mobile/WASM libs Mar 9, 2021

Robert-Steiner force-pushed the ffi-ci branch from 2705093 to 6a65077 Compare March 9, 2021 12:47

Robert-Steiner changed the title ~~CI build mobile/WASM libs~~ CI build mobile libs Mar 10, 2021

Robert-Steiner force-pushed the ffi-ci branch 2 times, most recently from de46fcc to 86e5703 Compare March 12, 2021 14:36

janpetschexain force-pushed the ffi-ci branch 2 times, most recently from ebded9c to f63e059 Compare March 17, 2021 13:37

acrrd reviewed Mar 18, 2021

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

janpetschexain force-pushed the ffi-ci branch 3 times, most recently from bbbe556 to 3514dd4 Compare March 25, 2021 15:22

janpetschexain force-pushed the ffi-ci branch 3 times, most recently from ed08f13 to 85d6a2a Compare April 6, 2021 16:15

janpetschexain force-pushed the ffi-ci branch from 85d6a2a to adcb3a9 Compare April 7, 2021 12:59

Robert-Steiner marked this pull request as ready for review April 7, 2021 15:24

janpetschexain approved these changes Apr 7, 2021

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

Robert-Steiner force-pushed the ffi-ci branch from 1ae3230 to 13beb35 Compare April 7, 2021 16:38

acrrd approved these changes Apr 7, 2021

View reviewed changes

.github/workflows/audit.yml Show resolved Hide resolved

.github/workflows/ci.yml Show resolved Hide resolved

Robert-Steiner added 9 commits April 8, 2021 14:27

build ios/android libs

3b6f1f0

fmt [skip ci]

d6b214a

[skip ci]

20bf51a

test

08e5965

fix

e6ff656

download model

c6a94ae

release

414f394

test cache

e6a62f5

test podman

fac9504

Robert-Steiner and others added 19 commits April 8, 2021 14:27

test whole ci

715c91f

set fail-fast to false

807dd25

install right toolchain

974e8d9

fix cache key [skip ci]

cea5fa7

rename cargo lipo job [skip ci]

173157f

update ci working dir of xayn ai ffi [skip ci]

239afbe

add rust flags

84e5567

install target

9a1ee1e

use env.RUST_STABLE version [skip ci]

a5ad5a0

enable deny on warnings [skip ci]

f3f298b

remove ndk related code [skip ci]

930a5bd

print test execution time [skip ci]

31eb2b0

simplify cargo lipo install

cc911bd

link pr link to ci file [skip ci]

262dd27

add nl

c4f5ac0

Update .github/workflows/ci.yml [skip ci]

307cba3

Co-authored-by: janpetschexain <58227040+janpetschexain@users.noreply.github.com>

remove todos

7d11453

checksum binary

a98fc00

fmt

edd2e49

Robert-Steiner force-pushed the ffi-ci branch from 6d69e28 to edd2e49 Compare April 8, 2021 12:28

Robert-Steiner merged commit 3845553 into master Apr 8, 2021

Robert-Steiner deleted the ffi-ci branch April 8, 2021 12:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CI build mobile libs #21

CI build mobile libs #21

Uh oh!

Robert-Steiner commented Mar 9, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janpetschexain commented Mar 25, 2021

Uh oh!

Robert-Steiner commented Mar 26, 2021

Uh oh!

janpetschexain left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CI build mobile libs #21

CI build mobile libs #21

Uh oh!

Conversation

Robert-Steiner commented Mar 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation details

Use a build matrix for testing/building android/ios libraries on different cpu architectures

Optimised android test

Using cross for testing

Using cargo-ndk for building the release binaries

Why we chose RUSTFLAGS instead of [profile.test]

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janpetschexain commented Mar 25, 2021

Uh oh!

Robert-Steiner commented Mar 26, 2021

Uh oh!

janpetschexain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Robert-Steiner commented Mar 9, 2021 •

edited

Loading

Using `cross` for testing

Using `cargo-ndk` for building the release binaries

Why we chose `RUSTFLAGS` instead of `[profile.test]`