Removing Rust Dependency Bloat

August 20, 2020

One aspect that always bothered me with rust projects are the seemingly huge number of depdencies that get built whenever you do a fresh build. This post is just a log of what I did to lower compile times of my open source encrypted backup tool bupstash and reduce my dependency burden.

First, lets get a starting point on dependencies and compile times.

The full dependency tree:

$ cargo tree
bupstash v0.1.0 (/home/ac/src/bupstash)
├── cfg-if v0.1.10
├── chrono v0.4.13
│   ├── num-integer v0.1.43
│   │   └── num-traits v0.2.12
│   │       [build-dependencies]
│   │       └── autocfg v1.0.0
│   │   [build-dependencies]
│   │   └── autocfg v1.0.0
│   ├── num-traits v0.2.12 (*)
│   ├── serde v1.0.114
│   │   └── serde_derive v1.0.114
│   │       ├── proc-macro2 v1.0.18
│   │       │   └── unicode-xid v0.2.0
│   │       ├── quote v1.0.7
│   │       │   └── proc-macro2 v1.0.18 (*)
│   │       └── syn v1.0.33
│   │           ├── proc-macro2 v1.0.18 (*)
│   │           ├── quote v1.0.7 (*)
│   │           └── unicode-xid v0.2.0
│   └── time v0.1.43
│       └── libc v0.2.71
├── codemap v0.1.3
├── codemap-diagnostic v0.1.1
│   ├── atty v0.2.14
│   │   └── libc v0.2.71
│   ├── codemap v0.1.3
│   └── termcolor v1.1.0
├── crossbeam v0.7.3
│   ├── cfg-if v0.1.10
│   ├── crossbeam-channel v0.4.2
│   │   ├── crossbeam-utils v0.7.2
│   │   │   ├── cfg-if v0.1.10
│   │   │   └── lazy_static v1.4.0
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.0.0
│   │   └── maybe-uninit v2.0.0
│   ├── crossbeam-deque v0.7.3
│   │   ├── crossbeam-epoch v0.8.2
│   │   │   ├── cfg-if v0.1.10
│   │   │   ├── crossbeam-utils v0.7.2 (*)
│   │   │   ├── lazy_static v1.4.0
│   │   │   ├── maybe-uninit v2.0.0
│   │   │   ├── memoffset v0.5.4
│   │   │   │   [build-dependencies]
│   │   │   │   └── autocfg v1.0.0
│   │   │   └── scopeguard v1.1.0
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.0.0
│   │   ├── crossbeam-utils v0.7.2 (*)
│   │   └── maybe-uninit v2.0.0
│   ├── crossbeam-epoch v0.8.2 (*)
│   ├── crossbeam-queue v0.2.3
│   │   ├── cfg-if v0.1.10
│   │   ├── crossbeam-utils v0.7.2 (*)
│   │   └── maybe-uninit v2.0.0
│   └── crossbeam-utils v0.7.2 (*)
├── failure v0.1.8
│   ├── backtrace v0.3.49
│   │   ├── addr2line v0.12.2
│   │   │   └── gimli v0.21.0
│   │   ├── cfg-if v0.1.10
│   │   ├── libc v0.2.71
│   │   ├── miniz_oxide v0.3.7
│   │   │   └── adler32 v1.1.0
│   │   ├── object v0.20.0
│   │   └── rustc-demangle v0.1.16
│   └── failure_derive v0.1.8
│       ├── proc-macro2 v1.0.18 (*)
│       ├── quote v1.0.7 (*)
│       ├── syn v1.0.33 (*)
│       └── synstructure v0.12.4
│           ├── proc-macro2 v1.0.18 (*)
│           ├── quote v1.0.7 (*)
│           ├── syn v1.0.33 (*)
│           └── unicode-xid v0.2.0
├── fs2 v0.4.3
│   └── libc v0.2.71
├── getopts v0.2.21
│   └── unicode-width v0.1.7
├── glob v0.3.0
├── libc v0.2.71
├── nix v0.17.0
│   ├── bitflags v1.2.1
│   ├── cfg-if v0.1.10
│   ├── libc v0.2.71
│   └── void v1.0.2
├── parse_duration v2.1.0
│   ├── lazy_static v1.4.0
│   ├── num v0.2.1
│   │   ├── num-bigint v0.2.6
│   │   │   ├── num-integer v0.1.43 (*)
│   │   │   └── num-traits v0.2.12 (*)
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.0.0
│   │   ├── num-complex v0.2.4
│   │   │   └── num-traits v0.2.12 (*)
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.0.0
│   │   ├── num-integer v0.1.43 (*)
│   │   ├── num-iter v0.1.41
│   │   │   ├── num-integer v0.1.43 (*)
│   │   │   └── num-traits v0.2.12 (*)
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.0.0
│   │   ├── num-rational v0.2.4
│   │   │   ├── num-bigint v0.2.6 (*)
│   │   │   ├── num-integer v0.1.43 (*)
│   │   │   └── num-traits v0.2.12 (*)
│   │   │   [build-dependencies]
│   │   │   └── autocfg v1.0.0
│   │   └── num-traits v0.2.12 (*)
│   └── regex v1.3.9
│       ├── aho-corasick v0.7.12
│       │   └── memchr v2.3.3
│       ├── memchr v2.3.3
│       ├── regex-syntax v0.6.18
│       └── thread_local v1.0.1
│           └── lazy_static v1.4.0
├── path-clean v0.1.0
├── pem v0.8.1
│   ├── base64 v0.12.2
│   ├── once_cell v1.4.0
│   └── regex v1.3.9 (*)
├── rand v0.7.3
│   ├── getrandom v0.1.14
│   │   ├── cfg-if v0.1.10
│   │   └── libc v0.2.71
│   ├── libc v0.2.71
│   ├── rand_chacha v0.2.2
│   │   ├── ppv-lite86 v0.2.8
│   │   └── rand_core v0.5.1
│   │       └── getrandom v0.1.14 (*)
│   └── rand_core v0.5.1 (*)
├── regex v1.3.9 (*)
├── rusqlite v0.23.1
│   ├── bitflags v1.2.1
│   ├── fallible-iterator v0.2.0
│   ├── fallible-streaming-iterator v0.1.9
│   ├── libsqlite3-sys v0.18.0
│   │   [build-dependencies]
│   │   └── pkg-config v0.3.17
│   ├── lru-cache v0.1.2
│   │   └── linked-hash-map v0.5.3
│   ├── memchr v2.3.3
│   ├── smallvec v1.4.1
│   └── time v0.1.43 (*)
├── serde v1.0.114 (*)
├── serde_bare v0.2.0 (https://git.sr.ht/~tdeo/serde_bare?rev=2e330a7e37ca433c515d2a05128308fc98335300#2e330a7e)
│   └── serde v1.0.114 (*)
├── serde_json v1.0.55
│   ├── itoa v0.4.6
│   ├── ryu v1.0.5
│   └── serde v1.0.114 (*)
├── shlex v0.1.1
├── tar v0.4.29
│   ├── filetime v0.2.10
│   │   ├── cfg-if v0.1.10
│   │   └── libc v0.2.71
│   ├── libc v0.2.71
│   └── xattr v0.2.2
│       └── libc v0.2.71
├── tempfile v3.1.0
│   ├── cfg-if v0.1.10
│   ├── libc v0.2.71
│   ├── rand v0.7.3 (*)
│   └── remove_dir_all v0.5.3
├── walkdir v2.3.1
│   └── same-file v1.0.6
└── zstd v0.5.3+zstd.1.4.5
    └── zstd-safe v2.0.5+zstd.1.4.5
        ├── libc v0.2.71
        └── zstd-sys v1.4.17+zstd.1.4.5
            └── libc v0.2.71
            [build-dependencies]
            ├── cc v1.0.54
            │   └── jobserver v0.1.21
            │       └── libc v0.2.71
            ├── glob v0.3.0
            └── itertools v0.9.0
                └── either v1.5.3
[build-dependencies]
├── bindgen v0.53.3
│   ├── bitflags v1.2.1
│   ├── cexpr v0.4.0
│   │   └── nom v5.1.2
│   │       └── memchr v2.3.3
│   │       [build-dependencies]
│   │       └── version_check v0.9.2
│   ├── cfg-if v0.1.10
│   ├── clang-sys v0.29.3
│   │   ├── glob v0.3.0
│   │   ├── libc v0.2.71
│   │   └── libloading v0.5.2
│   │       [build-dependencies]
│   │       └── cc v1.0.54 (*)
│   │   [build-dependencies]
│   │   └── glob v0.3.0
│   ├── clap v2.33.1
│   │   ├── ansi_term v0.11.0
│   │   ├── atty v0.2.14 (*)
│   │   ├── bitflags v1.2.1
│   │   ├── strsim v0.8.0
│   │   ├── textwrap v0.11.0
│   │   │   └── unicode-width v0.1.7
│   │   ├── unicode-width v0.1.7
│   │   └── vec_map v0.8.2
│   ├── env_logger v0.7.1
│   │   ├── atty v0.2.14 (*)
│   │   ├── humantime v1.3.0
│   │   │   └── quick-error v1.2.3
│   │   ├── log v0.4.8
│   │   │   └── cfg-if v0.1.10
│   │   ├── regex v1.3.9 (*)
│   │   └── termcolor v1.1.0
│   ├── lazy_static v1.4.0
│   ├── lazycell v1.2.1
│   ├── log v0.4.8 (*)
│   ├── peeking_take_while v0.1.2
│   ├── proc-macro2 v1.0.18 (*)
│   ├── quote v1.0.7 (*)
│   ├── regex v1.3.9 (*)
│   ├── rustc-hash v1.1.0
│   ├── shlex v0.1.1
│   └── which v3.1.1
│       └── libc v0.2.71
└── pkg-config v0.3.17

A full project build:

$ time cargo build
...
real    0m30.061s
user    5m58.615s
sys     0m23.546s

Wow! a total of 171 dependencies built, how did I accumulate so much bloat!?

Remove unused dependencies

Doing some simple grepping of the code, I can see ‘walkdir’ is not used, and I also seem to be pulling in all of crossbeam but only using crossbeam channels. We should be

Simply replacing crossbeam with crossbeam-channel, and removing walkdir and we drop to 159 dependencies and lower our build time by ~ 10 cpu seconds.

$ time cargo build
...
real    0m30.515s
user    5m49.444s
sys     0m22.758s

Pregenerating C bindings

We link against a C library called libsodium for fast encryption primitives and are generating rust bindings for these using the rust bindgen package. Because we only call a small number of stable functions, we shouldn’t need to do this every time, instead we should be able to just check our generated bindings into version control.

For this step I removed binding generation from the build.rs file, and instead added a small shell script to generate the bindings manually when needed.

The result:

$ time cargo build
...
real    0m28.132s
user    4m41.921s
sys     0m19.409s

A whole minute off our total build time, and down to 130 dependencies. Even better, our tool no longer depends on the very complex libclang at build time anymore.

Separate dev dependencies

Shifting the tempdir dependency from the [dependencies] to [dev.dependencies] section in our Cargo.toml file removes a build dependency, as it is only built when running unit tests.

Remove duplicate functionality

My libsodium bindings provided a random number generator, so with minor refactoring I was able to use that instead of the rand module.

$ time cargo build
...
real    0m27.801s
user    4m36.618s
sys     0m17.914s

This change brought the project down to 121 build time dependencies.

Choose simpler dependencies

I noticed the parse_duration dependency depended on a “big number” library, it seemed unlikely for my simple use cases that I would need arbitrary precision durations, so decided to change to an alternative library humantime. The code changes required were minimal.

$ time cargo build
...
real    0m26.794s
user    3m55.898s
sys     0m15.024s

This brought a clean build down to 108 dependencies.

Summary and results

The final fresh build time of bupstash and it’s dependencies for a single cpu went from 5m58.615s down to 3m55.898s and the build time dependencies count went from 171 down to 108.

Overall I feel like this was a decent result, as each second we shave off will be a blessing for our users and people wanting to package our software.

I feel there are still gains to be made by avoiding serde when we can, and also removing the failure crate (which is deprecated anyway), but it is good enough progress for now.

Some lessons to take away for your own rust projects:

Check for and remove unused dependencies, cargo still builds them and doesn’t seem to warn you.
Avoid complicated logic in your build.rs such as bindgen invocations if you can.
Separate your dev dependencies into the [dev.dependencies] section of your cargo.toml so end users don’t need to build them unconditionally.
Look for places where you use similar functionality from multiple dependencies and try to refactor one of the dependencies away.
Choose the simpler dependency when you have a choice between a few alternatives that meet your requirements.
Avoid procedural macros if you have a simpler alternative.

The final dependency tree for completeness:

bupstash v0.1.0 (/home/ac/src/bupstash)
├── cfg-if v0.1.10
├── chrono v0.4.13
│   ├── num-integer v0.1.43
│   │   └── num-traits v0.2.12
│   │       [build-dependencies]
│   │       └── autocfg v1.0.0
│   │   [build-dependencies]
│   │   └── autocfg v1.0.0
│   ├── num-traits v0.2.12 (*)
│   ├── serde v1.0.114
│   │   └── serde_derive v1.0.114
│   │       ├── proc-macro2 v1.0.18
│   │       │   └── unicode-xid v0.2.0
│   │       ├── quote v1.0.7
│   │       │   └── proc-macro2 v1.0.18 (*)
│   │       └── syn v1.0.33
│   │           ├── proc-macro2 v1.0.18 (*)
│   │           ├── quote v1.0.7 (*)
│   │           └── unicode-xid v0.2.0
│   └── time v0.1.43
│       └── libc v0.2.71
├── codemap v0.1.3
├── codemap-diagnostic v0.1.1
│   ├── atty v0.2.14
│   │   └── libc v0.2.71
│   ├── codemap v0.1.3
│   └── termcolor v1.1.0
├── crossbeam-channel v0.4.2
│   ├── crossbeam-utils v0.7.2
│   │   ├── cfg-if v0.1.10
│   │   └── lazy_static v1.4.0
│   │   [build-dependencies]
│   │   └── autocfg v1.0.0
│   └── maybe-uninit v2.0.0
├── failure v0.1.8
│   ├── backtrace v0.3.49
│   │   ├── addr2line v0.12.2
│   │   │   └── gimli v0.21.0
│   │   ├── cfg-if v0.1.10
│   │   ├── libc v0.2.71
│   │   ├── miniz_oxide v0.3.7
│   │   │   └── adler32 v1.1.0
│   │   ├── object v0.20.0
│   │   └── rustc-demangle v0.1.16
│   └── failure_derive v0.1.8
│       ├── proc-macro2 v1.0.18 (*)
│       ├── quote v1.0.7 (*)
│       ├── syn v1.0.33 (*)
│       └── synstructure v0.12.4
│           ├── proc-macro2 v1.0.18 (*)
│           ├── quote v1.0.7 (*)
│           ├── syn v1.0.33 (*)
│           └── unicode-xid v0.2.0
├── fs2 v0.4.3
│   └── libc v0.2.71
├── getopts v0.2.21
│   └── unicode-width v0.1.7
├── glob v0.3.0
├── humantime v2.0.1
├── libc v0.2.71
├── nix v0.17.0
│   ├── bitflags v1.2.1
│   ├── cfg-if v0.1.10
│   ├── libc v0.2.71
│   └── void v1.0.2
├── path-clean v0.1.0
├── pem v0.8.1
│   ├── base64 v0.12.2
│   ├── once_cell v1.4.0
│   └── regex v1.3.9
│       ├── aho-corasick v0.7.12
│       │   └── memchr v2.3.3
│       ├── memchr v2.3.3
│       ├── regex-syntax v0.6.18
│       └── thread_local v1.0.1
│           └── lazy_static v1.4.0
├── regex v1.3.9 (*)
├── rusqlite v0.23.1
│   ├── bitflags v1.2.1
│   ├── fallible-iterator v0.2.0
│   ├── fallible-streaming-iterator v0.1.9
│   ├── libsqlite3-sys v0.18.0
│   │   [build-dependencies]
│   │   └── pkg-config v0.3.17
│   ├── lru-cache v0.1.2
│   │   └── linked-hash-map v0.5.3
│   ├── memchr v2.3.3
│   ├── smallvec v1.4.1
│   └── time v0.1.43 (*)
├── serde v1.0.114 (*)
├── serde_bare v0.2.0 (https://git.sr.ht/~tdeo/serde_bare?rev=2e330a7e37ca433c515d2a05128308fc98335300#2e330a7e)
│   └── serde v1.0.114 (*)
├── serde_json v1.0.55
│   ├── itoa v0.4.6
│   ├── ryu v1.0.5
│   └── serde v1.0.114 (*)
├── shlex v0.1.1
├── tar v0.4.29
│   ├── filetime v0.2.10
│   │   ├── cfg-if v0.1.10
│   │   └── libc v0.2.71
│   ├── libc v0.2.71
│   └── xattr v0.2.2
│       └── libc v0.2.71
└── zstd v0.5.3+zstd.1.4.5
    └── zstd-safe v2.0.5+zstd.1.4.5
        ├── libc v0.2.71
        └── zstd-sys v1.4.17+zstd.1.4.5
            └── libc v0.2.71
            [build-dependencies]
            ├── cc v1.0.54
            │   └── jobserver v0.1.21
            │       └── libc v0.2.71
            ├── glob v0.3.0
            └── itertools v0.9.0
                └── either v1.5.3
[build-dependencies]
└── pkg-config v0.3.17
[dev-dependencies]
└── tempfile v3.1.0
    ├── cfg-if v0.1.10
    ├── libc v0.2.71
    ├── rand v0.7.3
    │   ├── getrandom v0.1.14
    │   │   ├── cfg-if v0.1.10
    │   │   └── libc v0.2.71
    │   ├── libc v0.2.71
    │   ├── rand_chacha v0.2.2
    │   │   ├── ppv-lite86 v0.2.8
    │   │   └── rand_core v0.5.1
    │   │       └── getrandom v0.1.14 (*)
    │   └── rand_core v0.5.1 (*)
    └── remove_dir_all v0.5.3

Thank you for reading :)