itsscb/cargo
1
0
Fork 0
mirror of https://github.com/rust-lang/cargo.git synced 2025-09-28 11:20:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

Auto merge of #12634 - ehuss:last-use, r=epage

Browse Source 
Add cache garbage collection

### What does this PR try to resolve?

This introduces a new garbage collection system which can track the last time files were used in cargo's global cache, and delete old, unused files either automatically or manually.

### How should we test and review this PR?

This is broken up into a large number of commits, and each commit should have a short overview of what it does. I am breaking some of these out into separate PRs as well (unfortunately GitHub doesn't really support stacked pull requests). I expect to reduce the size of this PR if those other PRs are accepted.

I would first review `unstable.md` to give you an idea of what the user side of this looks like. I would then skim over each commit message to give an overview of all the changes. The core change is the introduction of the `GlobalCacheTracker` which is an interface to a sqlite database which is used for tracking the timestamps.

### Additional information

I think the interface for this will almost certainly change over time. This is just a stab to create a starting point where we can start testing and discussing what actual user flags should be exposed. This is also intended to start the process of getting experience using sqlite, and getting some testing in real-world environments to see how things might fail.

I'd like to ask for the review to not focus too much on bikeshedding flag names and options. I expect them to change, so this is by no means a concrete proposal for where it will end up. For example, the options are very granular, and I would like to have fewer options. However, it isn't clear how that might best work. The size-tracking options almost certainly need to change, but I do not know exactly what the use cases for size-tracking are, so that will need some discussion with people who are interested in that.

I decided to place the gc commands in cargo's `cargo clean` command because I would like to have a single place for users to go for deleting cache artifacts. It may be possible that they get moved to another command, however introducing new subcommands is quite difficult (due to shadowing existing third-party commands). Other options might be `cargo gc`, `cargo maintenance`, `cargo cache`, etc. But there are existing extensions that would interfere with.

There are also more directions to go in the future. For example, we could add a `cargo clean info` subcommand which could be used for querying cache information (like the sizes and such). There is also the rest of the steps in the original proposal at https://hackmd.io/U_k79wk7SkCQ8_dJgIXwJg for rolling out sqlite support.

See #12633 for the tracking issue
This commit is contained in:
bors 2023-11-11 20:45:02 +00:00
commit 9a1b0924c6
37 changed files with 5704 additions and 38 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

91
Cargo.lock generated
View File

@ -8,6 +8,17 @@ version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe"
[[package]]
name = "ahash"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c99f64d1e06488f620f932677e24bc6e2897582980441ae90a671415bd7ec2f"
dependencies = [
"cfg-if",
"once_cell",
"version_check",
]
[[package]]
name = "aho-corasick"
version = "1.0.2"
@ -17,6 +28,12 @@ dependencies = [
"memchr",
]
[[package]]
name = "allocator-api2"
version = "0.2.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0942ffc6dcaadf03badf6e6a2d0228460359d5e34b57ccdc720b7382dfbd5ec5"
[[package]]
name = "anes"
version = "0.1.6"
@ -112,8 +129,10 @@ name = "benchsuite"
version = "0.0.0"
dependencies = [
"cargo",
"cargo-util",
"criterion",
"flate2",
"rand",
"tar",
"url",
]
@ -270,6 +289,8 @@ dependencies = [
"pathdiff",
"pulldown-cmark",
"rand",
"regex",
"rusqlite",
"rustfix",
"same-file",
"semver",
@ -389,6 +410,7 @@ dependencies = [
"time",
"toml",
"url",
"walkdir",
"windows-sys",
]
@ -400,6 +422,7 @@ dependencies = [
"core-foundation",
"filetime",
"hex",
"ignore",
"jobserver",
"libc",
"miow",
@ -885,6 +908,18 @@ dependencies = [
"serde_json",
]
[[package]]
name = "fallible-iterator"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7"
[[package]]
name = "fallible-streaming-iterator"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
[[package]]
name = "faster-hex"
version = "0.8.1"
@ -1805,6 +1840,19 @@ name = "hashbrown"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c6201b9ff9fd90a5a3bac2e56a830d0caa509576f0e503818ee82c181b3437a"
dependencies = [
"ahash",
"allocator-api2",
]
[[package]]
name = "hashlink"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e8094feaf31ff591f651a2664fb9cfd92bba7a60ce3197265e9482ebe753c8f7"
dependencies = [
"hashbrown",
]
[[package]]
name = "hermit-abi"
@ -2059,6 +2107,17 @@ dependencies = [
"libc",
]
[[package]]
name = "libsqlite3-sys"
version = "0.26.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "afc22eff61b133b115c6e8c74e818c628d6d5e7a502afea6f64dee076dd94326"
dependencies = [
"cc",
"pkg-config",
"vcpkg",
]
[[package]]
name = "libssh2-sys"
version = "0.3.0"
@ -2614,7 +2673,7 @@ dependencies = [
"rand",
"rand_chacha",
"rand_xorshift",
"regex-syntax 0.7.2",
"regex-syntax 0.7.5",
"rusty-fork",
"tempfile",
"unarray",
@ -2742,13 +2801,14 @@ dependencies = [
[[package]]
name = "regex"
version = "1.8.4"
version = "1.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0ab3ca65655bb1e41f2a8c8cd662eb4fb035e67c3f78da1d61dffe89d07300f"
checksum = "697061221ea1b4a94a624f67d0ae2bfe4e22b8a17b6a192afb11046542cc8c47"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax 0.7.2",
"regex-automata 0.3.8",
"regex-syntax 0.7.5",
]
[[package]]
@ -2765,6 +2825,11 @@ name = "regex-automata"
version = "0.3.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2f401f4955220693b56f8ec66ee9c78abffd8d1c4f23dc41a23839eb88f0795"
dependencies = [
"aho-corasick",
"memchr",
"regex-syntax 0.7.5",
]
[[package]]
name = "regex-syntax"
@ -2774,9 +2839,9 @@ checksum = "f162c6dd7b008981e4d40210aca20b4bd0f9b60ca9271061b07f78537722f2e1"
[[package]]
name = "regex-syntax"
version = "0.7.2"
version = "0.7.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "436b050e76ed2903236f032a59761c1eb99e1b0aead2c257922771dab1fc8c78"
checksum = "dbb5fb1acd8a1a18b3dd5be62d25485eb770e05afb408a9627d14d451bae12da"
[[package]]
name = "resolver-tests"
@ -2798,6 +2863,20 @@ dependencies = [
"subtle",
]
[[package]]
name = "rusqlite"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "549b9d036d571d42e6e85d1c1425e2ac83491075078ca9a15be021c56b1641f2"
dependencies = [
"bitflags 2.4.0",
"fallible-iterator",
"fallible-streaming-iterator",
"hashlink",
"libsqlite3-sys",
"smallvec",
]
[[package]]
name = "rustc-hash"
version = "1.1.0"

4
Cargo.toml
View File

@ -73,6 +73,8 @@ pretty_assertions = "1.4.0"
proptest = "1.3.1"
pulldown-cmark = { version = "0.9.3", default-features = false }
rand = "0.8.5"
regex = "1.9.3"
rusqlite = { version = "0.29.0", features = ["bundled"] }
rustfix = "0.6.1"
same-file = "1.0.6"
security-framework = "2.9.2"
@ -162,6 +164,8 @@ pasetors.workspace = true
pathdiff.workspace = true
pulldown-cmark.workspace = true
rand.workspace = true
regex.workspace = true
rusqlite.workspace = true
rustfix.workspace = true
semver.workspace = true
serde = { workspace = true, features = ["derive"] }

39
benches/README.md
View File

@ -9,7 +9,23 @@ cd benches/benchsuite
cargo bench
```
The tests involve downloading the index and benchmarking against some
However, running all benchmarks would take many minutes, so in most cases it
is recommended to just run the benchmarks relevant to whatever section of code
you are working on.
## Benchmarks
There are several different kinds of benchmarks in the `benchsuite/benches` directory:
* `global_cache_tracker` — Benchmarks saving data to the global cache tracker
database using samples of real-world data.
* `resolve` — Benchmarks the resolver against simulations of real-world workspaces.
* `workspace_initialization` — Benchmarks initialization of a workspace
against simulations of real-world workspaces.
### Resolve benchmarks
The resolve benchmarks involve downloading the index and benchmarking against some
real-world and artificial workspaces located in the [`workspaces`](workspaces)
directory.
@ -21,7 +37,7 @@ faster. You can (and probably should) specify individual benchmarks to run to
narrow it down to a more reasonable set, for example:
```sh
cargo bench -- resolve_ws/rust
cargo bench -p benchsuite --bench resolve -- resolve_ws/rust
```
This will only download what's necessary for the rust-lang/rust workspace
@ -29,7 +45,24 @@ This will only download what's necessary for the rust-lang/rust workspace
about a minute). To get a list of all the benchmarks, run:
```sh
cargo bench -- --list
cargo bench -p benchsuite --bench resolve -- --list
```
### Global cache tracker
The `global_cache_tracker` benchmark tests saving data to the global cache
tracker database using samples of real-world data. This benchmark should run
relatively quickly.
The real-world data is based on a capture of my personal development
environment which has accumulated a large cache. So it is somewhat arbitrary,
but hopefully representative of a challenging environment. Capturing of the
data is done with the `capture-last-use` binary, which you can run if you need
to rebuild the database. Just try to run on a system with a relatively full
cache in your cargo home directory.
```sh
cargo bench -p benchsuite --bench global_cache_tracker
```
## Viewing reports

6
benches/benchsuite/Cargo.toml
View File

@ -11,8 +11,10 @@ publish = false
[dependencies]
cargo.workspace = true
cargo-util.workspace = true
criterion.workspace = true
flate2.workspace = true
rand.workspace = true
tar.workspace = true
url.workspace = true
@ -26,3 +28,7 @@ harness = false
[[bench]]
name = "workspace_initialization"
harness = false
[[bench]]
name = "global_cache_tracker"
harness = false

159
benches/benchsuite/benches/global_cache_tracker.rs Normal file
View File

@ -0,0 +1,159 @@
//! Benchmarks for the global cache tracker.
use cargo::core::global_cache_tracker::{self, DeferredGlobalLastUse, GlobalCacheTracker};
use cargo::util::cache_lock::CacheLockMode;
use cargo::util::interning::InternedString;
use cargo::util::Config;
use criterion::{criterion_group, criterion_main, Criterion};
use std::fs;
use std::path::{Path, PathBuf};
// Samples of real-world data.
const GLOBAL_CACHE_SAMPLE: &str = "global-cache-tracker/global-cache-sample";
const GLOBAL_CACHE_RANDOM: &str = "global-cache-tracker/random-sample";
/// A scratch directory where the benchmark can place some files.
fn root() -> PathBuf {
let mut p = PathBuf::from(env!("CARGO_TARGET_TMPDIR"));
p.push("bench_global_cache_tracker");
p
}
fn cargo_home() -> PathBuf {
let mut p = root();
p.push("chome");
p
}
fn initialize_config() -> Config {
// Set up config.
let shell = cargo::core::Shell::new();
let homedir = cargo_home();
if !homedir.exists() {
fs::create_dir_all(&homedir).unwrap();
}
let cwd = homedir.clone();
let mut config = Config::new(shell, cwd, homedir);
config.nightly_features_allowed = true;
config.set_search_stop_path(root());
config
.configure(
0,
false,
None,
false,
false,
false,
&None,
&["gc".to_string()],
&[],
)
.unwrap();
// Set up database sample.
let db_path = GlobalCacheTracker::db_path(&config).into_path_unlocked();
if db_path.exists() {
fs::remove_file(&db_path).unwrap();
}
let sample = Path::new(env!("CARGO_MANIFEST_DIR")).join(GLOBAL_CACHE_SAMPLE);
fs::copy(sample, &db_path).unwrap();
config
}
/// Benchmarks how long it takes to initialize `GlobalCacheTracker` with an already
/// existing full database.
fn global_tracker_init(c: &mut Criterion) {
let config = initialize_config();
let _lock = config
.acquire_package_cache_lock(CacheLockMode::DownloadExclusive)
.unwrap();
c.bench_function("global_tracker_init", |b| {
b.iter(|| {
GlobalCacheTracker::new(&config).unwrap();
})
});
}
/// Benchmarks how long it takes to save a `GlobalCacheTracker` when there are zero
/// updates.
fn global_tracker_empty_save(c: &mut Criterion) {
let config = initialize_config();
let _lock = config
.acquire_package_cache_lock(CacheLockMode::DownloadExclusive)
.unwrap();
let mut deferred = DeferredGlobalLastUse::new();
let mut tracker = GlobalCacheTracker::new(&config).unwrap();
c.bench_function("global_tracker_empty_save", |b| {
b.iter(|| {
deferred.save(&mut tracker).unwrap();
})
});
}
fn load_random_sample() -> Vec<(InternedString, InternedString, u64)> {
let path = Path::new(env!("CARGO_MANIFEST_DIR")).join(GLOBAL_CACHE_RANDOM);
fs::read_to_string(path)
.unwrap()
.lines()
.map(|s| {
let mut s = s.split(',');
(
s.next().unwrap().into(),
s.next().unwrap().into(),
s.next().unwrap().parse().unwrap(),
)
})
.collect()
}
/// Tests performance of updating the last-use timestamps in an already
/// populated database.
///
/// This runs for different sizes of number of crates to update (selecting
/// from the random sample stored on disk).
fn global_tracker_update(c: &mut Criterion) {
let config = initialize_config();
let _lock = config
.acquire_package_cache_lock(CacheLockMode::DownloadExclusive)
.unwrap();
let sample = Path::new(env!("CARGO_MANIFEST_DIR")).join(GLOBAL_CACHE_SAMPLE);
let db_path = GlobalCacheTracker::db_path(&config).into_path_unlocked();
let random_sample = load_random_sample();
let mut group = c.benchmark_group("global_tracker_update");
for size in [1, 10, 100, 500] {
if db_path.exists() {
fs::remove_file(&db_path).unwrap();
}
fs::copy(&sample, &db_path).unwrap();
let mut deferred = DeferredGlobalLastUse::new();
let mut tracker = GlobalCacheTracker::new(&config).unwrap();
group.bench_with_input(size.to_string(), &size, |b, &size| {
b.iter(|| {
for (encoded_registry_name, name, size) in &random_sample[..size] {
deferred.mark_registry_crate_used(global_cache_tracker::RegistryCrate {
encoded_registry_name: *encoded_registry_name,
crate_filename: format!("{}.crate", name).into(),
size: *size,
});
deferred.mark_registry_src_used(global_cache_tracker::RegistrySrc {
encoded_registry_name: *encoded_registry_name,
package_dir: *name,
size: Some(*size),
});
}
deferred.save(&mut tracker).unwrap();
})
});
}
}
criterion_group!(
benches,
global_tracker_init,
global_tracker_empty_save,
global_tracker_update
);
criterion_main!(benches);

BIN
benches/benchsuite/global-cache-tracker/global-cache-sample Normal file
View File

Binary file not shown.

500
benches/benchsuite/global-cache-tracker/random-sample Normal file
View File

@ -0,0 +1,500 @@
github.com-1ecc6299db9ec823,tungstenite-0.18.0,218740
github.com-1ecc6299db9ec823,integer-encoding-1.1.5,30672
github.com-1ecc6299db9ec823,tungstenite-0.14.0,315676
github.com-1ecc6299db9ec823,oxcable-0.5.1,163196
github.com-1ecc6299db9ec823,swc_ecma_transforms_typescript-0.32.0,245522
github.com-1ecc6299db9ec823,hyper-0.12.35,601153
github.com-1ecc6299db9ec823,resiter-0.4.0,59880
github.com-1ecc6299db9ec823,net2-0.2.37,115813
github.com-1ecc6299db9ec823,str_inflector-0.12.0,182460
github.com-1ecc6299db9ec823,derive_builder_macro-0.10.2,16441
github.com-1ecc6299db9ec823,smol_str-0.1.23,42436
github.com-1ecc6299db9ec823,wasm-bindgen-multi-value-xform-0.2.83,35347
github.com-1ecc6299db9ec823,time-macros-0.1.0,1620
github.com-1ecc6299db9ec823,unicode-bidi-0.3.7,140153
github.com-1ecc6299db9ec823,socket2-0.4.0,167295
github.com-1ecc6299db9ec823,ppv-lite86-0.2.10,125234
github.com-1ecc6299db9ec823,tracing-wasm-0.2.1,31449
github.com-1ecc6299db9ec823,eframe-0.19.0,158130
github.com-1ecc6299db9ec823,block-modes-0.7.0,42530
github.com-1ecc6299db9ec823,rangemap-0.1.11,144157
github.com-1ecc6299db9ec823,metal-0.23.1,1038699
github.com-1ecc6299db9ec823,os_str_bytes-6.0.1,86390
github.com-1ecc6299db9ec823,plotters-backend-0.3.4,53018
github.com-1ecc6299db9ec823,spidev-0.4.0,45301
github.com-1ecc6299db9ec823,axum-macros-0.2.3,102058
github.com-1ecc6299db9ec823,embedded-time-0.12.1,246450
github.com-1ecc6299db9ec823,envmnt-0.10.4,2328079
github.com-1ecc6299db9ec823,camino-1.1.1,133976
github.com-1ecc6299db9ec823,siphasher-0.3.5,46666
github.com-1ecc6299db9ec823,lexical-write-integer-0.8.5,388374
github.com-1ecc6299db9ec823,reqwest-0.11.14,686608
github.com-1ecc6299db9ec823,enum-map-2.4.1,51184
github.com-1ecc6299db9ec823,sentry-panic-0.29.0,18211
github.com-1ecc6299db9ec823,msf-srtp-0.2.0,73164
github.com-1ecc6299db9ec823,near-sandbox-utils-0.4.1,7543
github.com-1ecc6299db9ec823,ablescript-0.5.2,129318
github.com-1ecc6299db9ec823,apecs-derive-0.2.3,10620
github.com-1ecc6299db9ec823,libc-0.2.133,3417382
github.com-1ecc6299db9ec823,tracing-0.1.35,380627
github.com-1ecc6299db9ec823,serde-wasm-bindgen-0.3.1,55371
github.com-1ecc6299db9ec823,compiler_builtins-0.1.71,692853
github.com-1ecc6299db9ec823,mockito-0.7.2,1179718
github.com-1ecc6299db9ec823,tonic-0.5.2,420299
github.com-1ecc6299db9ec823,tracing-core-0.1.30,240058
github.com-1ecc6299db9ec823,tower-timeout-0.3.0-alpha.2,7486
github.com-1ecc6299db9ec823,js-intern-0.3.1,7026
github.com-1ecc6299db9ec823,json-ld-context-processing-0.12.1,78101
github.com-1ecc6299db9ec823,generic-array-0.14.6,67349
github.com-1ecc6299db9ec823,synstructure-0.12.3,93523
github.com-1ecc6299db9ec823,version-compare-0.0.10,74950
github.com-1ecc6299db9ec823,dirs-1.0.5,51075
github.com-1ecc6299db9ec823,worker-kv-0.5.1,67351
github.com-1ecc6299db9ec823,vsimd-0.8.0,170805
github.com-1ecc6299db9ec823,mockall-0.9.1,187734
github.com-1ecc6299db9ec823,nan-preserving-float-0.1.0,6341
github.com-1ecc6299db9ec823,wasmer-types-2.3.0,192436
github.com-1ecc6299db9ec823,sodiumoxide-0.2.7,5131115
github.com-1ecc6299db9ec823,tracing-attributes-0.1.11,74857
github.com-1ecc6299db9ec823,treediff-4.0.2,72588
github.com-1ecc6299db9ec823,wiggle-generate-5.0.0,103044
github.com-1ecc6299db9ec823,lapin-1.6.6,497368
github.com-1ecc6299db9ec823,cranelift-entity-0.93.1,114206
github.com-1ecc6299db9ec823,pcap-parser-0.13.3,184131
github.com-1ecc6299db9ec823,rustfft-5.1.1,1638221
github.com-1ecc6299db9ec823,string_cache-0.7.5,75074
github.com-1ecc6299db9ec823,maybe-uninit-2.0.0,38492
github.com-1ecc6299db9ec823,diesel_full_text_search-2.0.0,10179
github.com-1ecc6299db9ec823,quinn-proto-0.8.4,687565
github.com-1ecc6299db9ec823,semver-0.5.1,73365
github.com-1ecc6299db9ec823,rocket_http-0.5.0-rc.2,409939
github.com-1ecc6299db9ec823,dialoguer-0.7.1,95159
github.com-1ecc6299db9ec823,fallible_collections-0.4.5,244152
github.com-1ecc6299db9ec823,parking_lot_core-0.9.0,138932
github.com-1ecc6299db9ec823,relative-path-1.6.0,103315
github.com-1ecc6299db9ec823,lua52-sys-0.1.2,584054
github.com-1ecc6299db9ec823,actix-files-0.6.0,126121
github.com-1ecc6299db9ec823,crates-io-0.35.1,29498
github.com-1ecc6299db9ec823,sentry-backtrace-0.19.1,20268
github.com-1ecc6299db9ec823,text_unit-0.1.10,26100
github.com-1ecc6299db9ec823,ascii-1.0.0,143025
github.com-1ecc6299db9ec823,crossbeam-utils-0.8.6,169542
github.com-1ecc6299db9ec823,nelf-0.1.0,28868
github.com-1ecc6299db9ec823,colorsys-0.6.5,86989
github.com-1ecc6299db9ec823,enum-iterator-1.2.0,31042
github.com-1ecc6299db9ec823,ansi-str-0.7.2,111689
github.com-1ecc6299db9ec823,anyhow-1.0.68,209123
github.com-1ecc6299db9ec823,gix-lock-5.0.1,65110
github.com-1ecc6299db9ec823,nom-supreme-0.8.0,147530
github.com-1ecc6299db9ec823,path-slash-0.1.4,28655
github.com-1ecc6299db9ec823,crates-io-0.35.0,29406
github.com-1ecc6299db9ec823,stb_truetype-0.2.8,22939
github.com-1ecc6299db9ec823,proc-macro2-1.0.50,185288
github.com-1ecc6299db9ec823,snapbox-0.4.1,169526
github.com-1ecc6299db9ec823,hyper-0.14.9,764075
github.com-1ecc6299db9ec823,ab_glyph-0.2.15,61722
github.com-1ecc6299db9ec823,uuid-0.1.18,47889
github.com-1ecc6299db9ec823,data-url-0.2.0,123480
github.com-1ecc6299db9ec823,threadpool-1.7.1,59558
github.com-1ecc6299db9ec823,thiserror-impl-1.0.29,65149
github.com-1ecc6299db9ec823,sha1-0.6.0,31102
github.com-1ecc6299db9ec823,tokio-tls-0.2.1,51467
github.com-1ecc6299db9ec823,locspan-derive-0.6.0,59360
github.com-1ecc6299db9ec823,ureq-1.5.1,249335
github.com-1ecc6299db9ec823,protoc-rust-2.24.1,13459
github.com-1ecc6299db9ec823,serde-1.0.159,509060
github.com-1ecc6299db9ec823,unescape-0.1.0,6047
github.com-1ecc6299db9ec823,data-encoding-2.2.0,113191
github.com-1ecc6299db9ec823,bytestring-1.1.0,23705
github.com-1ecc6299db9ec823,ab_glyph_rasterizer-0.1.8,34773
github.com-1ecc6299db9ec823,syn-0.12.15,912964
github.com-1ecc6299db9ec823,reqwest-0.11.9,656209
github.com-1ecc6299db9ec823,rustls-0.17.0,903717
github.com-1ecc6299db9ec823,term_size-0.3.2,36226
github.com-1ecc6299db9ec823,ordered-float-3.1.0,91357
github.com-1ecc6299db9ec823,cookie-0.2.5,44912
github.com-1ecc6299db9ec823,debugid-0.8.0,44521
github.com-1ecc6299db9ec823,conrod-0.51.1,2154016
github.com-1ecc6299db9ec823,indexmap-1.6.1,247801
github.com-1ecc6299db9ec823,target-spec-1.3.1,68315
github.com-1ecc6299db9ec823,lexical-parse-integer-0.8.6,139671
github.com-1ecc6299db9ec823,time-0.1.38,131629
github.com-1ecc6299db9ec823,glib-macros-0.14.1,102959
github.com-1ecc6299db9ec823,metrics-macros-0.6.0,37750
github.com-1ecc6299db9ec823,structopt-0.3.12,224213
github.com-1ecc6299db9ec823,criterion-0.3.2,439241
github.com-1ecc6299db9ec823,lyon_path-0.17.7,186745
github.com-1ecc6299db9ec823,miette-5.5.0,312945
github.com-1ecc6299db9ec823,tokio-codec-0.2.0-alpha.6,118193
github.com-1ecc6299db9ec823,structopt-derive-0.4.14,84883
github.com-1ecc6299db9ec823,objekt-0.1.2,24191
github.com-1ecc6299db9ec823,sqlx-macros-0.5.7,110890
github.com-1ecc6299db9ec823,systemstat-0.1.10,127295
github.com-1ecc6299db9ec823,colorful-0.2.2,99698
github.com-1ecc6299db9ec823,quick-xml-0.20.0,645935
github.com-1ecc6299db9ec823,selinux-sys-0.6.2,27060
github.com-1ecc6299db9ec823,vsmtp-mail-parser-1.4.0-rc.10,137699
github.com-1ecc6299db9ec823,sec1-0.7.2,64870
github.com-1ecc6299db9ec823,nix-0.22.1,1161830
github.com-1ecc6299db9ec823,snow-0.9.0,2658286
github.com-1ecc6299db9ec823,per_test_directory_macros-0.1.0,2962
github.com-1ecc6299db9ec823,syn-helpers-0.4.3,58801
github.com-1ecc6299db9ec823,terminal_size-0.2.2,29633
github.com-1ecc6299db9ec823,bevy_hierarchy-0.7.0,41018
github.com-1ecc6299db9ec823,dynamic_reload-0.4.0,74455
github.com-1ecc6299db9ec823,http-signature-normalization-actix-0.5.0-beta.14,126857
github.com-1ecc6299db9ec823,http-body-0.4.1,24138
github.com-1ecc6299db9ec823,gix-index-0.13.0,207795
github.com-1ecc6299db9ec823,darling_macro-0.13.1,4156
github.com-1ecc6299db9ec823,serde_json-1.0.66,543072
github.com-1ecc6299db9ec823,minreq-1.4.1,41355
github.com-1ecc6299db9ec823,sct-0.6.1,60974
github.com-1ecc6299db9ec823,openssl-0.10.50,1173941
github.com-1ecc6299db9ec823,bevy_pbr-0.6.0,201163
github.com-1ecc6299db9ec823,security-framework-2.3.1,290512
github.com-1ecc6299db9ec823,pin-project-internal-0.4.30,128419
github.com-1ecc6299db9ec823,serde_yaml-0.7.5,158524
github.com-1ecc6299db9ec823,cid-0.3.2,17269
github.com-1ecc6299db9ec823,plotters-backend-0.3.0,51995
github.com-1ecc6299db9ec823,serde_yaml-0.8.12,179579
github.com-1ecc6299db9ec823,cosmwasm-schema-derive-1.1.9,34956
github.com-1ecc6299db9ec823,docopt-0.6.86,175553
github.com-1ecc6299db9ec823,git-testament-0.2.4,27685
github.com-1ecc6299db9ec823,htmlescape-0.3.1,143378
github.com-1ecc6299db9ec823,is_proc_translated-0.1.1,16533
github.com-1ecc6299db9ec823,futures-macro-0.3.4,33147
github.com-1ecc6299db9ec823,futures-intrusive-0.4.2,520476
github.com-1ecc6299db9ec823,rustix-0.35.13,1581355
github.com-1ecc6299db9ec823,glsl-layout-0.3.2,75515
github.com-1ecc6299db9ec823,darling-0.12.0,67446
github.com-1ecc6299db9ec823,blake3-0.1.5,394136
github.com-1ecc6299db9ec823,async-stripe-0.15.0,3157635
github.com-1ecc6299db9ec823,hbs-common-sys-0.2.1,1034
github.com-1ecc6299db9ec823,base58-0.1.0,7019
github.com-1ecc6299db9ec823,time-0.2.23,342720
github.com-1ecc6299db9ec823,memoffset-0.5.6,27595
github.com-1ecc6299db9ec823,colored-1.9.3,85161
github.com-1ecc6299db9ec823,lrpar-0.13.1,153317
github.com-1ecc6299db9ec823,clap-2.34.0,975823
github.com-1ecc6299db9ec823,chalk-engine-0.55.0,203718
github.com-1ecc6299db9ec823,cosmic-space-0.3.6,800331
github.com-1ecc6299db9ec823,syn-1.0.93,1886902
github.com-1ecc6299db9ec823,futures-core-0.3.5,43430
github.com-1ecc6299db9ec823,prost-derive-0.11.6,99428
github.com-1ecc6299db9ec823,toml_edit-0.15.0,491549
github.com-1ecc6299db9ec823,pcb-llvm-0.2.0,17328
github.com-1ecc6299db9ec823,rusticata-macros-2.1.0,35537
github.com-1ecc6299db9ec823,rustyline-with-hint-fix-10.1.0,548833
github.com-1ecc6299db9ec823,sharded-slab-0.1.1,239224
github.com-1ecc6299db9ec823,literally-0.1.3,20415
github.com-1ecc6299db9ec823,riff-1.0.1,20582
github.com-1ecc6299db9ec823,futures-macro-0.3.23,38691
github.com-1ecc6299db9ec823,criterion-0.3.1,431723
github.com-1ecc6299db9ec823,atty-0.2.14,14567
github.com-1ecc6299db9ec823,vergen-3.1.0,49089
github.com-1ecc6299db9ec823,peeking_take_while-0.1.2,18604
github.com-1ecc6299db9ec823,serde_derive-1.0.156,316173
github.com-1ecc6299db9ec823,geo-0.23.1,1022596
github.com-1ecc6299db9ec823,persy-1.4.3,778219
github.com-1ecc6299db9ec823,futures-lite-1.13.0,214632
github.com-1ecc6299db9ec823,ms_dtyp-0.0.3,44387
github.com-1ecc6299db9ec823,thiserror-1.0.33,66618
github.com-1ecc6299db9ec823,marksman_escape-0.1.2,587235
github.com-1ecc6299db9ec823,serde_derive-1.0.101,289156
github.com-1ecc6299db9ec823,gix-ref-0.29.0,214105
github.com-1ecc6299db9ec823,der-0.7.5,384316
github.com-1ecc6299db9ec823,promptly-0.3.0,35216
github.com-1ecc6299db9ec823,libc-0.2.115,3166629
github.com-1ecc6299db9ec823,ppv-lite86-0.1.2,33514
github.com-1ecc6299db9ec823,gfx-hal-0.6.0,254453
github.com-1ecc6299db9ec823,as-slice-0.1.3,20306
github.com-1ecc6299db9ec823,gpu-alloc-0.3.0,78823
github.com-1ecc6299db9ec823,arc-swap-0.4.8,167950
github.com-1ecc6299db9ec823,libusb1-sys-0.5.0,1458763
github.com-1ecc6299db9ec823,sysinfo-0.26.8,609932
github.com-1ecc6299db9ec823,refinery-macros-0.8.7,6514
github.com-1ecc6299db9ec823,assert_float_eq-1.1.3,38445
github.com-1ecc6299db9ec823,tinyvec-1.1.0,363582
github.com-1ecc6299db9ec823,predicates-1.0.7,1168580
github.com-1ecc6299db9ec823,pulldown-cmark-0.9.3,595681
github.com-1ecc6299db9ec823,aws-sigv4-0.46.0,97885
github.com-1ecc6299db9ec823,fastrand-1.5.0,39175
github.com-1ecc6299db9ec823,futures-channel-0.3.17,131816
github.com-1ecc6299db9ec823,usbd_scsi-0.1.0,172205
github.com-1ecc6299db9ec823,tinyvec-1.4.0,379505
github.com-1ecc6299db9ec823,structsy-0.5.1,513822
github.com-1ecc6299db9ec823,aws-sdk-ssm-0.21.0,9755619
github.com-1ecc6299db9ec823,pin-project-lite-0.1.1,63942
github.com-1ecc6299db9ec823,tokio-rustls-0.13.0,78252
github.com-1ecc6299db9ec823,tinyvec_macros-0.1.0,2912
github.com-1ecc6299db9ec823,extended_matrix_float-1.0.0,6233
github.com-1ecc6299db9ec823,displaydoc-0.2.3,68676
github.com-1ecc6299db9ec823,typed-arena-2.0.2,43549
github.com-1ecc6299db9ec823,cranelift-0.86.1,16294
github.com-1ecc6299db9ec823,modular-bitfield-impl-0.10.0,64389
github.com-1ecc6299db9ec823,schemafy_core-0.5.2,7696
github.com-1ecc6299db9ec823,sea-orm-macros-0.8.0,86930
github.com-1ecc6299db9ec823,core-foundation-sys-0.4.6,61859
github.com-1ecc6299db9ec823,move-symbol-pool-0.3.2,14473
github.com-1ecc6299db9ec823,glutin-0.25.1,300518
github.com-1ecc6299db9ec823,postcard-cobs-0.2.0,41524
github.com-1ecc6299db9ec823,quote-0.6.11,69636
github.com-1ecc6299db9ec823,encoding_rs-0.8.32,5022316
github.com-1ecc6299db9ec823,clap-2.32.0,946148
github.com-1ecc6299db9ec823,term-0.6.1,181220
github.com-1ecc6299db9ec823,enumset-1.0.12,85911
github.com-1ecc6299db9ec823,ctest2-0.4.1,100745
github.com-1ecc6299db9ec823,serde-xml-any-0.0.3,70554
github.com-1ecc6299db9ec823,proc-macro-hack-0.5.11,39025
github.com-1ecc6299db9ec823,remove_dir_all-0.5.1,23418
github.com-1ecc6299db9ec823,weezl-0.1.5,134218
github.com-1ecc6299db9ec823,windows_x86_64_gnullvm-0.42.1,3254874
github.com-1ecc6299db9ec823,rocket-0.5.0-rc.2,1225987
github.com-1ecc6299db9ec823,pin-project-0.4.27,282004
github.com-1ecc6299db9ec823,criterion-cycles-per-byte-0.1.3,18296
github.com-1ecc6299db9ec823,coco-0.1.1,107143
github.com-1ecc6299db9ec823,solana-bloom-1.15.1,22207
github.com-1ecc6299db9ec823,qoqo_calculator-1.1.1,163666
github.com-1ecc6299db9ec823,aes-gcm-0.9.4,381036
github.com-1ecc6299db9ec823,blowfish-0.9.1,39658
github.com-1ecc6299db9ec823,pango-0.14.3,258440
github.com-1ecc6299db9ec823,clap_derive-3.0.0,129105
github.com-1ecc6299db9ec823,content_inspector-0.2.4,27568
github.com-1ecc6299db9ec823,jsona-0.2.0,104104
github.com-1ecc6299db9ec823,gix-quote-0.4.3,32314
github.com-1ecc6299db9ec823,bcs-0.1.3,93194
github.com-1ecc6299db9ec823,statrs-0.14.0,681982
github.com-1ecc6299db9ec823,cw-controllers-0.16.0,32195
github.com-1ecc6299db9ec823,hyper-0.12.36,578470
github.com-1ecc6299db9ec823,argon2-0.4.1,112707
github.com-1ecc6299db9ec823,fraction-0.12.2,482976
github.com-1ecc6299db9ec823,quickcheck-0.7.2,89884
github.com-1ecc6299db9ec823,typetag-0.1.8,135149
github.com-1ecc6299db9ec823,object-0.20.0,916661
github.com-1ecc6299db9ec823,pest_derive-2.2.1,60318
github.com-1ecc6299db9ec823,coremidi-sys-3.1.0,40849
github.com-1ecc6299db9ec823,either-1.6.0,48881
github.com-1ecc6299db9ec823,tarpc-0.29.0,244416
github.com-1ecc6299db9ec823,num-integer-0.1.42,88403
github.com-1ecc6299db9ec823,oid-registry-0.6.0,46996
github.com-1ecc6299db9ec823,historian-3.0.11,23818
github.com-1ecc6299db9ec823,ui-sys-0.1.3,1784250
github.com-1ecc6299db9ec823,cranelift-frontend-0.92.0,166902
github.com-1ecc6299db9ec823,pin-project-lite-0.1.12,77882
github.com-1ecc6299db9ec823,piston2d-gfx_graphics-0.72.0,91826
github.com-1ecc6299db9ec823,stylist-macros-0.9.2,78647
github.com-1ecc6299db9ec823,valico-3.4.0,1394467
github.com-1ecc6299db9ec823,inventory-0.3.3,40329
github.com-1ecc6299db9ec823,wrapping_arithmetic-0.1.0,8774
github.com-1ecc6299db9ec823,serde-1.0.138,502921
github.com-1ecc6299db9ec823,ra_common-0.1.3,16920
github.com-1ecc6299db9ec823,markup5ever-0.10.0,213742
github.com-1ecc6299db9ec823,libp2p-core-0.20.1,460422
github.com-1ecc6299db9ec823,inout-0.1.2,40474
github.com-1ecc6299db9ec823,flatbuffers-23.1.21,103944
github.com-1ecc6299db9ec823,gdk-pixbuf-sys-0.10.0,42914
github.com-1ecc6299db9ec823,miniz_oxide-0.5.1,223551
github.com-1ecc6299db9ec823,merge-0.1.0,70214
github.com-1ecc6299db9ec823,pagecache-0.6.0,260742
github.com-1ecc6299db9ec823,ritelinked-0.3.2,142063
github.com-1ecc6299db9ec823,ethers-contract-1.0.2,589452
github.com-1ecc6299db9ec823,color_quant-1.1.0,21284
github.com-1ecc6299db9ec823,libykpers-sys-0.3.1,14270
github.com-1ecc6299db9ec823,cgmath-0.17.0,367702
github.com-1ecc6299db9ec823,clap-4.0.18,1096299
github.com-1ecc6299db9ec823,ears-0.5.1,165152
github.com-1ecc6299db9ec823,h2-0.2.5,765073
github.com-1ecc6299db9ec823,image-0.22.5,725576
github.com-1ecc6299db9ec823,digest-0.10.1,83013
github.com-1ecc6299db9ec823,js-sys-0.3.46,410849
github.com-1ecc6299db9ec823,psl-types-2.0.11,25329
github.com-1ecc6299db9ec823,apub-core-0.2.0,52434
github.com-1ecc6299db9ec823,thiserror-1.0.22,59077
github.com-1ecc6299db9ec823,num-complex-0.4.3,139539
github.com-1ecc6299db9ec823,autocfg-1.0.1,41521
github.com-1ecc6299db9ec823,amethyst_locale-0.15.3,4896
github.com-1ecc6299db9ec823,tokio-timer-0.2.11,167147
github.com-1ecc6299db9ec823,pipe-trait-0.2.1,11031
github.com-1ecc6299db9ec823,http-muncher-0.3.2,259101
github.com-1ecc6299db9ec823,thin-dst-1.1.0,46297
github.com-1ecc6299db9ec823,float-ord-0.2.0,21145
github.com-1ecc6299db9ec823,trust-dns-proto-0.21.2,1312809
github.com-1ecc6299db9ec823,ordered-multimap-0.4.3,178966
github.com-1ecc6299db9ec823,bitflags-0.4.0,33932
github.com-1ecc6299db9ec823,windows_x86_64_gnullvm-0.42.0,3240134
github.com-1ecc6299db9ec823,cargo-util-0.1.2,72189
github.com-1ecc6299db9ec823,serde_with_macros-1.5.2,72325
github.com-1ecc6299db9ec823,wasmer-2.3.0,529984
github.com-1ecc6299db9ec823,tokio-codec-0.1.2,30428
github.com-1ecc6299db9ec823,pico-args-0.5.0,54991
github.com-1ecc6299db9ec823,migformatting-0.1.1,1680
github.com-1ecc6299db9ec823,lexical-core-0.6.7,2382284
github.com-1ecc6299db9ec823,katex-wasmbind-0.10.0,274096
github.com-1ecc6299db9ec823,blender-armature-0.0.1,51371
github.com-1ecc6299db9ec823,twoway-0.2.1,129719
github.com-1ecc6299db9ec823,sha3-0.10.0,540582
github.com-1ecc6299db9ec823,ringbuf-0.2.8,92733
github.com-1ecc6299db9ec823,pest_meta-2.1.3,175833
github.com-1ecc6299db9ec823,selectme-macros-0.7.1,79130
github.com-1ecc6299db9ec823,secp256k1-sys-0.7.0,5303296
github.com-1ecc6299db9ec823,panic-probe-0.3.0,18841
github.com-1ecc6299db9ec823,ron-0.6.6,208755
github.com-1ecc6299db9ec823,defmt-macros-0.3.3,78405
github.com-1ecc6299db9ec823,winapi-x86_64-pc-windows-gnu-0.4.0,53158182
github.com-1ecc6299db9ec823,aph-0.2.0,30088
github.com-1ecc6299db9ec823,winnow-0.4.6,959730
github.com-1ecc6299db9ec823,syntex_syntax-0.54.0,1272567
github.com-1ecc6299db9ec823,prost-derive-0.11.9,99428
github.com-1ecc6299db9ec823,commoncrypto-sys-0.2.0,16095
github.com-1ecc6299db9ec823,yew-router-macro-0.15.0,42667
github.com-1ecc6299db9ec823,http-range-header-0.3.0,29647
github.com-1ecc6299db9ec823,crossbeam-queue-0.2.3,60131
github.com-1ecc6299db9ec823,slice-deque-0.3.0,271889
github.com-1ecc6299db9ec823,libc-0.2.65,2334946
github.com-1ecc6299db9ec823,minidom-0.14.0,102507
github.com-1ecc6299db9ec823,tokio-native-tls-0.3.0,60313
github.com-1ecc6299db9ec823,glam-0.17.3,1191013
github.com-1ecc6299db9ec823,semver-1.0.6,114819
github.com-1ecc6299db9ec823,cortex-m-rtfm-macros-0.5.1,112048
github.com-1ecc6299db9ec823,bitvec-1.0.0,1006982
github.com-1ecc6299db9ec823,gfx-backend-metal-0.6.5,660301
github.com-1ecc6299db9ec823,object-0.30.1,1467041
github.com-1ecc6299db9ec823,proc-macro-error-attr-0.4.11,18220
github.com-1ecc6299db9ec823,proteus-0.5.0,179567
github.com-1ecc6299db9ec823,crunchy-0.1.6,6678
github.com-1ecc6299db9ec823,once_cell-1.7.2,121632
github.com-1ecc6299db9ec823,rel-0.2.0,14524
github.com-1ecc6299db9ec823,lexical-core-0.7.5,2355166
github.com-1ecc6299db9ec823,windows_x86_64_gnu-0.42.1,10581222
github.com-1ecc6299db9ec823,thread_local-1.1.5,49409
github.com-1ecc6299db9ec823,openssl-sys-0.9.63,285709
github.com-1ecc6299db9ec823,simplelog-0.11.2,85170
github.com-1ecc6299db9ec823,thiserror-impl-1.0.25,55249
github.com-1ecc6299db9ec823,quanta-0.10.0,82241
github.com-1ecc6299db9ec823,vsmtp-common-1.4.0-rc.10,122740
github.com-1ecc6299db9ec823,tonic-0.1.0-alpha.6,302938
github.com-1ecc6299db9ec823,ecdsa-0.16.1,121203
github.com-1ecc6299db9ec823,deltae-0.3.0,2871017
github.com-1ecc6299db9ec823,phf_shared-0.11.1,30454
github.com-1ecc6299db9ec823,trustfall-rustdoc-adapter-22.5.2,5348192
github.com-1ecc6299db9ec823,mockall_derive-0.11.0,227736
github.com-1ecc6299db9ec823,wasm-bindgen-0.2.64,584320
github.com-1ecc6299db9ec823,sg-std-0.12.0,27020
github.com-1ecc6299db9ec823,chalk-ir-0.87.0,288472
github.com-1ecc6299db9ec823,environment-0.1.1,9957
github.com-1ecc6299db9ec823,crash-handler-0.3.3,125183
github.com-1ecc6299db9ec823,bindgen-0.59.2,958852
github.com-1ecc6299db9ec823,serde_path_to_error-0.1.7,101591
github.com-1ecc6299db9ec823,tinyvec-0.3.3,77508
github.com-1ecc6299db9ec823,precomputed-hash-0.1.1,2853
github.com-1ecc6299db9ec823,rustc-rayon-core-0.4.1,264995
github.com-1ecc6299db9ec823,gix-sec-0.6.2,57428
github.com-1ecc6299db9ec823,pistoncore-input-0.19.0,83490
github.com-1ecc6299db9ec823,gloo-utils-0.1.5,15602
github.com-1ecc6299db9ec823,redox_intelflash-0.1.3,28056
github.com-1ecc6299db9ec823,block2-0.2.0-alpha.6,39192
github.com-1ecc6299db9ec823,fastly-shared-0.9.1,19292
github.com-1ecc6299db9ec823,ibc-chain-registry-0.1.0,48243
github.com-1ecc6299db9ec823,socket2-0.4.4,205035
github.com-1ecc6299db9ec823,futures-channel-0.3.19,132274
github.com-1ecc6299db9ec823,structopt-0.3.16,217443
github.com-1ecc6299db9ec823,rusty-fork-0.2.2,64570
github.com-1ecc6299db9ec823,parking_lot_core-0.9.7,139601
github.com-1ecc6299db9ec823,async-lock-2.6.0,99844
github.com-1ecc6299db9ec823,bindgen-0.56.0,923373
github.com-1ecc6299db9ec823,quad-rand-0.2.1,9108
github.com-1ecc6299db9ec823,wasmflow-codec-0.10.0,12343
github.com-1ecc6299db9ec823,gix-0.38.0,883190
github.com-1ecc6299db9ec823,futures-macro-0.3.27,38519
github.com-1ecc6299db9ec823,portable-atomic-0.3.13,549649
github.com-1ecc6299db9ec823,portable-atomic-1.3.2,799707
github.com-1ecc6299db9ec823,bevy-crevice-derive-0.6.0,16165
github.com-1ecc6299db9ec823,gltf-json-0.15.2,118263
github.com-1ecc6299db9ec823,struple-impl-0.1.0,4096
github.com-1ecc6299db9ec823,annotate-snippets-0.9.1,153174
github.com-1ecc6299db9ec823,futures-core-0.3.28,46207
github.com-1ecc6299db9ec823,wezterm-bidi-0.2.2,361283
github.com-1ecc6299db9ec823,mildew-0.1.2,3002
github.com-1ecc6299db9ec823,bytecount-0.6.3,46567
github.com-1ecc6299db9ec823,numext-fixed-hash-core-0.1.6,7403
github.com-1ecc6299db9ec823,bytesize-1.1.0,34012
github.com-1ecc6299db9ec823,oxsdatatypes-0.1.0,174662
github.com-1ecc6299db9ec823,hostname-0.1.5,4811
github.com-1ecc6299db9ec823,io-lifetimes-1.0.4,207652
github.com-1ecc6299db9ec823,derive_builder_core-0.11.2,135502
github.com-1ecc6299db9ec823,ttf-parser-0.15.2,711615
github.com-1ecc6299db9ec823,tracing-opentelemetry-0.17.4,187675
github.com-1ecc6299db9ec823,ab_glyph_rasterizer-0.1.7,34278
github.com-1ecc6299db9ec823,bevy_diagnostic-0.6.0,14396
github.com-1ecc6299db9ec823,toml_datetime-0.5.0,34801
github.com-1ecc6299db9ec823,wasm-parser-0.1.7,39726
github.com-1ecc6299db9ec823,ppv-null-0.1.2,26098
github.com-1ecc6299db9ec823,ci_info-0.10.2,1197933
github.com-1ecc6299db9ec823,jobserver-0.1.21,72720
github.com-1ecc6299db9ec823,sentencepiece-sys-0.10.0,10055292
github.com-1ecc6299db9ec823,zstd-sys-2.0.1+zstd.1.5.2,3387955
github.com-1ecc6299db9ec823,byte-strings-proc_macros-0.2.2,7886
github.com-1ecc6299db9ec823,snapbox-0.4.11,193312
github.com-1ecc6299db9ec823,ron-0.6.4,198516
github.com-1ecc6299db9ec823,gix-object-0.28.0,102536
github.com-1ecc6299db9ec823,strum_macros-0.23.1,87403
github.com-1ecc6299db9ec823,defmt-0.3.2,93568
github.com-1ecc6299db9ec823,openssl-0.10.35,971227
github.com-1ecc6299db9ec823,gtk-sys-0.14.0,1376726
github.com-1ecc6299db9ec823,gpu-alloc-0.4.7,99476
github.com-1ecc6299db9ec823,colored-2.0.0,91075
github.com-1ecc6299db9ec823,fixedbitset-0.4.2,67872
github.com-1ecc6299db9ec823,argparse-0.2.2,95032
github.com-1ecc6299db9ec823,bevy_mod_raycast-0.6.2,456756
github.com-1ecc6299db9ec823,byte-strings-0.2.2,35209
github.com-1ecc6299db9ec823,mem_tools-0.1.0,937956
github.com-1ecc6299db9ec823,deno_core-0.167.0,11067700
github.com-1ecc6299db9ec823,rocksdb-0.19.0,628015
github.com-1ecc6299db9ec823,num-traits-0.2.12,231414
github.com-1ecc6299db9ec823,type-info-derive-0.2.0,56221
github.com-1ecc6299db9ec823,structopt-derive-0.3.4,68017
github.com-1ecc6299db9ec823,extendr-macros-0.3.1,49695
github.com-1ecc6299db9ec823,secret-cosmwasm-std-1.0.0,632711
github.com-1ecc6299db9ec823,skim-0.7.0,380243
github.com-1ecc6299db9ec823,serde-1.0.135,501463
github.com-1ecc6299db9ec823,lock_api-0.1.5,109183
github.com-1ecc6299db9ec823,cw-multi-test-0.16.2,445599
github.com-1ecc6299db9ec823,quote-1.0.10,120640
github.com-1ecc6299db9ec823,safemem-0.3.2,17382
github.com-1ecc6299db9ec823,gloo-dialogs-0.1.1,4653
github.com-1ecc6299db9ec823,dashmap-4.0.2,105438
github.com-1ecc6299db9ec823,oorandom-11.1.0,31893
github.com-1ecc6299db9ec823,polars-core-0.21.1,1678691
github.com-1ecc6299db9ec823,claxon-0.4.2,259276
github.com-1ecc6299db9ec823,cc-1.0.35,179169
github.com-1ecc6299db9ec823,cocoa-0.19.1,296083
github.com-1ecc6299db9ec823,tokio-1.9.0,2490393
github.com-1ecc6299db9ec823,gix-refspec-0.10.1,105495
github.com-1ecc6299db9ec823,futures-task-0.3.12,39561
github.com-1ecc6299db9ec823,sqlx-core-0.4.2,1064795
github.com-1ecc6299db9ec823,futures-task-0.3.14,39566
github.com-1ecc6299db9ec823,datastore_grpc-0.4.0,18233399
github.com-1ecc6299db9ec823,directories-4.0.1,74013
github.com-1ecc6299db9ec823,wgpu-hal-0.15.1,1201034
github.com-1ecc6299db9ec823,discard-1.0.4,14342
github.com-1ecc6299db9ec823,tinytga-0.1.0,102322
github.com-1ecc6299db9ec823,prost-types-0.10.1,126121
github.com-1ecc6299db9ec823,assert2-0.3.6,36145
github.com-1ecc6299db9ec823,syn-inline-mod-0.5.0,35740
github.com-1ecc6299db9ec823,bat-0.22.1,5407476
github.com-1ecc6299db9ec823,minidumper-child-0.1.0,32329
github.com-1ecc6299db9ec823,libp2p-kad-0.21.0,416675
github.com-1ecc6299db9ec823,asn1_der-0.6.3,1102166
github.com-1ecc6299db9ec823,h2-0.2.4,764682
github.com-1ecc6299db9ec823,ena-0.14.2,90713
github.com-1ecc6299db9ec823,prost-build-0.8.0,31248726
github.com-1ecc6299db9ec823,wasmer-compiler-cranelift-3.1.1,300456
github.com-1ecc6299db9ec823,gfx-hal-0.7.0,238750
github.com-1ecc6299db9ec823,nom-4.2.3,644514
github.com-1ecc6299db9ec823,os_str_bytes-2.4.0,52159
github.com-1ecc6299db9ec823,sourcemap-6.2.1,135303
github.com-1ecc6299db9ec823,actix-router-0.5.1,150753
github.com-1ecc6299db9ec823,markup5ever-0.9.0,229731
github.com-1ecc6299db9ec823,gloo-worker-0.2.1,31624
github.com-1ecc6299db9ec823,object-0.25.3,1313095
github.com-1ecc6299db9ec823,rustversion-1.0.0,41602

148
benches/benchsuite/src/bin/capture-last-use.rs Normal file
View File

@ -0,0 +1,148 @@
//! Utility for capturing a global cache last-use database based on the files
//! on a real-world system.
//!
//! This will look in the CARGO_HOME of the current system and record last-use
//! data for all files in the cache. This is intended to provide a real-world
//! example for a benchmark that should be close to what a real set of data
//! should look like.
//!
//! See `benches/global_cache_tracker.rs` for the benchmark that uses this
//! data.
//!
//! The database is kept in git. It usually shouldn't need to be re-generated
//! unless there is a change in the schema or the benchmark.
use cargo::core::global_cache_tracker::{self, DeferredGlobalLastUse, GlobalCacheTracker};
use cargo::util::cache_lock::CacheLockMode;
use cargo::util::interning::InternedString;
use cargo::Config;
use rand::prelude::SliceRandom;
use std::collections::HashMap;
use std::fs;
use std::fs::File;
use std::io::Write;
use std::path::Path;
fn main() {
// Set up config.
let shell = cargo::core::Shell::new();
let homedir = Path::new(env!("CARGO_MANIFEST_DIR")).join("global-cache-tracker");
let cwd = homedir.clone();
let mut config = Config::new(shell, cwd, homedir.clone());
config
.configure(
0,
false,
None,
false,
false,
false,
&None,
&["gc".to_string()],
&[],
)
.unwrap();
let db_path = GlobalCacheTracker::db_path(&config).into_path_unlocked();
if db_path.exists() {
fs::remove_file(&db_path).unwrap();
}
let _lock = config
.acquire_package_cache_lock(CacheLockMode::DownloadExclusive)
.unwrap();
let mut deferred = DeferredGlobalLastUse::new();
let mut tracker = GlobalCacheTracker::new(&config).unwrap();
let real_home = cargo::util::homedir(&std::env::current_dir().unwrap()).unwrap();
let cache_dir = real_home.join("registry/cache");
for dir_ent in fs::read_dir(cache_dir).unwrap() {
let registry = dir_ent.unwrap();
let encoded_registry_name = InternedString::new(&registry.file_name().to_string_lossy());
for krate in fs::read_dir(registry.path()).unwrap() {
let krate = krate.unwrap();
let meta = krate.metadata().unwrap();
deferred.mark_registry_crate_used_stamp(
global_cache_tracker::RegistryCrate {
encoded_registry_name,
crate_filename: krate.file_name().to_string_lossy().as_ref().into(),
size: meta.len(),
},
Some(&meta.modified().unwrap()),
);
}
}
let mut src_entries = Vec::new();
let cache_dir = real_home.join("registry/src");
for dir_ent in fs::read_dir(cache_dir).unwrap() {
let registry = dir_ent.unwrap();
let encoded_registry_name = InternedString::new(&registry.file_name().to_string_lossy());
for krate in fs::read_dir(registry.path()).unwrap() {
let krate = krate.unwrap();
let meta = krate.metadata().unwrap();
let src = global_cache_tracker::RegistrySrc {
encoded_registry_name,
package_dir: krate.file_name().to_string_lossy().as_ref().into(),
size: Some(cargo_util::du(&krate.path(), &[]).unwrap()),
};
src_entries.push(src.clone());
let timestamp = meta.modified().unwrap();
deferred.mark_registry_src_used_stamp(src, Some(&timestamp));
}
}
let git_co_dir = real_home.join("git/checkouts");
for dir_ent in fs::read_dir(git_co_dir).unwrap() {
let git_source = dir_ent.unwrap();
let encoded_git_name = InternedString::new(&git_source.file_name().to_string_lossy());
for co in fs::read_dir(git_source.path()).unwrap() {
let co = co.unwrap();
let meta = co.metadata().unwrap();
deferred.mark_git_checkout_used_stamp(
global_cache_tracker::GitCheckout {
encoded_git_name,
short_name: co.file_name().to_string_lossy().as_ref().into(),
size: Some(cargo_util::du(&co.path(), &[]).unwrap()),
},
Some(&meta.modified().unwrap()),
);
}
}
deferred.save(&mut tracker).unwrap();
drop(deferred);
drop(tracker);
fs::rename(&db_path, homedir.join("global-cache-sample")).unwrap();
// Clean up the lock file created above.
fs::remove_file(homedir.join(".package-cache")).unwrap();
// Save a random sample of crates that the benchmark should update.
// Pick whichever registry has the most entries. This is to be somewhat
// realistic for the common case that all dependencies come from one
// registry (crates.io).
let mut counts = HashMap::new();
for src in &src_entries {
let c: &mut u32 = counts.entry(src.encoded_registry_name).or_default();
*c += 1;
}
let mut counts: Vec<_> = counts.into_iter().map(|(k, v)| (v, k)).collect();
counts.sort();
let biggest = counts.last().unwrap().1;
src_entries.retain(|src| src.encoded_registry_name == biggest);
let mut rng = &mut rand::thread_rng();
let sample: Vec<_> = src_entries.choose_multiple(&mut rng, 500).collect();
let mut f = File::create(homedir.join("random-sample")).unwrap();
for src in sample {
writeln!(
f,
"{},{},{}",
src.encoded_registry_name,
src.package_dir,
src.size.unwrap()
)
.unwrap();
}
}

1
crates/cargo-test-support/Cargo.toml
View File

@ -29,6 +29,7 @@ tar.workspace = true
time.workspace = true
toml.workspace = true
url.workspace = true
walkdir.workspace = true
[target.'cfg(windows)'.dependencies]
windows-sys = { workspace = true, features = ["Win32_Storage_FileSystem"] }

12
crates/cargo-test-support/src/paths.rs
View File

@ -114,6 +114,10 @@ pub trait CargoPathExt {
fn rm_rf(&self);
fn mkdir_p(&self);
/// Returns a list of all files and directories underneath the given
/// directory, recursively, including the starting path.
fn ls_r(&self) -> Vec<PathBuf>;
fn move_into_the_past(&self) {
self.move_in_time(|sec, nsec| (sec - 3600, nsec))
}
@ -155,6 +159,14 @@ impl CargoPathExt for Path {
.unwrap_or_else(|e| panic!("failed to mkdir_p {}: {}", self.display(), e))
}
fn ls_r(&self) -> Vec<PathBuf> {
walkdir::WalkDir::new(self)
.sort_by_file_name()
.into_iter()
.filter_map(|e| e.map(|e| e.path().to_owned()).ok())
.collect()
}
fn move_in_time<F>(&self, travel_amount: F)
where
F: Fn(i64, u32) -> (i64, u32),

1
crates/cargo-util/Cargo.toml
View File

@ -12,6 +12,7 @@ description = "Miscellaneous support code used by Cargo."
anyhow.workspace = true
filetime.workspace = true
hex.workspace = true
ignore.workspace = true
jobserver.workspace = true
libc.workspace = true
same-file.workspace = true

77
crates/cargo-util/src/du.rs Normal file
View File

@ -0,0 +1,77 @@
//! A simple disk usage estimator.
use anyhow::{Context, Result};
use ignore::overrides::OverrideBuilder;
use ignore::{WalkBuilder, WalkState};
use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
/// Determines the disk usage of all files in the given directory.
///
/// The given patterns are gitignore style patterns relative to the given
/// path. If there are patterns, it will only count things matching that
/// pattern. `!` can be used to exclude things. See [`OverrideBuilder::add`]
/// for more info.
///
/// This is a primitive implementation that doesn't handle hard links, and
/// isn't particularly fast (for example, not using `getattrlistbulk` on
/// macOS). It also only uses actual byte sizes instead of block counts (and
/// thus vastly undercounts directories with lots of small files). It would be
/// nice to improve this or replace it with something better.
pub fn du(path: &Path, patterns: &[&str]) -> Result<u64> {
du_inner(path, patterns).with_context(|| format!("failed to walk `{}`", path.display()))
}
fn du_inner(path: &Path, patterns: &[&str]) -> Result<u64> {
let mut builder = OverrideBuilder::new(path);
for pattern in patterns {
builder.add(pattern)?;
}
let overrides = builder.build()?;
let mut builder = WalkBuilder::new(path);
builder
.overrides(overrides)
.hidden(false)
.parents(false)
.ignore(false)
.git_global(false)
.git_ignore(false)
.git_exclude(false);
let walker = builder.build_parallel();
let total = Arc::new(AtomicU64::new(0));
// A slot used to indicate there was an error while walking.
//
// It is possible that more than one error happens (such as in different
// threads). The error returned is arbitrary in that case.
let err = Arc::new(Mutex::new(None));
walker.run(|| {
Box::new(|entry| {
match entry {
Ok(entry) => match entry.metadata() {
Ok(meta) => {
if meta.is_file() {
total.fetch_add(meta.len(), Ordering::SeqCst);
}
}
Err(e) => {
*err.lock().unwrap() = Some(e.into());
return WalkState::Quit;
}
},
Err(e) => {
*err.lock().unwrap() = Some(e.into());
return WalkState::Quit;
}
}
WalkState::Continue
})
});
if let Some(e) = err.lock().unwrap().take() {
return Err(e);
}
Ok(total.load(Ordering::SeqCst))
}

2
crates/cargo-util/src/lib.rs
View File

@ -1,10 +1,12 @@
//! Miscellaneous support code used by Cargo.
pub use self::read2::read2;
pub use du::du;
pub use process_builder::ProcessBuilder;
pub use process_error::{exit_status_to_string, is_simple_exit_code, ProcessError};
pub use sha256::Sha256;
mod du;
pub mod paths;
mod process_builder;
mod process_error;

159
src/bin/cargo/commands/clean.rs
View File

@ -1,7 +1,12 @@
use crate::command_prelude::*;
use crate::util::cache_lock::CacheLockMode;
use cargo::core::gc::Gc;
use cargo::core::gc::{parse_human_size, parse_time_span, GcOpts};
use cargo::core::global_cache_tracker::GlobalCacheTracker;
use cargo::ops::CleanContext;
use cargo::ops::{self, CleanOptions};
use cargo::util::print_available_packages;
use std::time::Duration;
pub fn cli() -> Command {
subcommand("clean")
@ -15,12 +20,123 @@ pub fn cli() -> Command {
.arg_target_dir()
.arg_manifest_path()
.arg_dry_run("Display what would be deleted without deleting anything")
.args_conflicts_with_subcommands(true)
.subcommand(
subcommand("gc")
.about("Clean global caches")
.hide(true)
// FIXME: arg_quiet doesn't work because `config_configure`
// doesn't know about subcommands.
.arg_dry_run("Display what would be deleted without deleting anything")
// NOTE: Not all of these options may get stabilized. Some of them are
// very low-level details, and may not be something typical users need.
.arg(
opt(
"max-src-age",
"Deletes source cache files that have not been used \
since the given age (unstable)",
)
.value_name("DURATION")
.value_parser(parse_time_span),
)
.arg(
opt(
"max-crate-age",
"Deletes crate cache files that have not been used \
since the given age (unstable)",
)
.value_name("DURATION")
.value_parser(parse_time_span),
)
.arg(
opt(
"max-index-age",
"Deletes registry indexes that have not been used \
since the given age (unstable)",
)
.value_name("DURATION")
.value_parser(parse_time_span),
)
.arg(
opt(
"max-git-co-age",
"Deletes git dependency checkouts that have not been used \
since the given age (unstable)",
)
.value_name("DURATION")
.value_parser(parse_time_span),
)
.arg(
opt(
"max-git-db-age",
"Deletes git dependency clones that have not been used \
since the given age (unstable)",
)
.value_name("DURATION")
.value_parser(parse_time_span),
)
.arg(
opt(
"max-download-age",
"Deletes any downloaded cache data that has not been used \
since the given age (unstable)",
)
.value_name("DURATION")
.value_parser(parse_time_span),
)
.arg(
opt(
"max-src-size",
"Deletes source cache files until the cache is under the \
given size (unstable)",
)
.value_name("SIZE")
.value_parser(parse_human_size),
)
.arg(
opt(
"max-crate-size",
"Deletes crate cache files until the cache is under the \
given size (unstable)",
)
.value_name("SIZE")
.value_parser(parse_human_size),
)
.arg(
opt(
"max-git-size",
"Deletes git dependency caches until the cache is under \
the given size (unstable)",
)
.value_name("SIZE")
.value_parser(parse_human_size),
)
.arg(
opt(
"max-download-size",
"Deletes downloaded cache data until the cache is under \
the given size (unstable)",
)
.value_name("SIZE")
.value_parser(parse_human_size),
),
)
.after_help(color_print::cstr!(
"Run `<cyan,bold>cargo help clean</>` for more detailed information.\n"
))
}
pub fn exec(config: &mut Config, args: &ArgMatches) -> CliResult {
match args.subcommand() {
Some(("gc", args)) => {
return gc(config, args);
}
Some((cmd, _)) => {
unreachable!("unexpected command {}", cmd)
}
None => {}
}
let ws = args.workspace(config)?;
if args.is_present_with_zero_values("package") {
@ -39,3 +155,44 @@ pub fn exec(config: &mut Config, args: &ArgMatches) -> CliResult {
ops::clean(&ws, &opts)?;
Ok(())
}
fn gc(config: &Config, args: &ArgMatches) -> CliResult {
config.cli_unstable().fail_if_stable_command(
config,
"clean gc",
12633,
"gc",
config.cli_unstable().gc,
)?;
let size_opt = |opt| -> Option<u64> { args.get_one::<u64>(opt).copied() };
let duration_opt = |opt| -> Option<Duration> { args.get_one::<Duration>(opt).copied() };
let mut gc_opts = GcOpts {
max_src_age: duration_opt("max-src-age"),
max_crate_age: duration_opt("max-crate-age"),
max_index_age: duration_opt("max-index-age"),
max_git_co_age: duration_opt("max-git-co-age"),
max_git_db_age: duration_opt("max-git-db-age"),
max_src_size: size_opt("max-src-size"),
max_crate_size: size_opt("max-crate-size"),
max_git_size: size_opt("max-git-size"),
max_download_size: size_opt("max-download-size"),
};
if let Some(age) = duration_opt("max-download-age") {
gc_opts.set_max_download_age(age);
}
// If the user sets any options, then only perform the options requested.
// If no options are set, do the default behavior.
if !gc_opts.is_download_cache_opt_set() {
gc_opts.update_for_auto_gc(config)?;
}
let _lock = config.acquire_package_cache_lock(CacheLockMode::MutateExclusive)?;
let mut cache_track = GlobalCacheTracker::new(&config)?;
let mut gc = Gc::new(config, &mut cache_track)?;
let mut clean_ctx = CleanContext::new(config);
clean_ctx.dry_run = args.dry_run();
gc.gc(&mut clean_ctx, &gc_opts)?;
clean_ctx.display_summary()?;
Ok(())
}

10
src/bin/cargo/commands/config.rs
View File

@ -31,9 +31,13 @@ pub fn cli() -> Command {
}
pub fn exec(config: &mut Config, args: &ArgMatches) -> CliResult {
config
.cli_unstable()
.fail_if_stable_command(config, "config", 9301)?;
config.cli_unstable().fail_if_stable_command(
config,
"config",
9301,
"unstable-options",
config.cli_unstable().unstable_options,
)?;
match args.subcommand() {
Some(("get", args)) => {
let opts = cargo_config::GetOptions {

27
src/cargo/core/features.rs
View File

@ -741,6 +741,7 @@ unstable_cli_options!(
doctest_xcompile: bool = ("Compile and run doctests for non-host target using runner config"),
dual_proc_macros: bool = ("Build proc-macros for both the host and the target"),
features: Option<Vec<String>> = (HIDDEN),
gc: bool = ("Track cache usage and \"garbage collect\" unused files"),
gitoxide: Option<GitoxideFeatures> = ("Use gitoxide for the given git interactions, or all of them if no argument is given"),
host_config: bool = ("Enable the [host] section in the .cargo/config.toml file"),
lints: bool = ("Pass `[lints]` to the linting tools"),
@ -1077,6 +1078,7 @@ impl CliUnstable {
"direct-minimal-versions" => self.direct_minimal_versions = parse_empty(k, v)?,
"doctest-xcompile" => self.doctest_xcompile = parse_empty(k, v)?,
"dual-proc-macros" => self.dual_proc_macros = parse_empty(k, v)?,
"gc" => self.gc = parse_empty(k, v)?,
"gitoxide" => {
self.gitoxide = v.map_or_else(
|| Ok(Some(GitoxideFeatures::all())),
@ -1114,7 +1116,17 @@ impl CliUnstable {
/// Generates an error if `-Z unstable-options` was not used for a new,
/// unstable command-line flag.
pub fn fail_if_stable_opt(&self, flag: &str, issue: u32) -> CargoResult<()> {
if !self.unstable_options {
self.fail_if_stable_opt_custom_z(flag, issue, "unstable-options", self.unstable_options)
}
pub fn fail_if_stable_opt_custom_z(
&self,
flag: &str,
issue: u32,
z_name: &str,
enabled: bool,
) -> CargoResult<()> {
if !enabled {
let see = format!(
"See https://github.com/rust-lang/cargo/issues/{issue} for more \
information about the `{flag}` flag."
@ -1123,7 +1135,7 @@ impl CliUnstable {
let channel = channel();
if channel == "nightly" || channel == "dev" {
bail!(
"the `{flag}` flag is unstable, pass `-Z unstable-options` to enable it\n\
"the `{flag}` flag is unstable, pass `-Z {z_name}` to enable it\n\
{see}"
);
} else {
@ -1145,8 +1157,10 @@ impl CliUnstable {
config: &Config,
command: &str,
issue: u32,
z_name: &str,
enabled: bool,
) -> CargoResult<()> {
if self.unstable_options {
if enabled {
return Ok(());
}
let see = format!(
@ -1156,10 +1170,9 @@ impl CliUnstable {
);
if config.nightly_features_allowed {
bail!(
"the `cargo {}` command is unstable, pass `-Z unstable-options` to enable it\n\
{}",
command,
see
"the `cargo {command}` command is unstable, pass `-Z {z_name}` \
to enable it\n\
{see}",
);
} else {
bail!(

509
src/cargo/core/gc.rs Normal file
View File

@ -0,0 +1,509 @@
//! Support for garbage collecting unused files from downloaded files or
//! artifacts from the target directory.
//!
//! The [`Gc`] type provides the high-level interface for the
//! garbage-collection system.
//!
//! Garbage collection can be done "automatically" by cargo, which it does by
//! default once a day when running any command that does a lot of work (like
//! `cargo build`). The entry point for this is the [`auto_gc`] function,
//! which handles some basic setup, creating the [`Gc`], and calling
//! [`Gc::auto`].
//!
//! Garbage collection can also be done manually via the `cargo clean` command
//! by passing any option that requests deleting unused files. That is
//! implemented by calling the [`Gc::gc`] method.
//!
//! Garbage collection for the global cache is guided by the last-use tracking
//! implemented in the [`crate::core::global_cache_tracker`] module. See that
//! module documentation for an in-depth explanation of how global cache
//! tracking works.
use crate::core::global_cache_tracker::{self, GlobalCacheTracker};
use crate::ops::CleanContext;
use crate::util::cache_lock::{CacheLock, CacheLockMode};
use crate::{CargoResult, Config};
use anyhow::{format_err, Context};
use serde::Deserialize;
use std::time::Duration;
/// Default max age to auto-clean extracted sources, which can be recovered
/// without downloading anything.
const DEFAULT_MAX_AGE_EXTRACTED: &str = "1 month";
/// Default max ago to auto-clean cache data, which must be downloaded to
/// recover.
const DEFAULT_MAX_AGE_DOWNLOADED: &str = "3 months";
/// How often auto-gc will run by default unless overridden in the config.
const DEFAULT_AUTO_FREQUENCY: &str = "1 day";
/// Performs automatic garbage collection.
///
/// This is called in various places in Cargo where garbage collection should
/// be performed automatically based on the config settings. The default
/// behavior is to only clean once a day.
///
/// This should only be called in code paths for commands that are already
/// doing a lot of work. It should only be called *after* crates are
/// downloaded so that the last-use data is updated first.
///
/// It should be cheap to call this multiple times (subsequent calls are
/// ignored), but try not to abuse that.
pub fn auto_gc(config: &Config) {
if !config.cli_unstable().gc {
return;
}
if !config.network_allowed() {
// As a conservative choice, auto-gc is disabled when offline. If the
// user is indefinitely offline, we don't want to delete things they
// may later depend on.
tracing::trace!(target: "gc", "running offline, auto gc disabled");
return;
}
if let Err(e) = auto_gc_inner(config) {
if global_cache_tracker::is_silent_error(&e) && !config.extra_verbose() {
tracing::warn!(target: "gc", "failed to auto-clean cache data: {e:?}");
} else {
crate::display_warning_with_error(
"failed to auto-clean cache data",
&e,
&mut config.shell(),
);
}
}
}
fn auto_gc_inner(config: &Config) -> CargoResult<()> {
let _lock = match config.try_acquire_package_cache_lock(CacheLockMode::MutateExclusive)? {
Some(lock) => lock,
None => {
tracing::debug!(target: "gc", "unable to acquire mutate lock, auto gc disabled");
return Ok(());
}
};
// This should not be called when there are pending deferred entries, so check that.
let deferred = config.deferred_global_last_use()?;
debug_assert!(deferred.is_empty());
let mut global_cache_tracker = config.global_cache_tracker()?;
let mut gc = Gc::new(config, &mut global_cache_tracker)?;
let mut clean_ctx = CleanContext::new(config);
gc.auto(&mut clean_ctx)?;
Ok(())
}
/// Automatic garbage collection settings from the `gc.auto` config table.
///
/// NOTE: Not all of these options may get stabilized. Some of them are very
/// low-level details, and may not be something typical users need.
///
/// If any of these options are `None`, the built-in default is used.
#[derive(Deserialize, Default)]
#[serde(rename_all = "kebab-case")]
struct AutoConfig {
/// The maximum frequency that automatic garbage collection happens.
frequency: Option<String>,
/// Anything older than this duration will be deleted in the source cache.
max_src_age: Option<String>,
/// Anything older than this duration will be deleted in the compressed crate cache.
max_crate_age: Option<String>,
/// Any index older than this duration will be deleted from the index cache.
max_index_age: Option<String>,
/// Any git checkout older than this duration will be deleted from the checkout cache.
max_git_co_age: Option<String>,
/// Any git clone older than this duration will be deleted from the git cache.
max_git_db_age: Option<String>,
}
/// Options to use for garbage collection.
#[derive(Clone, Debug, Default)]
pub struct GcOpts {
/// The `--max-src-age` CLI option.
pub max_src_age: Option<Duration>,
// The `--max-crate-age` CLI option.
pub max_crate_age: Option<Duration>,
/// The `--max-index-age` CLI option.
pub max_index_age: Option<Duration>,
/// The `--max-git-co-age` CLI option.
pub max_git_co_age: Option<Duration>,
/// The `--max-git-db-age` CLI option.
pub max_git_db_age: Option<Duration>,
/// The `--max-src-size` CLI option.
pub max_src_size: Option<u64>,
/// The `--max-crate-size` CLI option.
pub max_crate_size: Option<u64>,
/// The `--max-git-size` CLI option.
pub max_git_size: Option<u64>,
/// The `--max-download-size` CLI option.
pub max_download_size: Option<u64>,
}
impl GcOpts {
/// Returns whether any download cache cleaning options are set.
pub fn is_download_cache_opt_set(&self) -> bool {
self.max_src_age.is_some()
|| self.max_crate_age.is_some()
|| self.max_index_age.is_some()
|| self.max_git_co_age.is_some()
|| self.max_git_db_age.is_some()
|| self.max_src_size.is_some()
|| self.max_crate_size.is_some()
|| self.max_git_size.is_some()
|| self.max_download_size.is_some()
}
/// Returns whether any download cache cleaning options based on size are set.
pub fn is_download_cache_size_set(&self) -> bool {
self.max_src_size.is_some()
|| self.max_crate_size.is_some()
|| self.max_git_size.is_some()
|| self.max_download_size.is_some()
}
/// Updates the `GcOpts` to incorporate the specified max download age.
///
/// "Download" means any cached data that can be re-downloaded.
pub fn set_max_download_age(&mut self, max_download_age: Duration) {
self.max_src_age = Some(maybe_newer_span(max_download_age, self.max_src_age));
self.max_crate_age = Some(maybe_newer_span(max_download_age, self.max_crate_age));
self.max_index_age = Some(maybe_newer_span(max_download_age, self.max_index_age));
self.max_git_co_age = Some(maybe_newer_span(max_download_age, self.max_git_co_age));
self.max_git_db_age = Some(maybe_newer_span(max_download_age, self.max_git_db_age));
}
/// Updates the configuration of this [`GcOpts`] to incorporate the
/// settings from config.
pub fn update_for_auto_gc(&mut self, config: &Config) -> CargoResult<()> {
let auto_config = config
.get::<Option<AutoConfig>>("gc.auto")?
.unwrap_or_default();
self.update_for_auto_gc_config(&auto_config)
}
fn update_for_auto_gc_config(&mut self, auto_config: &AutoConfig) -> CargoResult<()> {
self.max_src_age = newer_time_span_for_config(
self.max_src_age,
"gc.auto.max-src-age",
auto_config
.max_src_age
.as_deref()
.unwrap_or(DEFAULT_MAX_AGE_EXTRACTED),
)?;
self.max_crate_age = newer_time_span_for_config(
self.max_crate_age,
"gc.auto.max-crate-age",
auto_config
.max_crate_age
.as_deref()
.unwrap_or(DEFAULT_MAX_AGE_DOWNLOADED),
)?;
self.max_index_age = newer_time_span_for_config(
self.max_index_age,
"gc.auto.max-index-age",
auto_config
.max_index_age
.as_deref()
.unwrap_or(DEFAULT_MAX_AGE_DOWNLOADED),
)?;
self.max_git_co_age = newer_time_span_for_config(
self.max_git_co_age,
"gc.auto.max-git-co-age",
auto_config
.max_git_co_age
.as_deref()
.unwrap_or(DEFAULT_MAX_AGE_EXTRACTED),
)?;
self.max_git_db_age = newer_time_span_for_config(
self.max_git_db_age,
"gc.auto.max-git-db-age",
auto_config
.max_git_db_age
.as_deref()
.unwrap_or(DEFAULT_MAX_AGE_DOWNLOADED),
)?;
Ok(())
}
}
/// Garbage collector.
///
/// See the module docs at [`crate::core::gc`] for more information on GC.
pub struct Gc<'a, 'config> {
config: &'config Config,
global_cache_tracker: &'a mut GlobalCacheTracker,
/// A lock on the package cache.
///
/// This is important to be held, since we don't want multiple cargos to
/// be allowed to write to the cache at the same time, or for others to
/// read while we are modifying the cache.
#[allow(dead_code)] // Held for drop.
lock: CacheLock<'config>,
}
impl<'a, 'config> Gc<'a, 'config> {
pub fn new(
config: &'config Config,
global_cache_tracker: &'a mut GlobalCacheTracker,
) -> CargoResult<Gc<'a, 'config>> {
let lock = config.acquire_package_cache_lock(CacheLockMode::MutateExclusive)?;
Ok(Gc {
config,
global_cache_tracker,
lock,
})
}
/// Performs automatic garbage cleaning.
///
/// This returns immediately without doing work if garbage collection has
/// been performed recently (since `gc.auto.frequency`).
fn auto(&mut self, clean_ctx: &mut CleanContext<'config>) -> CargoResult<()> {
if !self.config.cli_unstable().gc {
return Ok(());
}
let auto_config = self
.config
.get::<Option<AutoConfig>>("gc.auto")?
.unwrap_or_default();
let Some(freq) = parse_frequency(
auto_config
.frequency
.as_deref()
.unwrap_or(DEFAULT_AUTO_FREQUENCY),
)?
else {
tracing::trace!(target: "gc", "auto gc disabled");
return Ok(());
};
if !self.global_cache_tracker.should_run_auto_gc(freq)? {
return Ok(());
}
let mut gc_opts = GcOpts::default();
gc_opts.update_for_auto_gc_config(&auto_config)?;
self.gc(clean_ctx, &gc_opts)?;
if !clean_ctx.dry_run {
self.global_cache_tracker.set_last_auto_gc()?;
}
Ok(())
}
/// Performs garbage collection based on the given options.
pub fn gc(
&mut self,
clean_ctx: &mut CleanContext<'config>,
gc_opts: &GcOpts,
) -> CargoResult<()> {
self.global_cache_tracker.clean(clean_ctx, gc_opts)?;
// In the future, other gc operations go here, such as target cleaning.
Ok(())
}
}
/// Returns the shorter duration from `cur_span` versus `config_span`.
///
/// This is used because the user may specify multiple options which overlap,
/// and this will pick whichever one is shorter.
///
/// * `cur_span` is the span we are comparing against (the value from the CLI
/// option). If None, just returns the config duration.
/// * `config_name` is the name of the config option the span is loaded from.
/// * `config_span` is the span value loaded from config.
fn newer_time_span_for_config(
cur_span: Option<Duration>,
config_name: &str,
config_span: &str,
) -> CargoResult<Option<Duration>> {
let config_span = parse_time_span_for_config(config_name, config_span)?;
Ok(Some(maybe_newer_span(config_span, cur_span)))
}
/// Returns whichever [`Duration`] is shorter.
fn maybe_newer_span(a: Duration, b: Option<Duration>) -> Duration {
match b {
Some(b) => {
if b < a {
b
} else {
a
}
}
None => a,
}
}
/// Parses a frequency string.
///
/// Returns `Ok(None)` if the frequency is "never".
fn parse_frequency(frequency: &str) -> CargoResult<Option<Duration>> {
if frequency == "always" {
return Ok(Some(Duration::new(0, 0)));
} else if frequency == "never" {
return Ok(None);
}
let duration = maybe_parse_time_span(frequency).ok_or_else(|| {
format_err!(
"config option `gc.auto.frequency` expected a value of \"always\", \"never\", \
or \"N seconds/minutes/days/weeks/months\", got: {frequency:?}"
)
})?;
Ok(Some(duration))
}
/// Parses a time span value fetched from config.
///
/// This is here to provide better error messages specific to reading from
/// config.
fn parse_time_span_for_config(config_name: &str, span: &str) -> CargoResult<Duration> {
maybe_parse_time_span(span).ok_or_else(|| {
format_err!(
"config option `{config_name}` expected a value of the form \
\"N seconds/minutes/days/weeks/months\", got: {span:?}"
)
})
}
/// Parses a time span string.
///
/// Returns None if the value is not valid. See [`parse_time_span`] if you
/// need a variant that generates an error message.
fn maybe_parse_time_span(span: &str) -> Option<Duration> {
let Some(right_i) = span.find(|c: char| !c.is_ascii_digit()) else {
return None;
};
let (left, mut right) = span.split_at(right_i);
if right.starts_with(' ') {
right = &right[1..];
}
let count: u64 = left.parse().ok()?;
let factor = match right {
"second" | "seconds" => 1,
"minute" | "minutes" => 60,
"hour" | "hours" => 60 * 60,
"day" | "days" => 24 * 60 * 60,
"week" | "weeks" => 7 * 24 * 60 * 60,
"month" | "months" => 2_629_746, // average is 30.436875 days
_ => return None,
};
Some(Duration::from_secs(factor * count))
}
/// Parses a time span string.
pub fn parse_time_span(span: &str) -> CargoResult<Duration> {
maybe_parse_time_span(span).ok_or_else(|| {
format_err!(
"expected a value of the form \
\"N seconds/minutes/days/weeks/months\", got: {span:?}"
)
})
}
/// Parses a file size using metric or IEC units.
pub fn parse_human_size(input: &str) -> CargoResult<u64> {
let re = regex::Regex::new(r"(?i)^([0-9]+(\.[0-9])?) ?(b|kb|mb|gb|kib|mib|gib)?$").unwrap();
let cap = re.captures(input).ok_or_else(|| {
format_err!(
"invalid size `{input}`, \
expected a number with an optional B, kB, MB, GB, kiB, MiB, or GiB suffix"
)
})?;
let factor = match cap.get(3) {
Some(suffix) => match suffix.as_str().to_lowercase().as_str() {
"b" => 1.0,
"kb" => 1_000.0,
"mb" => 1_000_000.0,
"gb" => 1_000_000_000.0,
"kib" => 1024.0,
"mib" => 1024.0 * 1024.0,
"gib" => 1024.0 * 1024.0 * 1024.0,
s => unreachable!("suffix `{s}` out of sync with regex"),
},
None => {
return cap[1]
.parse()
.with_context(|| format!("expected an integer size, got `{}`", &cap[1]))
}
};
let num = cap[1]
.parse::<f64>()
.with_context(|| format!("expected an integer or float, found `{}`", &cap[1]))?;
Ok((num * factor) as u64)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn time_spans() {
let d = |x| Some(Duration::from_secs(x));
assert_eq!(maybe_parse_time_span("0 seconds"), d(0));
assert_eq!(maybe_parse_time_span("1second"), d(1));
assert_eq!(maybe_parse_time_span("23 seconds"), d(23));
assert_eq!(maybe_parse_time_span("5 minutes"), d(60 * 5));
assert_eq!(maybe_parse_time_span("2 hours"), d(60 * 60 * 2));
assert_eq!(maybe_parse_time_span("1 day"), d(60 * 60 * 24));
assert_eq!(maybe_parse_time_span("2 weeks"), d(60 * 60 * 24 * 14));
assert_eq!(maybe_parse_time_span("6 months"), d(2_629_746 * 6));
assert_eq!(parse_frequency("5 seconds").unwrap(), d(5));
assert_eq!(parse_frequency("always").unwrap(), d(0));
assert_eq!(parse_frequency("never").unwrap(), None);
}
#[test]
fn time_span_errors() {
assert_eq!(maybe_parse_time_span(""), None);
assert_eq!(maybe_parse_time_span("1"), None);
assert_eq!(maybe_parse_time_span("second"), None);
assert_eq!(maybe_parse_time_span("+2 seconds"), None);
assert_eq!(maybe_parse_time_span("day"), None);
assert_eq!(maybe_parse_time_span("-1 days"), None);
assert_eq!(maybe_parse_time_span("1.5 days"), None);
assert_eq!(maybe_parse_time_span("1 dayz"), None);
assert_eq!(maybe_parse_time_span("always"), None);
assert_eq!(maybe_parse_time_span("never"), None);
assert_eq!(maybe_parse_time_span("1 day "), None);
assert_eq!(maybe_parse_time_span(" 1 day"), None);
assert_eq!(maybe_parse_time_span("1 second"), None);
let e = parse_time_span_for_config("gc.auto.max-src-age", "-1 days").unwrap_err();
assert_eq!(
e.to_string(),
"config option `gc.auto.max-src-age` \
expected a value of the form \"N seconds/minutes/days/weeks/months\", \
got: \"-1 days\""
);
let e = parse_frequency("abc").unwrap_err();
assert_eq!(
e.to_string(),
"config option `gc.auto.frequency` \
expected a value of \"always\", \"never\", or \"N seconds/minutes/days/weeks/months\", \
got: \"abc\""
);
}
#[test]
fn human_sizes() {
assert_eq!(parse_human_size("0").unwrap(), 0);
assert_eq!(parse_human_size("123").unwrap(), 123);
assert_eq!(parse_human_size("123b").unwrap(), 123);
assert_eq!(parse_human_size("123B").unwrap(), 123);
assert_eq!(parse_human_size("123 b").unwrap(), 123);
assert_eq!(parse_human_size("123 B").unwrap(), 123);
assert_eq!(parse_human_size("1kb").unwrap(), 1_000);
assert_eq!(parse_human_size("5kb").unwrap(), 5_000);
assert_eq!(parse_human_size("1mb").unwrap(), 1_000_000);
assert_eq!(parse_human_size("1gb").unwrap(), 1_000_000_000);
assert_eq!(parse_human_size("1kib").unwrap(), 1_024);
assert_eq!(parse_human_size("1mib").unwrap(), 1_048_576);
assert_eq!(parse_human_size("1gib").unwrap(), 1_073_741_824);
assert_eq!(parse_human_size("1.5kb").unwrap(), 1_500);
assert_eq!(parse_human_size("1.7b").unwrap(), 1);
assert!(parse_human_size("").is_err());
assert!(parse_human_size("x").is_err());
assert!(parse_human_size("1x").is_err());
assert!(parse_human_size("1 2").is_err());
assert!(parse_human_size("1.5").is_err());
assert!(parse_human_size("+1").is_err());
assert!(parse_human_size("123 b").is_err());
}
}

1810
src/cargo/core/global_cache_tracker.rs Normal file
View File

File diff suppressed because it is too large Load Diff

2
src/cargo/core/mod.rs
View File

@ -19,6 +19,8 @@ pub use crate::util::toml::schema::InheritableFields;
pub mod compiler;
pub mod dependency;
pub mod features;
pub mod gc;
pub mod global_cache_tracker;
pub mod manifest;
pub mod package;
pub mod package_id;

4
src/cargo/core/package.rs
View File

@ -491,6 +491,10 @@ impl<'cfg> PackageSet<'cfg> {
pkgs.push(downloads.wait()?);
}
downloads.success = true;
drop(downloads);
let mut deferred = self.config.deferred_global_last_use()?;
deferred.save_no_error(self.config);
Ok(pkgs)
}

2
src/cargo/ops/cargo_clean.rs
View File

@ -389,7 +389,7 @@ impl<'cfg> CleanContext<'cfg> {
Ok(())
}
fn display_summary(&self) -> CargoResult<()> {
pub fn display_summary(&self) -> CargoResult<()> {
let status = if self.dry_run { "Summary" } else { "Removed" };
let byte_count = if self.total_bytes_removed == 0 {
String::new()

1
src/cargo/ops/cargo_compile/mod.rs
View File

@ -153,6 +153,7 @@ pub fn compile_ws<'a>(
unit_graph::emit_serialized_unit_graph(&bcx.roots, &bcx.unit_graph, ws.config())?;
return Compilation::new(&bcx);
}
crate::core::gc::auto_gc(bcx.config);
let _p = profile::start("compiling");
let cx = Context::new(&bcx)?;
cx.compile(exec)

1
src/cargo/ops/cargo_fetch.rs
View File

@ -76,6 +76,7 @@ pub fn fetch<'a>(
}
packages.get_many(to_download)?;
crate::core::gc::auto_gc(config);
Ok((resolve, packages))
}

2
src/cargo/ops/mod.rs
View File

@ -1,6 +1,6 @@
use crate::sources::CRATES_IO_DOMAIN;
pub use self::cargo_clean::{clean, CleanOptions};
pub use self::cargo_clean::{clean, CleanContext, CleanOptions};
pub use self::cargo_compile::{
compile, compile_with_exec, compile_ws, create_bcx, print, resolve_all_features, CompileOptions,
};

3
src/cargo/ops/resolve.rs
View File

@ -530,6 +530,9 @@ pub fn resolve_with_previous<'cfg>(
if let Some(previous) = previous {
resolved.merge_from(previous)?;
}
let config = ws.config();
let mut deferred = config.deferred_global_last_use()?;
deferred.save_no_error(config);
Ok(resolved)
}

31
src/cargo/sources/git/source.rs
View File

@ -1,5 +1,6 @@
//! See [GitSource].
use crate::core::global_cache_tracker;
use crate::core::GitReference;
use crate::core::SourceId;
use crate::core::{Dependency, Package, PackageId, Summary};
@ -11,6 +12,7 @@ use crate::sources::PathSource;
use crate::util::cache_lock::CacheLockMode;
use crate::util::errors::CargoResult;
use crate::util::hex::short_hash;
use crate::util::interning::InternedString;
use crate::util::Config;
use anyhow::Context;
use cargo_util::paths::exclude_from_backups_and_indexing;
@ -74,9 +76,10 @@ pub struct GitSource<'cfg> {
source_id: SourceId,
/// The underlying path source to discover packages inside the Git repository.
path_source: Option<PathSource<'cfg>>,
short_id: Option<InternedString>,
/// The identifier of this source for Cargo's Git cache directory.
/// See [`ident`] for more.
ident: String,
ident: InternedString,
config: &'cfg Config,
/// Disables status messages.
quiet: bool,
@ -104,7 +107,8 @@ impl<'cfg> GitSource<'cfg> {
locked_rev,
source_id,
path_source: None,
ident,
short_id: None,
ident: ident.into(),
config,
quiet: false,
};
@ -127,6 +131,17 @@ impl<'cfg> GitSource<'cfg> {
}
self.path_source.as_mut().unwrap().read_packages()
}
fn mark_used(&self, size: Option<u64>) -> CargoResult<()> {
self.config
.deferred_global_last_use()?
.mark_git_checkout_used(global_cache_tracker::GitCheckout {
encoded_git_name: self.ident,
short_name: self.short_id.expect("update before download"),
size,
});
Ok(())
}
}
/// Create an identifier from a URL,
@ -200,6 +215,7 @@ impl<'cfg> Source for GitSource<'cfg> {
fn block_until_ready(&mut self) -> CargoResult<()> {
if self.path_source.is_some() {
self.mark_used(None)?;
return Ok(());
}
@ -290,8 +306,16 @@ impl<'cfg> Source for GitSource<'cfg> {
let path_source = PathSource::new_recursive(&checkout_path, source_id, self.config);
self.path_source = Some(path_source);
self.short_id = Some(short_id.as_str().into());
self.locked_rev = Some(actual_rev);
self.path_source.as_mut().unwrap().update()
self.path_source.as_mut().unwrap().update()?;
// Hopefully this shouldn't incur too much of a performance hit since
// most of this should already be in cache since it was just
// extracted.
let size = global_cache_tracker::du_git_checkout(&checkout_path)?;
self.mark_used(Some(size))?;
Ok(())
}
fn download(&mut self, id: PackageId) -> CargoResult<MaybePackage> {
@ -300,6 +324,7 @@ impl<'cfg> Source for GitSource<'cfg> {
id,
self.remote
);
self.mark_used(None)?;
self.path_source
.as_mut()
.expect("BUG: `update()` must be called before `get()`")

18
src/cargo/sources/registry/download.rs
View File

@ -3,11 +3,13 @@
//! [`HttpRegistry`]: super::http_remote::HttpRegistry
//! [`RemoteRegistry`]: super::remote::RemoteRegistry
use crate::util::interning::InternedString;
use anyhow::Context;
use cargo_credential::Operation;
use cargo_util::registry::make_dep_path;
use cargo_util::Sha256;
use crate::core::global_cache_tracker;
use crate::core::PackageId;
use crate::sources::registry::MaybeLock;
use crate::sources::registry::RegistryConfig;
@ -34,6 +36,7 @@ const CHECKSUM_TEMPLATE: &str = "{sha256-checksum}";
pub(super) fn download(
cache_path: &Filesystem,
config: &Config,
encoded_registry_name: InternedString,
pkg: PackageId,
checksum: &str,
registry_config: RegistryConfig,
@ -50,6 +53,13 @@ pub(super) fn download(
if let Ok(dst) = File::open(path) {
let meta = dst.metadata()?;
if meta.len() > 0 {
config.deferred_global_last_use()?.mark_registry_crate_used(
global_cache_tracker::RegistryCrate {
encoded_registry_name,
crate_filename: pkg.tarball_name().into(),
size: meta.len(),
},
);
return Ok(MaybeLock::Ready(dst));
}
}
@ -106,6 +116,7 @@ pub(super) fn download(
pub(super) fn finish_download(
cache_path: &Filesystem,
config: &Config,
encoded_registry_name: InternedString,
pkg: PackageId,
checksum: &str,
data: &[u8],
@ -115,6 +126,13 @@ pub(super) fn finish_download(
if actual != checksum {
anyhow::bail!("failed to verify the checksum of `{}`", pkg)
}
config.deferred_global_last_use()?.mark_registry_crate_used(
global_cache_tracker::RegistryCrate {
encoded_registry_name,
crate_filename: pkg.tarball_name().into(),
size: data.len() as u64,
},
);
cache_path.create_dir()?;
let path = cache_path.join(&pkg.tarball_name());

19
src/cargo/sources/registry/http_remote.rs
View File

@ -1,11 +1,13 @@
//! Access to a HTTP-based crate registry. See [`HttpRegistry`] for details.
use crate::core::global_cache_tracker;
use crate::core::{PackageId, SourceId};
use crate::sources::registry::download;
use crate::sources::registry::MaybeLock;
use crate::sources::registry::{LoadResponse, RegistryConfig, RegistryData};
use crate::util::cache_lock::CacheLockMode;
use crate::util::errors::{CargoResult, HttpNotSuccessful};
use crate::util::interning::InternedString;
use crate::util::network::http::http_handle;
use crate::util::network::retry::{Retry, RetryResult};
use crate::util::network::sleep::SleepTracker;
@ -52,6 +54,7 @@ const UNKNOWN: &'static str = "Unknown";
///
/// [RFC 2789]: https://github.com/rust-lang/rfcs/pull/2789
pub struct HttpRegistry<'cfg> {
name: InternedString,
/// Path to the registry index (`$CARGO_HOME/registry/index/$REG-HASH`).
///
/// To be fair, `HttpRegistry` doesn't store the registry index it
@ -199,6 +202,7 @@ impl<'cfg> HttpRegistry<'cfg> {
.expect("a url with the sparse+ stripped should still be valid");
Ok(HttpRegistry {
name: name.into(),
index_path: config.registry_index_path().join(name),
cache_path: config.registry_cache_path().join(name),
source_id,
@ -454,6 +458,11 @@ impl<'cfg> HttpRegistry<'cfg> {
impl<'cfg> RegistryData for HttpRegistry<'cfg> {
fn prepare(&self) -> CargoResult<()> {
self.config
.deferred_global_last_use()?
.mark_registry_index_used(global_cache_tracker::RegistryIndex {
encoded_registry_name: self.name,
});
Ok(())
}
@ -750,6 +759,7 @@ impl<'cfg> RegistryData for HttpRegistry<'cfg> {
download::download(
&self.cache_path,
&self.config,
self.name.clone(),
pkg,
checksum,
registry_config,
@ -762,7 +772,14 @@ impl<'cfg> RegistryData for HttpRegistry<'cfg> {
checksum: &str,
data: &[u8],
) -> CargoResult<File> {
download::finish_download(&self.cache_path, &self.config, pkg, checksum, data)
download::finish_download(
&self.cache_path,
&self.config,
self.name.clone(),
pkg,
checksum,
data,
)
}
fn is_crate_downloaded(&self, pkg: PackageId) -> bool {

20
src/cargo/sources/registry/mod.rs
View File

@ -201,6 +201,7 @@ use tar::Archive;
use tracing::debug;
use crate::core::dependency::Dependency;
use crate::core::global_cache_tracker;
use crate::core::{Package, PackageId, SourceId, Summary};
use crate::sources::source::MaybePackage;
use crate::sources::source::QueryKind;
@ -239,6 +240,7 @@ struct LockMetadata {
///
/// For general concepts of registries, see the [module-level documentation](crate::sources::registry).
pub struct RegistrySource<'cfg> {
name: InternedString,
/// The unique identifier of this source.
source_id: SourceId,
/// The path where crate files are extracted (`$CARGO_HOME/registry/src/$REG-HASH`).
@ -514,6 +516,7 @@ impl<'cfg> RegistrySource<'cfg> {
yanked_whitelist: &HashSet<PackageId>,
) -> RegistrySource<'cfg> {
RegistrySource {
name: name.into(),
src_path: config.registry_source_path().join(name),
config,
source_id,
@ -589,6 +592,13 @@ impl<'cfg> RegistrySource<'cfg> {
match fs::read_to_string(path) {
Ok(ok) => match serde_json::from_str::<LockMetadata>(&ok) {
Ok(lock_meta) if lock_meta.v == 1 => {
self.config
.deferred_global_last_use()?
.mark_registry_src_used(global_cache_tracker::RegistrySrc {
encoded_registry_name: self.name,
package_dir: package_dir.into(),
size: None,
});
return Ok(unpack_dir.to_path_buf());
}
_ => {
@ -613,6 +623,7 @@ impl<'cfg> RegistrySource<'cfg> {
set_mask(&mut tar);
tar
};
let mut bytes_written = 0;
let prefix = unpack_dir.file_name().unwrap();
let parent = unpack_dir.parent().unwrap();
for entry in tar.entries()? {
@ -644,6 +655,7 @@ impl<'cfg> RegistrySource<'cfg> {
continue;
}
// Unpacking failed
bytes_written += entry.size();
let mut result = entry.unpack_in(parent).map_err(anyhow::Error::from);
if cfg!(windows) && restricted_names::is_windows_reserved_path(&entry_path) {
result = result.with_context(|| {
@ -670,6 +682,14 @@ impl<'cfg> RegistrySource<'cfg> {
let lock_meta = LockMetadata { v: 1 };
write!(ok, "{}", serde_json::to_string(&lock_meta).unwrap())?;
self.config
.deferred_global_last_use()?
.mark_registry_src_used(global_cache_tracker::RegistrySrc {
encoded_registry_name: self.name,
package_dir: package_dir.into(),
size: Some(bytes_written),
});
Ok(unpack_dir.to_path_buf())
}

18
src/cargo/sources/registry/remote.rs
View File

@ -1,5 +1,6 @@
//! Access to a Git index based registry. See [`RemoteRegistry`] for details.
use crate::core::global_cache_tracker;
use crate::core::{GitReference, PackageId, SourceId};
use crate::sources::git;
use crate::sources::git::fetch::RemoteKind;
@ -47,6 +48,7 @@ use tracing::{debug, trace};
///
/// [`HttpRegistry`]: super::http_remote::HttpRegistry
pub struct RemoteRegistry<'cfg> {
name: InternedString,
/// Path to the registry index (`$CARGO_HOME/registry/index/$REG-HASH`).
index_path: Filesystem,
/// Path to the cache of `.crate` files (`$CARGO_HOME/registry/cache/$REG-HASH`).
@ -87,6 +89,7 @@ impl<'cfg> RemoteRegistry<'cfg> {
/// registry index are stored. Expect to be unique.
pub fn new(source_id: SourceId, config: &'cfg Config, name: &str) -> RemoteRegistry<'cfg> {
RemoteRegistry {
name: name.into(),
index_path: config.registry_index_path().join(name),
cache_path: config.registry_cache_path().join(name),
source_id,
@ -211,6 +214,11 @@ impl<'cfg> RemoteRegistry<'cfg> {
impl<'cfg> RegistryData for RemoteRegistry<'cfg> {
fn prepare(&self) -> CargoResult<()> {
self.repo()?;
self.config
.deferred_global_last_use()?
.mark_registry_index_used(global_cache_tracker::RegistryIndex {
encoded_registry_name: self.name,
});
Ok(())
}
@ -403,6 +411,7 @@ impl<'cfg> RegistryData for RemoteRegistry<'cfg> {
download::download(
&self.cache_path,
&self.config,
self.name,
pkg,
checksum,
registry_config,
@ -415,7 +424,14 @@ impl<'cfg> RegistryData for RemoteRegistry<'cfg> {
checksum: &str,
data: &[u8],
) -> CargoResult<File> {
download::finish_download(&self.cache_path, &self.config, pkg, checksum, data)
download::finish_download(
&self.cache_path,
&self.config,
self.name.clone(),
pkg,
checksum,
data,
)
}
fn is_crate_downloaded(&self, pkg: PackageId) -> bool {

24
src/cargo/util/config/mod.rs
View File

@ -68,6 +68,7 @@ use std::time::Instant;
use self::ConfigValue as CV;
use crate::core::compiler::rustdoc::RustdocExternMap;
use crate::core::global_cache_tracker::{DeferredGlobalLastUse, GlobalCacheTracker};
use crate::core::shell::Verbosity;
use crate::core::{features, CliUnstable, Shell, SourceId, Workspace, WorkspaceRootConfig};
use crate::ops::RegistryCredentialConfig;
@ -244,6 +245,8 @@ pub struct Config {
pub nightly_features_allowed: bool,
/// WorkspaceRootConfigs that have been found
pub ws_roots: RefCell<HashMap<PathBuf, WorkspaceRootConfig>>,
global_cache_tracker: LazyCell<RefCell<GlobalCacheTracker>>,
deferred_global_last_use: LazyCell<RefCell<DeferredGlobalLastUse>>,
}
impl Config {
@ -317,6 +320,8 @@ impl Config {
env_config: LazyCell::new(),
nightly_features_allowed: matches!(&*features::channel(), "nightly" | "dev"),
ws_roots: RefCell::new(HashMap::new()),
global_cache_tracker: LazyCell::new(),
deferred_global_last_use: LazyCell::new(),
}
}
@ -1919,6 +1924,25 @@ impl Config {
) -> CargoResult<Option<CacheLock<'_>>> {
self.package_cache_lock.try_lock(self, mode)
}
/// Returns a reference to the shared [`GlobalCacheTracker`].
///
/// The package cache lock must be held to call this function (and to use
/// it in general).
pub fn global_cache_tracker(&self) -> CargoResult<RefMut<'_, GlobalCacheTracker>> {
let tracker = self.global_cache_tracker.try_borrow_with(|| {
Ok::<_, anyhow::Error>(RefCell::new(GlobalCacheTracker::new(self)?))
})?;
Ok(tracker.borrow_mut())
}
/// Returns a reference to the shared [`DeferredGlobalLastUse`].
pub fn deferred_global_last_use(&self) -> CargoResult<RefMut<'_, DeferredGlobalLastUse>> {
let deferred = self.deferred_global_last_use.try_borrow_with(|| {
Ok::<_, anyhow::Error>(RefCell::new(DeferredGlobalLastUse::new()))
})?;
Ok(deferred.borrow_mut())
}
}
/// Internal error for serde errors.

1
src/cargo/util/mod.rs
View File

@ -62,6 +62,7 @@ mod queue;
pub mod restricted_names;
pub mod rustc;
mod semver_ext;
pub mod sqlite;
pub mod style;
pub mod toml;
pub mod toml_mut;

118
src/cargo/util/sqlite.rs Normal file
View File

@ -0,0 +1,118 @@
//! Utilities to help with working with sqlite.
use crate::util::interning::InternedString;
use crate::CargoResult;
use rusqlite::types::{FromSql, FromSqlError, ToSql, ToSqlOutput};
use rusqlite::{Connection, TransactionBehavior};
impl FromSql for InternedString {
fn column_result(value: rusqlite::types::ValueRef<'_>) -> Result<Self, FromSqlError> {
value.as_str().map(InternedString::new)
}
}
impl ToSql for InternedString {
fn to_sql(&self) -> Result<ToSqlOutput<'_>, rusqlite::Error> {
Ok(ToSqlOutput::from(self.as_str()))
}
}
/// A function or closure representing a database migration.
///
/// Migrations support evolving the schema and contents of the database across
/// new versions of cargo. The [`migrate`] function should be called
/// immediately after opening a connection to a database in order to configure
/// the schema. Whether or not a migration has been done is tracked by the
/// `pragma_user_version` value in the database. Typically you include the
/// initial `CREATE TABLE` statements in the initial list, but as time goes on
/// you can add new tables or `ALTER TABLE` statements. The migration code
/// will only execute statements that haven't previously been run.
///
/// Important things to note about how you define migrations:
///
/// * Never remove a migration entry from the list. Migrations are tracked by
/// the index number in the list.
/// * Never perform any schema modifications that would be backwards
/// incompatible. For example, don't drop tables or columns.
///
/// The [`basic_migration`] function is a convenience function for specifying
/// migrations that are simple SQL statements. If you need to do something
/// more complex, then you can specify a closure that takes a [`Connection`]
/// and does whatever is needed.
///
/// For example:
///
/// ```rust
/// # use cargo::util::sqlite::*;
/// # use rusqlite::Connection;
/// # let mut conn = Connection::open_in_memory()?;
/// # fn generate_name() -> String { "example".to_string() };
/// migrate(
/// &mut conn,
/// &[
/// basic_migration(
/// "CREATE TABLE foo (
/// id INTEGER PRIMARY KEY AUTOINCREMENT,
/// name STRING NOT NULL
/// )",
/// ),
/// Box::new(|conn| {
/// conn.execute("INSERT INTO foo (name) VALUES (?1)", [generate_name()])?;
/// Ok(())
/// }),
/// basic_migration("ALTER TABLE foo ADD COLUMN size INTEGER"),
/// ],
/// )?;
/// # Ok::<(), anyhow::Error>(())
/// ```
pub type Migration = Box<dyn Fn(&Connection) -> CargoResult<()>>;
/// A basic migration that is a single static SQL statement.
///
/// See [`Migration`] for more information.
pub fn basic_migration(stmt: &'static str) -> Migration {
Box::new(|conn| {
conn.execute(stmt, [])?;
Ok(())
})
}
/// Perform one-time SQL migrations.
///
/// See [`Migration`] for more information.
pub fn migrate(conn: &mut Connection, migrations: &[Migration]) -> CargoResult<()> {
// EXCLUSIVE ensures that it starts with an exclusive write lock. No other
// readers will be allowed. This generally shouldn't be needed if there is
// a file lock, but might be helpful in cases where cargo's `FileLock`
// failed.
let tx = conn.transaction_with_behavior(TransactionBehavior::Exclusive)?;
let user_version = tx.query_row("SELECT user_version FROM pragma_user_version", [], |row| {
row.get(0)
})?;
if user_version < migrations.len() {
for migration in &migrations[user_version..] {
migration(&tx)?;
}
tx.pragma_update(None, "user_version", &migrations.len())?;
}
tx.commit()?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn migrate_twice() -> CargoResult<()> {
// Check that a second migration will apply.
let mut conn = Connection::open_in_memory()?;
let mut migrations = vec![basic_migration("CREATE TABLE foo (a, b, c)")];
migrate(&mut conn, &migrations)?;
conn.execute("INSERT INTO foo VALUES (1,2,3)", [])?;
migrations.push(basic_migration("ALTER TABLE foo ADD COLUMN d"));
migrate(&mut conn, &migrations)?;
conn.execute("INSERT INTO foo VALUES (1,2,3,4)", [])?;
Ok(())
}
}

73
src/doc/src/reference/unstable.md
View File

@ -85,6 +85,7 @@ For the latest nightly, see the [nightly version] of this page.
* [check-cfg](#check-cfg) --- Compile-time validation of `cfg` expressions.
* [host-config](#host-config) --- Allows setting `[target]`-like configuration settings for host build targets.
* [target-applies-to-host](#target-applies-to-host) --- Alters whether certain flags will be passed to host build targets.
* [gc](#gc) --- Global cache garbage collection.
* rustdoc
* [rustdoc-map](#rustdoc-map) --- Provides mappings for documentation to link to external sites like [docs.rs](https://docs.rs/).
* [scrape-examples](#scrape-examples) --- Shows examples within documentation.
@ -1383,6 +1384,78 @@ This will not affect any hard-coded paths in the source code, such as in strings
Common paths requiring sanitization include `OUT_DIR` and `CARGO_MANIFEST_DIR`,
plus any other introduced by the build script, such as include directories.
## gc
* Tracking Issue: [#12633](https://github.com/rust-lang/cargo/issues/12633)
The `-Zgc` flag enables garbage-collection within cargo's global cache within the cargo home directory.
This includes downloaded dependencies such as compressed `.crate` files, extracted `src` directories, registry index caches, and git dependencies.
When `-Zgc` is present, cargo will track the last time any index and dependency was used,
and then uses those timestamps to manually or automatically delete cache entries that have not been used for a while.
```sh
cargo build -Zgc
```
### Automatic garbage collection
Automatic deletion happens on commands that are already doing a significant amount of work,
such as all of the build commands (`cargo build`, `cargo test`, `cargo check`, etc.), and `cargo fetch`.
The deletion happens just after resolution and packages have been downloaded.
Automatic deletion is only done once per day (see `gc.auto.frequency` to configure).
Automatic deletion is disabled if cargo is offline such as with `--offline` or `--frozen` to avoid deleting artifacts that may need to be used if you are offline for a long period of time.
#### Automatic gc configuration
The automatic gc behavior can be specified via a cargo configuration setting.
The settings available are:
```toml
# Example config.toml file.
# This table defines the behavior for automatic garbage collection.
[gc.auto]
# The maximum frequency that automatic garbage collection happens.
# Can be "never" to disable automatic-gc, or "always" to run on every command.
frequency = "1 day"
# Anything older than this duration will be deleted in the source cache.
max-src-age = "1 month"
# Anything older than this duration will be deleted in the compressed crate cache.
max-crate-age = "3 months"
# Any index older than this duration will be deleted from the index cache.
max-index-age = "3 months"
# Any git checkout older than this duration will be deleted from the checkout cache.
max-git-co-age = "1 month"
# Any git clone older than this duration will be deleted from the git cache.
max-git-db-age = "3 months"
```
### Manual garbage collection with `cargo clean`
Manual deletion can be done with the `cargo clean gc` command.
Deletion of cache contents can be performed by passing one of the cache options:
- `--max-src-age=DURATION` --- Deletes source cache files that have not been used since the given age.
- `--max-crate-age=DURATION` --- Deletes crate cache files that have not been used since the given age.
- `--max-index-age=DURATION` --- Deletes registry indexes that have not been used since then given age (including their `.crate` and `src` files).
- `--max-git-co-age=DURATION` --- Deletes git dependency checkouts that have not been used since then given age.
- `--max-git-db-age=DURATION` --- Deletes git dependency clones that have not been used since then given age.
- `--max-download-age=DURATION` --- Deletes any downloaded cache data that has not been used since then given age.
- `--max-src-size=SIZE` --- Deletes the oldest source cache files until the cache is under the given size.
- `--max-crate-size=SIZE` --- Deletes the oldest crate cache files until the cache is under the given size.
- `--max-git-size=SIZE` --- Deletes the oldest git dependency caches until the cache is under the given size.
- `--max-download-size=SIZE` --- Deletes the oldest downloaded cache data until the cache is under the given size.
A DURATION is specified in the form "N seconds/minutes/days/weeks/months" where N is an integer.
A SIZE is specified in the form "N *suffix*" where *suffix* is B, kB, MB, GB, kiB, MiB, or GiB, and N is an integer or floating point number. If no suffix is specified, the number is the number of bytes.
```sh
cargo clean gc
cargo clean gc --max-download-age=1week
cargo clean gc --max-git-size=0 --max-download-size=100MB
```
# Stabilized and removed features
## Compile progress

14
tests/testsuite/clean.rs
View File

@ -1,5 +1,6 @@
//! Tests for the `cargo clean` command.
use cargo_test_support::paths::CargoPathExt;
use cargo_test_support::registry::Package;
use cargo_test_support::{
basic_bin_manifest, basic_manifest, git, main_file, project, project_in, rustc_host,
@ -805,15 +806,6 @@ fn clean_dry_run() {
.file("src/lib.rs", "")
.build();
let ls_r = || -> Vec<_> {
let mut file_list: Vec<_> = walkdir::WalkDir::new(p.build_dir())
.into_iter()
.filter_map(|e| e.map(|e| e.path().to_owned()).ok())
.collect();
file_list.sort();
file_list
};
// Start with no files.
p.cargo("clean --dry-run")
.with_stdout("")
@ -823,7 +815,7 @@ fn clean_dry_run() {
)
.run();
p.cargo("check").run();
let before = ls_r();
let before = p.build_dir().ls_r();
p.cargo("clean --dry-run")
.with_stderr(
"[SUMMARY] [..] files, [..] total\n\
@ -831,7 +823,7 @@ fn clean_dry_run() {
)
.run();
// Verify it didn't delete anything.
let after = ls_r();
let after = p.build_dir().ls_r();
assert_eq!(before, after);
let expected = cargo::util::iter_join(before.iter().map(|p| p.to_str().unwrap()), "\n");
eprintln!("{expected}");

1835
tests/testsuite/global_cache_tracker.rs Normal file
View File

File diff suppressed because it is too large Load Diff

1
tests/testsuite/main.rs
View File

@ -98,6 +98,7 @@ mod git_auth;
mod git_gc;
mod git_shallow;
mod glob_targets;
mod global_cache_tracker;
mod help;
mod https;
mod inheritable_workspace_fields;