This stabilizes the doctest-xcompile feature by unconditionally enabling
it.
Closes https://github.com/rust-lang/cargo/issues/7040
Closes https://github.com/rust-lang/cargo/issues/12118
## What is being stabilized?
This changes it so that cargo will run doctests when using the
`--target` flag for a target that is not the host. Previously, cargo
would ignore doctests (and show a note if passing `--verbose`).
A wrapper for running the doctest can be specified with the
[`target.*.runner`](https://doc.rust-lang.org/cargo/reference/config.html#targettriplerunner)
configuration option (which is powered by the `--test-runtool` rustdoc
flag). This would typically be something like qemu to run under
emulation. It is my understanding that this should work just like
running other kinds of tests.
Additionally, the
[`target.*.linker`](https://doc.rust-lang.org/cargo/reference/config.html#targettriplelinker)
config option is honored for using a custom linker.
Already stabilized in rustdoc is the ability to [ignore tests
per-target](https://doc.rust-lang.org/nightly/rustdoc/write-documentation/documentation-tests.html#ignoring-targets).
## Motivation
The lack of doctest cross-compile support has always been simply due to
the lack of functionality in rustdoc to support this. Rustdoc gained the
ability to cross-compile doctests some time ago, but there were
additional flags like the test runner that were not stabilized until
just recently.
This is intended to ensure that projects have full test coverage even
when doing cross-compilation. It can be
[surprising](https://github.com/rust-lang/cargo/issues/12118) to some
that this was not happening, particularly since cargo is silent about
it.
## Risks
The cargo team had several conversations about how to roll out this
feature. Ultimately we decided to enable it unconditionally with the
understanding that most projects will probably want to have their
doctests covered, and that any breakage will be a local concern that can
be resolved by either fixing the test or ignoring the target.
Tests in rust-lang/rust run into this issue, [particularly on
android](https://github.com/rust-lang/rust/pull/119147#issuecomment-1863712897),
and those will need to be fixed before this reaches beta. This is
something I am looking into.
Some cross-compiling scenarios may need codegen flags that are not
supported. It's not clear how common this will be, or if ignoring will
be a solution, or how difficult it would be to update rustdoc and cargo
to support these. Additionally, the split between RUSTFLAGS and
RUSTDOCFLAGS can be cumbersome.
## Implementation history
- https://github.com/rust-lang/rust/pull/60387 -- Support added to
rustdoc to support the `--target` flag and runtool and
per-target-ignores.
- https://github.com/rust-lang/cargo/pull/6892 -- Initial support in
cargo.
- https://github.com/rust-lang/cargo/pull/7391 -- Added unstable
documentation.
- https://github.com/rust-lang/cargo/pull/8094 -- Fix target for doc
test cross compilation
- https://github.com/rust-lang/cargo/pull/8358 -- Fixed regression with
`--target=HOST` not working on stable.
- https://github.com/rust-lang/cargo/pull/10132 -- Added note about
doctests not running (under `--verbose`).
- https://github.com/rust-lang/rust/pull/112751 -- Fixed
`--test-run-directory` interaction with `--test-runtool`.
- https://github.com/rust-lang/rust/pull/137096 -- Stabilization (and
rename) of the rustdoc `--test-runtool` and `--test-runtool-arg` CLI
args, and drops `--enable-per-target-ignores` unconditionally enabling
it.
## Test coverage
Cargo tests:
-
[artifact_dep::cross_doctests_works_with_artifacts](56c08f84e2/tests/testsuite/artifact_dep.rs (L1248-L1326))
-- Checks that doctest has access to the artifact dependencies.
-
[build_script::duplicate_script_with_extra_env](56c08f84e2/tests/testsuite/build_script.rs (L5514-L5614))
-- Checks that build-script env and cfg values are correctly handled on
host versus target when cross running doctests.
-
[cross_compile::cross_tests](56c08f84e2/tests/testsuite/cross_compile.rs (L416-L502))
-- Basic test that cross-compiled tests work.
-
[cross_compile::doctest_xcompile_linker](56c08f84e2/tests/testsuite/cross_compile.rs (L1139-L1182))
-- Checks that the linker config argument works.
-
[custom_target::custom_target_minimal](56c08f84e2/tests/testsuite/custom_target.rs (L39-L71))
-- Checks that `.json` targets work with rustdoc cross tests.
-
[test::cargo_test_doctest_xcompile_ignores](56c08f84e2/tests/testsuite/test.rs (L4743-L4777))
-- Checks the `ignore-*` syntax works.
-
[test::cargo_test_doctest_xcompile_runner](2603268cda/tests/testsuite/test.rs (L4783-L4863))
-- Checks runner with cross doctests.
-
[test::cargo_test_doctest_xcompile_no_runner](2603268cda/tests/testsuite/test.rs (L4869-L4907))
-- Checks cross doctests without a runner.
Rustdoc tests:
-
[run-make/doctest-runtool](25cdf1f674/tests/run-make/doctests-runtool)
-- Tests behavior of `--test-run-directory` with relative paths of the
runner.
-
[rustdoc/doctest/doctest-runtool](25cdf1f674/tests/rustdoc/doctest/doctest-runtool.rs)
-- Tests for `--test-runtool` and `--test-runtool-arg`.
## Future concerns
There have been some discussions
(https://github.com/rust-lang/testing-devex-team/issues/5) about
changing how doctests are driven. My understanding is that stabilizing
this should not affect those plans, since if cargo becomes the driver,
it will simply need to build things with `--target` and use the
appropriate runner.
## Change notes
This PR changed tests a little:
- artifact_dep::no_cross_doctests_works_with_artifacts was changed now
that doctests actually work.
- cross_compile::cross_tests was changed to properly check doctests.
- cross_compile::no_cross_doctests dropped since it is no longer
relevant.
- standard_lib::doctest didn't need `-Zdoctest-xcompile` since
`-Zbuild-std` no longer uses a target.
- test::cargo_test_doctest_xcompile was removed since it is a duplicate
of cross_compile::cross_tests
I think this should probably wait until the next release cutoff, moving
this to 1.89 (will update the PR accordingly if that happens).
### What does this PR try to resolve?
Fixes#15006
This changes how cargo-publish works with the unstable
`-Zpackage-workspace` feature.
Before this, when publishing the entire workspace,
like `cargo publish --workspace`,
if there is a package with `package.pulibsh=false,
it'll fail the entire publish process.
After this, when `--workspace` is passed,
or when publishing the virtual workspace,
the intent is more like “publish all publishable in this workspace”,
so skip `publish=false` packages and proceed to publish others.
The new overall behavior looks like this:
* `cargo publish` (inside a `package.publish = false` package): error
* `cargo publish -p publishable -p unpublishable`: error
* `cargo publish --workspace`: skips `package.publish = false
See
https://github.com/rust-lang/cargo/issues/15006#issuecomment-2847660911
### How should we test and review this PR?
* `workspace_flag_with_unpublishable_packages` was added to ensure
`--workspace` work with non-virtual workspace.
* `unpublishable_package_as_versioned_dev_dep` was added to ensure
versioned dev-dependencies won't be skipped and still fail
cargo-publish, as they are required to be published.
There is a new scenario that nothing is going to publish.
I went with a warning instead of hard error because missing publish for
the entire worspace should be a fairly visible. However, this is open
for future to configure via Cargo lint system.
This doesn't need to be an hard error because missing
publish for the entire worspace should be a fairly visible.
However, this is open for future to configure via Cargo lint system.
This changes how cargo-publish works with the unstable
`-Zpackage-workspace` feature.
Before this, when publishing the entire workspace,
like `cargo publish --workspace`,
if there is a package with `package.pulibsh=false,
it'll fail the entire publish process.
After this, when `--workspace` is passed,
or when publishing the virtual workspace,
the intent is more like “publish all publishable in this workspace”,
so skip `publish=false` packages and proceed to publish others.
The new overall behavior looks like this:
- `cargo publish` (inside a `package.publish = false` package): error
- `cargo publish -p publishable -p unpublishable`: error
- `cargo publish --workspace`: skips `package.publish = false
See https://github.com/rust-lang/cargo/issues/15006#issuecomment-2847660911
### What does this PR try to resolve?
This PR implements glob pattern match for known_hosts file. Hosts
written with `*` and `?` now matches correctly.
### Tests
Tests are added accordingly.
### Miscs
This is my first time submitting PR, sorry if there's anything that's
off although I've read the contributor guide.
This reverts commit 15a7672dde9853a8b15b545a1e5da5bbee7244c9.
### What does this PR try to resolve?
There was a miscommunication about some outstanding concerns with #15480
and it got merged before we were ready.
Specifically
- How important is this for unstable docs?
- How universal is the use of `bash-completions` and what should we do
for people who don't have it?
- How likely is someone to catch that they don't have `XDG_DATA_HOME`
set and get tripped up over this documentation
- Should we be aligned with upstream clap?
### How should we test and review this PR?
### Additional information
`~/.local/share/bash-completion/completions/cargo` is more sensible, and
[recommended by
rustup](e4f3ad6f89/src/cli/help.rs (L275)).
<!--
Thanks for submitting a pull request 🎉! Here are some tips for you:
* If this is your first contribution, read "Cargo Contribution Guide"
first:
https://doc.crates.io/contrib/
* Run `cargo fmt --all` to format your code changes.
* Small commits and pull requests are always preferable and easy to
review.
* If your idea is large and needs feedback from the community, read how:
https://doc.crates.io/contrib/process/#working-on-large-features
* Cargo takes care of compatibility. Read our design principles:
https://doc.crates.io/contrib/design.html
* When changing help text of cargo commands, follow the steps to
generate docs:
https://github.com/rust-lang/cargo/tree/master/src/doc#building-the-man-pages
* If your PR is not finished, set it as "draft" PR or add "WIP" in its
title.
* It's ok to use the CI resources to test your PR, but please don't
abuse them.
### What does this PR try to resolve?
Explain the motivation behind this change.
A clear overview along with an in-depth explanation are helpful.
You can use `Fixes #<issue number>` to associate this PR to an existing
issue.
### How should we test and review this PR?
Demonstrate how you test this change and guide reviewers through your
PR.
With a smooth review process, a pull request usually gets reviewed
quicker.
If you don't know how to write and run your tests, please read the
guide:
https://doc.crates.io/contrib/tests
### Additional information
Other information you want to mention in this PR, such as prior arts,
future extensions, an unresolved problem, or a TODO list.
-->
### What does this PR try to resolve?
Cargo registries that return HTTP 429 when the service is overloaded
expect the client to retry the request automatically after a delay.
Cargo currently does not retry for HTTP 429.
### What changed?
* Adds HTTP 429 (too many requests) as a spurious HTTP error to enable
retries.
* Parse the
[Retry-After](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After)
HTTP header to determine how long to wait before a retry.
In this implementation, the maximum delay is limited to Cargo's existing
limit of 10 seconds. We could consider increasing that limit for this
case, since the server is explicitly requesting the delay.
I'm not certain, but I do not believe it was intentional in
https://github.com/rust-lang/cargo/pull/15000 to allow the `schema` job
to fail. This adds it to the required passing list.
The Renovate config in this repository needs migrating. Typically this
is because one or more configuration options you are using have been
renamed.
You don't need to merge this PR right away, because Renovate will
continue to migrate these fields internally each time it runs. But later
some of these fields may be fully deprecated and the migrations removed.
So it's a good idea to merge this migration PR soon.
#### [PLEASE
NOTE](https://docs.renovatebot.com/configuration-options#configmigration):
JSON5 config file migrated! All comments & trailing commas were removed.
🔕 **Ignore**: Close this PR and you won't be reminded about config
migration again, but one day your current config may no longer be valid.
❓ Got questions? Does something look wrong to you? Please don't hesitate
to [request help
here](https://redirect.github.com/renovatebot/renovate/discussions).
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/rust-lang/cargo).
I'm not certain, but I do not believe it was intentional in https://github.com/rust-lang/cargo/pull/15000 to allow the `schema` job to fail. This adds it to the required passing list.
### What does this PR try to resolve?
Cargo script considers itself part of the workspace if the script is
located in a sub-directory of a workspace (presumably since
https://github.com/rust-lang/cargo/pull/15168). This becomes an issue
when using a custom registry that is defined in the `.cargo/config.toml`
within the workspace. Cargo script does not take that file into account
and fails with ``registry index was not found in any configuration:
`test-reg` ``.
### How should we test and review this PR?
This PR adds a regression test and makes cargo script ignore the
surrounding workspace.
The test ~~will fail without the fix in the second commit and~~ can be
used to reproduce the issue.
### Additional information
The issue started occurring with `nightly-2025-02-16`.
Related to https://github.com/rust-lang/cargo/issues/12207.
<!--
Thanks for submitting a pull request 🎉! Here are some tips for you:
* If this is your first contribution, read "Cargo Contribution Guide"
first:
https://doc.crates.io/contrib/
* Run `cargo fmt --all` to format your code changes.
* Small commits and pull requests are always preferable and easy to
review.
* If your idea is large and needs feedback from the community, read how:
https://doc.crates.io/contrib/process/#working-on-large-features
* Cargo takes care of compatibility. Read our design principles:
https://doc.crates.io/contrib/design.html
* When changing help text of cargo commands, follow the steps to
generate docs:
https://github.com/rust-lang/cargo/tree/master/src/doc#building-the-man-pages
* If your PR is not finished, set it as "draft" PR or add "WIP" in its
title.
* It's ok to use the CI resources to test your PR, but please don't
abuse them.
Explain the motivation behind this change.
A clear overview along with an in-depth explanation are helpful.
You can use `Fixes #<issue number>` to associate this PR to an existing
issue.
### How should we test and review this PR?
Demonstrate how you test this change and guide reviewers through your
PR.
With a smooth review process, a pull request usually gets reviewed
quicker.
If you don't know how to write and run your tests, please read the
guide:
https://doc.crates.io/contrib/tests
### Additional information
Other information you want to mention in this PR, such as prior arts,
future extensions, an unresolved problem, or a TODO list.
-->
### What does this PR try to resolve?
Fixes#15493
### How should we test and review this PR?
This takes the most surgical, direct route to addressing the problem.
Alternatively, we could look into why `cargo rustc` and `cargo check`
are different.
### Additional information
This takes the most surgical, direct route to addressing the problem.
Alternatively, we could look into why `cargo rustc` and `cargo check`
are different.
Fixes#15493
### What does this PR try to resolve?
This PR adds Cargo integration for the new unstable `-Zembed-metadata`
rustc flag, which was implemented in
https://github.com/rust-lang/rust/pull/137535 ([tracking
issue](https://github.com/rust-lang/rust/issues/139165)). The new
behavior has to be enabled explicitly using a new unstable CLI flag
`-Zno-embed-metadata`.
The `-Zembed-metadata=no` rustc flag can reduce disk usage of compiled
artifacts, and also the size of Rust dynamic library artifacts shipped
to users. However, it is not enough to just pass this flag through
`RUSTFLAGS`; it needs to be integrated within Cargo, because it
interacts with how the `--emit` flag is passed to rustc, and also how
`--extern` args are passed to the final linked artifact build by Cargo.
Furthermore, using the flag for all crates in a crate graph compiled by
Cargo would be suboptimal (this will all be described below).
When you pass `-Zembed-metadata=no` to rustc, it will not store Rust
metadata into the compiled artifact. This is important when compiling
libs/rlibs/dylibs, since it reduces their size on disk. However, this
also means that everytime we use this flag, we have to make sure that we
also:
- Include `metadata` in the `--emit` flag to generate a `.rmeta` file,
otherwise no metadata would be generated whatsoever, which would mean
that the artifact wouldn't be usable as a dependency.
- Pass also `--extern <dep>=<path>.rmeta` when compiling the final
linkable artifact. Before, Cargo would only pass `--extern
<dep>=<path>.[rlib|so|dll]`. Since with `-Zembed-metadata=no`, the
metadata is only in the `.rmeta` file and not in the rlib/dylib, this is
needed to help rustc find out where the metadata lies.
- Note: this essentially doubles the cmdline length when compiling the
final linked artifact. Not sure if that is a concern.
The two points above is what this PR implements, and why this rustc flag
needs Cargo integration.
The `-Zembed-metadata` flag is only passed to libs, rlibs and dylibs. It
does not seem to make sense for other crate types. The one situation
where it might make sense are proc macros, but according to @bjorn3 (who
initially came up with the idea for `-Zembed-metadata`, it isn't really
worth it).
Here is a table that summarizes the changes in passed flags and
generated files on disk for rlibs and dylibs:
| **Crate type** | **Flags** | **Generated files** | **Disk usage** |
|--|--|--|--|
| Rlib/Lib (before) | `--emit=dep-info,metadata,link` | `.rlib` (with
metadata), `.rmeta` (for pipelining) | - |
| Rlib/Lib (after) | `--emit=dep-info,metadata,link -Zembed-metadata=no`
| `.rlib` (without metadata), `.rmeta` (for metadata/pipelining) |
Reduced (metadata no longer duplicated) |
| Dylib (before) | `--emit=dep-info,link` | `[.so\|.dll]` (with
metadata) | - |
| Dylib (after) | `--emit=dep-info,metadata,link -Zembed-metadata=no` |
`[.so\|.dll]` (without metadata), `.rmeta` | Unchanged, but split
between two files |
Behavior for other target kinds/crate types should be unchanged.
From the table above, we can see two benefits of using
`-Zembed-metadata=no`:
- For rlibs/dylibs, we no longer store their metadata twice in the
target directory, thus reducing target directory size.
- For dylibs, we store esssentially the same amount of data on disk, but
the benefit is that the metadata is now in a separate .rmeta file. This
means that you can ship the dylib (`.so`/`.dll`) to users without also
shipping the metadata. This would slightly reduce e.g. the
[size](https://github.com/rust-lang/rust/pull/120855#issuecomment-1937018169)
of the shipped rustc toolchains (note that the size reduction here is
after the toolchain has been already heavily compressed).
Note that if this behavior ever becomes the default, it should be
possible to simplify the code quite a bit, and essentially merge the
`requires_upstream_objects` and `benefits_from_split_metadata`
functions.
I did a very simple initial benchmark to evaluate the space savings on
cargo itself and
[hyperqueue](https://github.com/It4innovations/hyperqueue) (a mid-size
crate from my work) using `cargo build` and `cargo build --release` with
and without `-Zembed-metadata=no`:
For debug/incremental builds, the effect is smaller, as the artifact
disk usage is dwarfed by incremental artifacts and debuginfo. But for
(non-incremental) release builds, the disk savings (and also performed
I/O operations) are significantly reduced.
### How should we test and review this PR?
I wrote two basic tests. The second one tests a situation where a crate
depends on a dylib dependency, which is quite rare, but the behavior of
this has actually changed in this PR (see comparison table above).
Testing this on various real-world projects (or even trying to enable it
by default across the whole Cargo suite?) might be beneficial.
## Unresolved questions
### Is this a breaking change?
With this new behavior, dylibs and rlibs will no longer contain
metadata. If they are compiled with Cargo, that shouldn't matter, but
other build systems might have to adapt.
### Should this become the default?
I think that in terms of disk size usage and performed I/O operations,
it is a pure win. It should either generate less disk data (for rlibs)
or the ~same amount of data for dylibs (the data will be a bit larger,
because the dylib will still contain a metadata stub header, but that's
like 50 bytes and doesn't scale with the size of the dylib, so it's
negligible).
So I think that eventually, we should just do this by default in Cargo,
unless some concerns are found. I suppose that before stabilizing we
should also benchmark the effect on compilation performance.
This moves the closure used for computing artifact dependency
information out into a separate function. The large line lengths were
causing rustfmt to fail to format the entire parent closure. Also, the
rightwards drift was getting quite extreme.
This also simplifies the code a little by avoiding all the transpose
stuff and needing the separate match at the end.
This moves the closure used for computing artifact dependency
information out into a separate function. The large line lengths were
causing rustfmt to fail to format the entire parent closure. Also,
the rightwards drift was getting quite extreme.
This also simplifies the code a little by avoiding all the transpose
stuff and needing the separate match at the end.
### What does this PR try to resolve?
- Added an error message when version in `CRATE[@<VER>]` or `--version
<VER>` starts with 'v' for `install`, `add`, `yank` and `update
--precise <VER>`
- Check if version is valid in `cargo yank`
Fixes#12331
### How should we test and review this PR?
Added tests for each subcommand
