I.e. the type definition, then a single inherent `impl` block, then the
trait `impl` blocks.
The lack of sensible ordering here has bugged me for some time.
This commit does the following.
- Changes it from `Lrc<Box<dyn ToAttrTokenStream>>` to
`Lrc<LazyAttrTokenStreamInner>`.
- Reworks `LazyAttrTokenStreamImpl` as `LazyAttrTokenStreamInner`, which
is a two-variant enum.
- Removes the `ToAttrTokenStream` trait and the two impls of it.
The recursion limit must be increased in some crates otherwise rustdoc
aborts.
Specifically: `TokenCursor`, `TokenTreeCursor`,
`LazyAttrTokenStreamImpl`, `FlatToken`, `make_attr_token_stream`,
`ParserRange`, `NodeRange`. `ParserReplacement`, and `NodeReplacement`.
These are all related to token streams, rather than actual parsing.
This will facilitate the simplifications in the next commit.
They are no longer needed.
This does slightly worsen the error message for a single test, but that
test contains code that is so badly broken that I'm not worried about
it.
Notes about tests:
- tests/ui/rfcs/rfc-2294-if-let-guard/feature-gate.rs: some messages are
now duplicated due to repeated parsing.
- tests/ui/rfcs/rfc-2497-if-let-chains/disallowed-positions.rs: ditto.
- `tests/ui/proc-macro/macro-rules-derive-cfg.rs`: the diff looks large
but the only difference is the insertion of a single
invisible-delimited group around a metavar.
- `tests/ui/attributes/nonterminal-expansion.rs`: a slight span
degradation, somehow related to the recent massive attr parsing
rewrite (#135726). I couldn't work out exactly what is going wrong,
but I don't think it's worth holding things up for a single slightly
suboptimal error message.
This involves replacing `nt_pretty_printing_compatibility_hack` with
`stream_pretty_printing_compatibility_hack`.
The handling of statements in `transcribe` is slightly different to
other nonterminal kinds, due to the lack of `from_ast` implementation
for empty statements.
Notable test changes:
- `tests/ui/proc-macro/expand-to-derive.rs`: the diff looks large but
the only difference is the insertion of a single invisible-delimited
group around a metavar.
For consistency with `rustc_lexer::TokenKind::Bang`, and because other
`ast::TokenKind` variants generally have syntactic names instead of
semantic names (e.g. `Star` and `DotDot` instead of `Mul` and `Range`).
Note: there was an existing code path involving `Interpolated` in
`MetaItem::from_tokens` that was dead. This commit transfers that to the
new form, but puts an `unreachable!` call inside it.
The one notable test change is `tests/ui/macros/trace_faulty_macros.rs`.
This commit removes the complicated `Interpolated` handling in
`expected_expression_found` that results in a longer error message. But
I think the new, shorter message is actually an improvement.
The original complaint was in #71039, when the error message started
with "error: expected expression, found `1 + 1`". That was confusing
because `1 + 1` is an expression. Other than that, the reporter said
"the whole error message is not too bad if you ignore the first line".
Subsequently, extra complexity and wording was added to the error
message. But I don't think the extra wording actually helps all that
much. In particular, it still says of the `1+1` that "this is expected
to be expression". This repeats the problem from the original complaint!
This commit removes the extra complexity, reverting to a simpler error
message. This is primarily because the traversal is a pain without
`Interpolated` tokens. Nonetheless, I think the error message is
*improved*. It now starts with "expected expression, found `pat`
metavariable", which is much clearer and the real problem. It also
doesn't say anything specific about `1+1`, which is good, because the
`1+1` isn't really relevant to the error -- it's the `$e:pat` that's
important.
Notes about tests:
- tests/ui/parser/macro/trait-object-macro-matcher.rs: the syntax error
is duplicated, because it occurs now when parsing the decl macro
input, and also when parsing the expanded decl macro. But this won't
show up for normal users due to error de-duplication.
- tests/ui/associated-consts/issue-93835.rs: similar, plus there are
some additional errors about this very broken code.
- The changes to metavariable descriptions in #132629 are now visible in
error message for several tests.
- Move it to `rustc_parse`, which is the only crate that uses it. This
lets us remove all the `pub` markers from it.
- Change `next_ref` and `look_ahead` to `get` and `bump`, which work
better for the `rustc_parse` uses.
- This requires adding a `TokenStream::get` method, which is simple.
- In `TokenCursor`, we currently duplicate the
`DelimSpan`/`DelimSpacing`/`Delimiter` from the surrounding
`TokenTree::Delimited` in the stack. This isn't necessary so long as
we don't prematurely move past the `Delimited`, and is a small perf
win on a very hot code path.
- In `parse_token_tree`, we clone the relevant `TokenTree::Delimited`
instead of constructing an identical one from pieces.
Currently there are two ways to peek at a `TokenStreamIter`.
- Wrap it in a `Peekable` and use that traits `peek` method.
- Use `TokenStreamIter`'s inherent `peek` method.
Some code uses one, some use the other. This commit converts all places
to the inherent method. This eliminates mixing of `TokenStreamIter` and
`Peekable<TokenStreamIter>` and some use of `impl Iterator` and `dyn
Iterator`.
Because `TokenStreamIter` is a much better name for a `TokenStream`
iterator. Also rename the `TokenStream::trees` method as
`TokenStream::iter`, and some local variables.
Since it's inception a long time ago, the parallel compiler and its cfgs
have been a maintenance burden. This was a necessary evil the allow
iteration while not degrading performance because of synchronization
overhead.
But this time is over. Thanks to the amazing work by the parallel
working group (and the dyn sync crimes), the parallel compiler has now
been fast enough to be shipped by default in nightly for quite a while
now.
Stable and beta have still been on the serial compiler, because they
can't use `-Zthreads` anyways.
But this is quite suboptimal:
- the maintenance burden still sucks
- we're not testing the serial compiler in nightly
Because of these reasons, it's time to end it. The serial compiler has
served us well in the years since it was split from the parallel one,
but it's over now.
Let the knight slay one head of the two-headed dragon!
This commit does the following.
- Pulls the code out of `AttrTokenStream::to_token_trees` into a new
function `attrs_and_tokens_to_token_trees`.
- Simplifies `TokenStream::from_ast` by calling the new function. This
is nicer than the old way, which created a temporary
`AttrTokenStream` containing a single `AttrsTarget` (which required
some cloning) just to call `to_token_trees` on it. (It is good to
remove this use of `AttrsTarget` which isn't related to `cfg_attr`
expansion.)
The only place it is meaningfully used is in a panic message in
`TokenStream::from_ast`. But `node.span()` doesn't need to be printed
because `node` is also printed and it must contain the span.
Currently it uses a mixture of functional style (`flat_map`) and
imperative style (`push`), which is a bit hard to read. This commit
converts it to fully imperative, which is more concise and avoids the
need for `smallvec`.
I.e. change the return type from `TokenStream` to `Vec<TokenTree>`.
Most of the callsites require a `TokenStream`, but the recursive call
used to create `target_tokens` requires a `Vec<TokenTree>`. It's easy
to convert a `Vec<TokenTree>` to a `TokenStream` (just call
`TokenStream::new`) but it's harder to convert a `TokenStream` to a
`Vec<TokenTree>` (either iterate/clone/collect, or use `Lrc::into_inner`
if appropriate).
So this commit changes the return value to simplify that `target_tokens`
call site.
The extra span is now recorded in the new `TokenKind::NtIdent` and
`TokenKind::NtLifetime`. These both consist of a single token, and so
there's no operator precedence problems with inserting them directly
into the token stream.
The other way to do this would be to wrap the ident/lifetime in invisible
delimiters, but there's a lot of code that assumes an interpolated
ident/lifetime fits in a single token, and changing all that code to work with
invisible delimiters would have been a pain. (Maybe it could be done in a
follow-up.)
This change might not seem like much of a win, but it's a first step toward the
much bigger and long-desired removal of `Nonterminal` and
`TokenKind::Interpolated`. That change is big and complex enough that it's
worth doing this piece separately. (Indeed, this commit is based on part of a
late commit in #114647, a prior attempt at that big and complex change.)
This span records the declaration of the metavariable in the LHS of the macro.
It's used in a couple of error messages. Unfortunately, it gets in the way of
the long-term goal of removing `TokenKind::Interpolated`. So this commit
removes it, which degrades a couple of (obscure) error messages but makes
things simpler and enables the next commit.
