They can't contain `\x` escapes, which means they can't contain high
bytes, which means we can used `unescape_unicode` instead of
`unescape_mixed` to unescape them. This avoids unnecessary used of
`MixedUnit`.
`unescape_literal` becomes `unescape_unicode`, and `unescape_c_string`
becomes `unescape_mixed`. Because rfc3349 will mean that C string
literals will no longer be the only mixed utf8 literals.
- Rename it as `MixedUnit`, because it will soon be used in more than
just C string literals.
- Change the `Byte` variant to `HighByte` and use it only for
`\x80`..`\xff` cases. This fixes the old inexactness where ASCII chars
could be encoded with either `Byte` or `Char`.
- Add useful comments.
- Remove `is_ascii`, in favour of `u8::is_ascii`.
The `T` type in these functions took me some time to understand, and I
find the explicit `T` in the use of `from` makes the code easier to
read, as does the `u8` annotation in `scan_escape`.
By making it an `EscapeError` instead of a `LitError`. This makes it
like the other errors produced when checking string literals contents,
e.g. for invalid escape sequences or bare CR chars.
NOTE: this means these errors are issued earlier, before expansion,
which changes behaviour. It will be possible to move the check back to
the later point if desired. If that happens, it's likely that all the
string literal contents checks will be delayed together.
One nice thing about this: the old approach had some code in
`report_lit_error` to calculate the span of the nul char from a range.
This code used a hardwired `+2` to account for the `c"` at the start of
a C string literal, but this should have changed to a `+3` for raw C
string literals to account for the `cr"`, which meant that the caret in
`cr"` nul error messages was one short of where it should have been. The
new approach doesn't need any of this and avoids the off-by-one error.
`unescape_raw_str_or_raw_byte_str` only does checking, no unescaping.
And it also now handles C string literals.
`unescape_raw_str` is used for all the non-raw strings.
- Add `use Mode::*` to avoid all the qualifiers.
- Reorder the variants. The existing order makes no particular sense,
which has bugged me for some time. I've chosen an order that makes
sense to me.
These don't really make sense since C string literals were added. This
commit removes them in favour for `mode: Mode` args. `ascii_check` still
has a `characters_should_be_ascii: bool` arg.
Also, `characters_should_be_ascii` is renamed to be shorter.
There are three kinds of "byte" literals: byte literals, byte string
literals, and raw byte string literals. None are allowed to have
non-ASCII chars in them.
Two `EscapeError` variants exist for when that constraint is violated.
- `NonAsciiCharInByte`: used for byte literals and byte string literals.
- `NonAsciiCharInByteString`: used for raw byte string literals.
As a result, the messages for raw byte string literals use different
wording, without good reason. Also, byte string literals are incorrectly
described as "byte constants" in some error messages.
This commit eliminates `NonAsciiCharInByteString` so the three cases are
handled similarly, and described correctly. The `mode` is enough to
distinguish them.
Note: Some existing error messages mention "byte constants" and some
mention "byte literals". I went with the latter here, because it's a
more correct name, as used by the Reference.
It's passed to numerous places where we just need an `is_byte` bool.
Passing the bool avoids the need for some assertions.
Also rename `is_bytes()` as `is_byte()`, to better match `Mode::Byte`,
`Mode::ByteStr`, and `Mode::RawByteStr`.
These have been bugging me for a while.
- `literal_text`: `src` is also used and is shorter and better.
- `first_char`: used even when "first" doesn't make sense; `c` is
shorter and better.
- `curr`: `c` is shorter and better.
- `unescaped_char`: `result` is also used and is shorter and better.
- `second_char`: these have a single use and can be elided.
- Rename `unescape_raw_str_or_raw_byte_str` as
`unescape_raw_str_or_byte_str`, which is more accurate.
- Remove the unused `Mode::in_single_quotes` method.
- Make some assertions more precise, and add a missing one to
`unescape_char_or_byte`.
- Change all the assertions to `debug_assert!`, because this code is
reasonably hot, and the assertions aren't required for memory safety,
and any violations are likely to be sufficiently obvious that normal
tests will trigger them.
`scan_escape` currently has a fast path (for when the first char isn't
'\\') and a slow path.
This commit changes `scan_escape` so it only handles the slow path, i.e.
the actual escaping code. The fast path is inlined into the two call
sites.
This change makes the code faster, because there is no function call
overhead on the fast path. (`scan_escape` is a big function and doesn't
get inlined.)
This change also improves readability, because it removes a bunch of
mode checks on the the fast paths.
The PR had some unforseen perf regressions that are not as easy to find.
Revert the PR for now.
This reverts commit 6ae8912a3e7d2c4c775024f58a7ba4b1aedc4073, reversing
changes made to 86d6d2b7389fe1b339402c1798edae8b695fc9ef.
