This example demonstrates how to use the `tracing-subscriber` crate's
`EnvFilter` type to filter log messages based on their metadata. The
example provides a text area where users can input an environment filter
string, and displays the log messages that would be captured by that
filter.
## Motivation
I was checking the example from `/examples/` and saw that
some of the module names were incorrect in README, so I
fixed them.
## Solution
Changed the incorrect name to the correct name.
Most of these changes are places where lifetimes were named, but can be
elided. Then a few cases where a lifetime was elided, but actually
resolves to a named lifetime. So lots of lifetimes.
This is the `v0.1.x` branch sister PR to #3164 (for the `master`
branch), since `clippy --fix` on another branch is a much better way to
apply these changes than backporting.
This results in a substantial performance improvement,
and is compatible with our MSRV.
Signed-off-by: Alex Saveau <saveau.alexandre@gmail.com>
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
There is an incompatibility with the version of Node available on our
test runners and wasm32 in Rust 1.82 (#3123).
To unblock the CI, this change pins Rust to 1.81 for the tests using the
`wasm32-unknown-unknown` target. This is the same strategy used in Tokio
to mitigate tokio-rs/tokio#6910 until a more permanent fix can be put in
place.
This change also bumps the MSRV on the `tracing-examples` crate from
1.63.0 to 1.64.0 to avoid triggering a lint about the MSRV after a
change in Tokio 1.41.0 which bumps the required Rust version for the
`try_join!` macro. The Tokio MSRV is 1.70 now, so needing this bump for
the examples seems reasonable.
Removing the `env_logger` feature in order to address GHSA-g98v-hv3f-hcfr.
In addition, this PR also removes the deprecated `trace_logger` module, in
preparation for an upcoming v0.2.0 of `tracing-log`.
For additional details on the approach, please refer to #2750. Note that this
PR depends on #2770, so this PR will temporarily have more commits
than intended.
---------
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
Currently, the `hyper_echo` example uses the `tracing-log` env logger
support for some weird reason. I believe this is due to Hyper previously
using `log` rather than `tracing`. However, `hyper` now emits native
`tracing` diagnostics, so all the `env_logger` nonsense can just be
removed from the example.
This branch does that.
; Conflicts:
; examples/Cargo.toml
; examples/examples/hyper-echo.rs
This PR removes tracing-opentelemetry to a dedicated repo located at
https://github.com/tokio-rs/tracing-opentelemetry. (Note that at time of
writing this PR, the new repo has not be made public). We're moving
tracing-opentelemetry to a dedicated repository for the following
reasons:
1. opentelemetry's MSRV is higher than that of `tracing`'s.
2. more importantly, the main `tracing` repo is getting a bit unweildy
and it feels unreasonable to maintain backports for crates that
integrate with the larger tracing ecosystem.
(https://github.com/tokio-rs/tracing-opentelemetry does not have the
examples present in this repo; this will occur in a PR that will be
linked from _this_ PR.)
## Motivation
Support the latest OpenTelemetry specification.
## Solution
Update `opentelemetry` to the latest `0.18.x` release. Breaking changes
in the metrics spec have removed value recorders and added histograms so
the metrics layer's `value.` prefix has been changed to `histogram.` and
behaves accordingly. Additionally the `PushController` configuration for
the metrics layer has been simplified to accept a `BasicController` that
can act in either push or pull modes. Finally trace sampling in the
sdk's `PreSampledTracer` impl has been updated to match the sampling
logic in https://github.com/open-telemetry/opentelemetry-rust/pull/839.
* Update MSRV to 1.56
* Update examples
* Fix async-trait dep
* Update msrv action
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
## Motivation
Fix minimal-versions failure.
## Solution
Upgrade all the dependencies to their most recent semver-compatible
version, adjusting back down as necessary for MSRV.
Essentially a cherry-pick of #2231, but redone by hand.
## Tests
- `cargo minimal-versions msrv verify -- cargo check --all-features`
- `cargo minimal-versions msrv verify -- cargo check --no-default-features`
## Methodology
- `cargo update && cargo upgrade --to-lockfile`
- Identify [a bug](https://github.com/killercup/cargo-edit/issues/750) and manually resolve it
- loop; upgrade transitive deps
- `cargo minimal-versions check --all-features`
- Identify failing dep
- `cargo minimal-versions tree -i dep --all-features`
- Find the closest dependency leading to pulling in `dep`
- `cargo add fixdep --optional` to force a more recent more-minimal-versions-correct version
- loop; downgrade to msrv
- `cargo minimal-versions msrv verify -- cargo check --all-features`
- Identify failing dep
- `cargo minimal-versions tree -i dep --all-features`
- Find the version that supports MSRV from lib.rs
- `cargo upgrade dep@msrv`
This patch adds a bit more context around why we are creating a smaller
scope for the spans, and also what happens when we call
`global::shutdown_tracer_provider()` (that comment was copied from
the`rust-opentelemetry` repo).
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
Replace `lazy_static` with `once_cell`. Fixes#2146.
## Motivation
`lazy_static!`, while a declarative macro, is a macro nonetheless. It
can add quite a bit of additional compilation time cost.
`once_cell::sync::Lazy` does the same thing with generics, and can be
used more flexibly (i.e. non-static lazily initialized values), and has
been proposed to be added to `std` (see linked issue).
I'm trying to reduce the compile time and dependency tree complexity of
a dependent project: [bevy](https://bevyengine.org), which is using
tracing. `lazy_static` and `once_cell` are both in our dependency tree
and both end up doing the same thing.
## Solution
Migrate to `once_cell`.
## Motivation
`Value` was already implemented for `dyn Error + 'static`, but rust
doesn't silently coerce trait objects. This means that passing an error
of type `dyn Error + Send + Sync + 'static` would not work. This is
related to #1308.
## Solution
Add impls for `dyn Error + …` variants for `Send`, `Sync`, and `Send +
Sync`. These extra impls just delegate to the existing `dyn Error +
'static` impl.
Also update one of the examples to use `dyn Error + Send + Sync` to
demonstrate that this works.
Refs: #1308
## Motivation
Currently, the RustDoc for the `format::Compact` formatter in
`tracing-subscriber` describes the output from the `master` (v0.2.x)
version of the formatter, not the version on the v0.1.x branch.
## Solution
This PR updates the documentation to describe the actual output format.
Also, I added an example of the formatter in the `examples` directory.
Closes#1909
This updates all crates' MSRVs to 1.49 if they were not already greater
than that (`tracing-appender` is at 1.53). Rust 1.49+ is required to
update `parking_lot` to v0.12 (see #1878). Also, `futures-task` (which I
believe is only needed as a transitive dep) now needs 1.45+, so this
also fixes our CI build.
Because `tracing-opentelemetry` previously required 1.46.0, it had a
separate CI MSRV job. Since 1.49.0 is greater than 1.46.0, the separate
check for `tracing-opentelemetry` is no longer needed.
In the process, I removed deprecated uses of
`core::atomic::spin_loop_hint`, which is replaced with
`core::hint::spin_loop` in 1.49.
This branch introduces support for `valuable` in `tracing-subscriber`'s
JSON formatter, and in `tracing-serde`, using the `valuable-serde`
bridge.
This allows the `fmt::Json` subscriber to actually record `valuable`
values as structured JSON. Here's an example, where a field is first
recorded using `fmt::Debug`, and then again using `field::valuable`:
```
:; RUSTFLAGS="--cfg tracing_unstable" cargo run --example valuable_json | jq
Finished dev [unoptimized + debuginfo] target(s) in 0.09s
Running `target/debug/examples/valuable_json`
{
"timestamp": "2022-01-25T21:36:30.729658Z",
"level": "INFO",
"fields": {
"valuable": false,
"user": "User { name: \"Arwen Undomiel\", age: 3000, address: Address { country: \"Middle Earth\", city: \"Rivendell\", street: \"leafy lane\" } }"
},
"target": "valuable_json"
}
{
"timestamp": "2022-01-25T21:36:30.729720Z",
"level": "INFO",
"fields": {
"valuable": true,
"user": {
"name": "Arwen Undomiel",
"age": 3000,
"address": {
"country": "Middle Earth",
"city": "Rivendell",
"street": "leafy lane"
}
}
},
"target": "valuable_json"
}
```
As a side note, this branch also nicely validates that recording `Valuable`
values from a subscriber is actually possible, which the previous valuable PR
didn't have an example of.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
* ci: add `tracing_unstable` to CI
Currently, the `valuable` support requires the `tracing_unstable` cfg to
be set. Because none of our current CI jobs set this, we aren't
currently testing that code, and have no way of even ensuring that it
compiles. This is Bad.
This PR adds a CI job to run tests with the unstable cfg enabled.
* core: fix wrong `self` types with `valuable`
This should fix the build.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
## Motivation
Support the latest OpenTelemetry specification.
## Solution
Update `opentelemetry` to the latest `0.17.x` release. Breaking changes
upstream in the tracking of parent contexts in otel's `SpanBuilder` have
necessitated a new `OtelData` struct to continue pairing tracing spans
with their associated otel `Context`.
# Conflicts:
# .github/workflows/check_msrv.yml
# tracing-opentelemetry/Cargo.toml
# tracing-opentelemetry/benches/trace.rs
# tracing-opentelemetry/src/layer.rs
# tracing-opentelemetry/src/span_ext.rs
# tracing-opentelemetry/tests/trace_state_propagation.rs
This changes the `valuable` integration so that `record_value` takes
instances of `valuable::Value<'_>`, rather than `&dyn
valuable::Valuable` trait objects. The primary advantage of this is that
a `Value` can be produced by calling `Valuable::as_value`, so it allows
users to write code like this:
```rust
#[derive(Valuable, Debug)]
struct Foo {
// ...
}
tracing::trace!(foo = foo.as_value());
```
rather than this:
```rust
#[derive(Valuable, Debug)]
struct Foo {
// ...
}
tracing::trace!(foo = tracing::field::valuable(&foo));
```
which feels a bit more ergonomic. It also simplifies the code in
`tracing-core`, since we no longer need our own `ValuableValue` wrapper
type to turn things into trait objects.
It might also reduce boilerplate a bit on the visitor side, as
`as_value()` doesn't have to be called on the trait object, although
that's probably not as big a deal.
I didn't remove the `field::valuable` function, as I thought it was nice
to have for consistency with the existing `field::debug` and
`field::display` functions.
## Performance Considerations
@carllerche pointed out that a `Value<'_>` might be slightly more bytes
to pass on the stack than a trait object (always two words). I believe
this is only the case when the `Value` is a `Listable`, `Enumerable`,
`Structable`, `Mappable`, or `Tupleable`, where the `Value` would be an
enum descriminant _and_ a wide pointer to a trait object. However, in
the cases where the value is a primitive, `Value` will be two words if
the primitive is word-sized (e.g. `u64` on 64-bit platforms), for the
enum descriminant + the value, or one word if the primitive is smaller
than word size (`bool`, `char`, etc). Also, for primitive `Value`s,
there's no pointer dereference, which the trait object always requires.
I'm not sure how the enum dispatch compares to vtable dispatch when
calling `visit` on the value. However, if the `tracing` visitor is going
to call `as_value()` on the recorded value, this approach is better,
because calling `as_value()` in the macro _prior_ to recording the
span/event will use the statically dispatched `as_value()` impl on a
known type, rather than the the dynamically dispatched `as_value()` impl
on the trait object. Since `as_value` impls are generally quite trivial,
I'd guess they usually (always?) will get inlined, which is never
possible with the dynamically dispatched call after passing a trait
object into `tracing`.
In practice I'm not sure if there's a huge perf diff either way, but it
was interesting to think through the implications.
This branch adds initial support for using the [`valuable`] crate as an
opt-in `Value` type in `tracing`. `valuable` provides a mechanism for
defining custom ways to record user-implemented types, as well as
structured recording of standard library data structures such as maps,
arrays, and sets.
For more details, see the tracking issue #1570.
In `tracing` v0.2, the intent is for `valuable` to replace the existing
`tracing_core::field::Value` trait. However, in v0.1.x, `valuable`
support must be added in a backwards-compatible way, so recording types
that implement `valuable::Valueable` currently requires opting in using
a `field::valuable` wrapper function.
Since this is the first release of `valuable` and the API is still
stabilizing, we don't want to tie `tracing-core`'s stability to
`valuable`'s. Therefore, the valuable dependency is feature-flagged
*and* also requires `RUSTFLAGS="--cfg tracing_unstable"`.
[`valuable`]: https://github.com/tokio-rs/valuable
Co-authored-by: Daniel McKenna <daniel@emotech.co>
Co-authored-by: David Barsky <me@davidbarsky.com>
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
## Motivation
Logging line numbers and file names can be useful for debugging. This
feature was suggested by #1326
## Solution
As per @hawkw's suggestions, fields were added on `Format`, along with
builder methods. Filename and line number information was gathered from
the `meta` variable.
The `Pretty` formatter already supports printing source locations, but
this is configured separately on the `Pretty` formatter rather than on
the `Format` type. This branch also changes `Pretty` to honor the
`Format`-level configurations and deprecates the `Pretty`-specific
method.
Fixes#1326Closes#1804
Co-authored-by: David Barsky <me@davidbarsky.com>
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
## Motivation
Currently, `tracing-appender`'s `RollingFileAppender` does not implement
the `MakeWriter` trait. This means it can only be used by either
wrapping it in `NonBlocking`, or by wrapping it in a `Mutex`. However,
this shouldn't be strictly necessary, as `&File` implements `io::Write`.
It should thus only be necessary to introduce locking when we are in the
process of _rotating_ the log file.
## Solution
This branch adds a `MakeWriter` implementation for
`RollingFileAppender`. This is done by moving the file itself inside of
an `RwLock`, so that a read lock is acquired to write to the file. This
allows multiple threads to write to the file without contention. When
the file needs to be rolled, the rolling thread acquires the write lock
to replace the file. Acquiring the write lock is guarded by an atomic
CAS on the timestamp, so that only a single thread will try to roll the
file. This prevents other threads from immediately rolling the file
_again_ when the write lock is released.
I...should probably write tests for that, though.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
# 0.2.0 (October 22, 2021)
This breaking change release adds support for the new v0.3.x series of
`tracing-subscriber`. In addition, it resolves the security advisory for
the `chrono` crate, [RUSTSEC-2020-0159].
This release increases the minimum supported Rust version (MSRV) to
1.51.0.
### Breaking Changes
- Updated `tracing-subscriber` to v0.3.x ([#1677])
- Changed `NonBlocking::error_counter` to return an `ErrorCounter` type,
rather than an `Arc<AtomicU64>` ([#1675])
### Changed
- Updated `tracing-subscriber` to v0.3.x ([#1677])
### Fixed
- **non-blocking**: Fixed compilation on 32-bit targets ([#1675])
- **rolling**: Replaced `chrono` dependency with `time` to resolve
[RUSTSEC-2020-0159] ([#1652])
- **rolling**: Fixed an issue where `RollingFileAppender` would fail to
print errors that occurred while flushing a previous logfile ([#1604])
Thanks to new contributors @dzvon and @zvkemp for contributing to this
release!
[RUSTSEC-2020-0159]: https://rustsec.org/advisories/RUSTSEC-2020-0159.html
[#1677]: https://github.com/tokio-rs/tracing/pull/1677
[#1675]: https://github.com/tokio-rs/tracing/pull/1675
[#1652]: https://github.com/tokio-rs/tracing/pull/1675
[#1604]: https://github.com/tokio-rs/tracing/pull/1604
# 0.3.0 (Oct 22, 2021)
This is a breaking release of `tracing-subscriber`. The primary breaking
change in this release is the removal of the dependency on the [`chrono`
crate], due to [RUSTSEC-2020-0159]. To replace `chrono`, support is
added for formatting timestamps using the [`time` crate] instead.
In addition, this release includes a number of other breaking API
changes, such as adding (limited) support for `#![no_std]` targets,
removing previously deprecated APIs, and more.
### Breaking Changes
- Removed APIs deprecated in the v0.2.x release series.
- Renamed `Layer::new_span` to `Layer::on_new_span` ([#1674])
- Removed `Layer` impl for `Arc<L: Layer<S>>` and `Arc<dyn Layer<S> +
...>` ([#1649])
- Replaced the [`chrono` crate] with the [`time` crate] for timestamp
formatting, to resolve [RUSTSEC-2020-0159] ([#1646])
- Removed `json` and `env-filter` from default features. They must now
be enabled explictly ([#1647])
- Changed `FormatEvent::format_event` and `FormatFields::format_fields`
trait methods to take a `Writer` type, rather than a `&mut dyn
fmt::Write` trait object ([#1661])
- Changed the signature of the `MakeWriter` trait by adding a lifetime
parameter ([#781])
### Changed
- **layer**: Renamed `Layer::new_span` to `Layer::on_new_span` ([#1674])
- **fmt**: Changed `FormatEvent::format_event` and
`FormatFields::format_fields` trait methods to take a `Writer` type,
rather than a `&mut dyn fmt::Write` trait object ([#1661])
- **json**, **env-filter**: `json` and `env-filter` feature flags are no
longer enabled by default ([#1647])
### Removed
- Removed deprecated `CurrentSpan` type ([#1320])
- **registry**: Removed deprecated `SpanRef::parents` iterator, replaced
by `SpanRef::scope` in [#1431] ([#1648)])
- **layer**: Removed deprecated `Context::scope` iterator, replaced by
`Context::span_scope` and `Context::event_scope` in [#1431] and
[#1434] ([#1648)])
- **layer**: Removed `Layer` impl for `Arc<L: Layer<S>>` and `Arc<dyn
Layer<S> + ...>`. These interfere with per-layer filtering. ([#1649])
- **fmt**: Removed deprecated `LayerBuilder` type ([#1673])
- **fmt**: Removed `fmt::Layer::on_event` (renamed to
`fmt::Layer::fmt_event`) ([#1673])
- **fmt**, **chrono**: Removed the `chrono` feature flag and APIs for
using the [`chrono` crate] for timestamp formatting ([#1646])
### Added
- **fmt**, **time**: `LocalTime` and `UtcTime` types for formatting
timestamps using the [`time` crate] ([#1646])
- **fmt**: Added a lifetime parameter to the `MakeWriter` trait,
allowing it to return a borrowed writer. This enables implementations
of `MakeWriter` for types such as `Mutex<T: io::Write>` and
`std::fs::File`. ([#781])
- **env-filter**: Documentation improvements ([#1637])
- Support for some APIs on `#![no_std]` targets, by disabling the `std`
feature flag ([#1660])
Thanks to @Folyd and @nmathewson for contributing to this release!
[#1320]: https://github.com/tokio-rs/tracing/pull/1320
[#1673]: https://github.com/tokio-rs/tracing/pull/1673
[#1674]: https://github.com/tokio-rs/tracing/pull/1674
[#1646]: https://github.com/tokio-rs/tracing/pull/1646
[#1647]: https://github.com/tokio-rs/tracing/pull/1647
[#1648]: https://github.com/tokio-rs/tracing/pull/1648
[#1649]: https://github.com/tokio-rs/tracing/pull/1649
[#1660]: https://github.com/tokio-rs/tracing/pull/1660
[#1661]: https://github.com/tokio-rs/tracing/pull/1661
[#1431]: https://github.com/tokio-rs/tracing/pull/1431
[#1434]: https://github.com/tokio-rs/tracing/pull/1434
[#781]: https://github.com/tokio-rs/tracing/pull/781
[`chrono` crate]: https://crates.io/crates/chrono
[`time` crate]: https://crates.io/crates/time
[RUSTSEC-2020-0159]: https://rustsec.org/advisories/RUSTSEC-2020-0159.html
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This changes `tracing-subscriber` so that the `env-filter`, `json`,
and `chrono` features are not enabled by default, and instead require
users to opt in. This should significantly reduce the default
dependency footprint.
Of course, this is a breaking change, and therefore will be part of
`tracing-subscriber` v0.3.
Fixes#1258
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
## Motivation
Currently, `tracing-subscriber` supports the `chrono` crate for
timestamp formatting, via a default-on feature flag. When this code was
initially added to `tracing-subscriber`, the `time` crate did not have
support for the timestamp formatting options we needed.
Unfortunately, the `chrono` crate's maintainance status is now in
question (see #1598). Furthermore, `chrono` depends on version 0.1 of
the `time` crate, which contains a security vulnerability
(https://rustsec.org/advisories/RUSTSEC-2020-0071.html). This
vulnerability is fixed in more recent releases of `time`, but `chrono`
still uses v0.1.
## Solution
Fortunately, the `time` crate now has its own timestamp formatting
support.
This branch replaces the `ChronoLocal` and `ChronoUtc` timestamp
formatters with new `LocalTime` and `UtcTime` formatters. These
formatters use the `time` crate's formatting APIs rather than
`chrono`'s. This removes the vulnerable dependency on `time` 0.1
Additionally, the new `time` APIs are feature flagged as an _opt-in_
feature, rather than as an _opt-out_ feature. This should make it easier
to avoid accidentally depending on the `time` crate when more
sophisticated timestamp formatting is _not_ required.
In a follow-up branch, we could also add support for `humantime` as an
option for timestamp formatting.
Naturally, since this removes existing APIs, this is a breaking change,
and will thus require publishing `tracing-subscriber` 0.3. We'll want to
do some other breaking changes as well.
Fixes#1598.
I noticed a handful of places where `v0.1.x` refers to `Subscriber`s as
"collectors". This probably happened because we backported some commits
from master and forgot to change every instance of "collector" back to
"subscriber".
This commit fixes that. Whoops.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This adds a new `Span::or_current` method that returns the span it's
called on, if that span is enabled, or the current span if that span
is not enabled.
This should provide a more efficient alternative to writing code like
```rust
tokio::spawn(
future
.instrument(some_span)
.in_current_span()
);
```
since it doesn't wrap the future in two different spans which are
(potentially) both entered on polls. It's also a little more concise
to write, which (hopefully) will encourage people to use it instead
of the less efficient alternative.
`Span::or_current` may be useful in other cases as well.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This commit updates the `inferno-flame` example to use the `tempfile`
crate as a replacement
for the unmaintained `tempdir` crate.
Also, the previous version of the example output the flamegraph inside
the temporary directory. Since the temporary directory is cleaned up
when the program exits, this means the user can't actually look at the
flamegraph when running this example. I've changed the example to put the
flamegraph in the current working dir instead, so the user can look at
it.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
This version includes patches for a couple of RUSTSEC advisories that
`cargo audit` is mad about. These aren't actually security-critical,
since the affected hyper versions are only used in examples, not in
actual`tracing` crates, but bumping makes `cargo audit` chill out. And
we should keep up to date regardless.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Currently, there are a few examples that include CLI argument parsing.
These currently use the `clap` crate. `clap` is one of the most flexible
and full-featured argument parsing libraries in Rust. However, or
perhaps *because* of this, `clap` is also a fairly heavy-weight
dependency which pulls in a lot of transitive deps.
We don't *need* most of `clap`'s features for the very simple argument
parsing in these examples. Therefore, this commit replaces `clap` with
`argh`, which is a much lighter dependency.
Also, `clap` is currently pulling a version of `bitflags` that breaks
our MSRV...
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Not sure that there's a issue tracking this, but anyways. Here's an
update to Tokio 1.0!
This closes the various Dependabot dependency update PRs that we can't
just merge due to breaking changes.
Closes#1149Closes#1164Closes#1177
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
## Motivation
Fixes#1347
## Solution
Change the format from `err=message, err: source1` to `err=message
err.sources=[source1, source2, source3, ...]`, as suggested in
https://github.com/tokio-rs/tracing/issues/1347#issuecomment-813674313
(still leaving out the sources if there are none).
## Caveats
Haven't changed the JSON formatter, since I'm not really sure about how
to do that. The current format is `{.., "fields": {.., "err":
"message"}}`, disregarding the sources entirely.
We could change that to `{.., "fields": {.., "err": "message",
"err.sources": ["source1", "source2", "source3", ..]}}`, which would
keep backwards compatibility but looks pretty ugly.
Another option would be `{.., "fields": {.., "err": {"message":
"message", "sources": ["source1", "source2", "source3", ..]}}}` which
leaves room for future expansion.
Then again, that begs the question about why the first error is special,
so maybe it ought to be `{.., "fields": {.., "err": {"message":
"message", "source": {"message": "source1", "source": ..}}}}`.
But that style of linked list is pretty annoying to parse, so maybe it
ought to be flattened to `{.., "fields": {.., "err": [{"message":
"message"}, {"message": "source1"}, ..]}}`?
Co-authored-by: Eliza Weisman <eliza@buoyant.io>
