1.99 beta crater regression: Various compile errors in tests of `typestate-pipeline`

[theemathas]

This regression was found by the 1.99 crater run.

There are many compile errors of various kinds that appeared while trying to compile the tests for the typestate-pipeline crate. See the full logs in the output below.

https://crater-reports.s3.amazonaws.com/beta-1.99-5/1.99.0-beta.1/gh/JohnSwan1503.typestate-pipeline/log.txt

[jnkel]

The crate does...this...for some reason:

/// The integration test catalog — every test file rendered as a
/// browsable page so you can see exactly what behavior is locked in.
///
/// Tests are grouped by domain. UI tests pair the trybuild source with
/// the pinned `.stderr` snapshot so you can see the exact diagnostic
/// users get for each kind of misuse.
///
/// Only present in rustdoc builds — `tests` is not an importable path
/// in downstream crates.
#[cfg(any(doc, docsrs))]
pub mod tests {
    /// `#[derive(TypestateFactory)]` feature coverage.
    pub mod factory {
        /// Construction & getters, Drop semantics, removable/overridable,
        /// defaults & conditional finalize, custom names, and sync
        /// transformers (infallible + fallible).
        ///
        #[doc = include_str!("../tests/factory/core/docs/tests.md")]
        ///
        /// ## Test Cases
        ///
        #[doc = include_str!("../tests/factory/core/docs/build_in_order.md")]
        ///
        /// ```rust
        #[doc = include_str!("../tests/factory/core/tests/build_in_order.rs")]
        /// ```
        ///
        #[doc = include_str!("../tests/factory/core/docs/build_in_arbitrary_order.md")]
        ///
        /// ```rust
        #[doc = include_str!("../tests/factory/core/tests/build_in_arbitrary_order.rs")]
        /// ```
        ///
        #[doc = include_str!("../tests/factory/core/docs/default_helper_fills_in_optional.md")]

        .........

...this goes on for over a thousand lines.

The failure occur trying to build these as doctests, and they're obviously they're not well-formed doctests (they often refer to items defined in the parent test module):

use super::{BadInput, UserProfileFactory};

pub async fn main() {
    let bag = UserProfileFactory::new().name("Bob".to_owned()).await;
    let result = bag.email(String::new()).await;
    match result {
        Err(BadInput::Empty) => {}
        Ok(_) => panic!("expected BadInput::Empty"),
    }
}

So I don't know why this should have worked before, unless something was causing those doctests not to run.

[cyrgani]

I suspect that #159014 might've caused these tests to no longer be ignored?

[jnkel]

Yes, I just bisected it to that PR.

[jnkel]

"Regressed" in #159014 (cc @GuillaumeGomez), though I think the old behavior was the bug and the new behavior is correct:

Minimized:

#[cfg(any(doc, false))]
pub mod tests {
    /// ```rust
    /// asdfghjkl
    /// ```
    pub mod foo {}
}

Output (before):

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
   Doc-tests foo

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Output (after):

    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.00s
   Doc-tests foo

running 1 test
test src/lib.rs - tests::foo (line 3) ... FAILED

failures:

---- src/lib.rs - tests::foo (line 3) stdout ----
error[E0425]: cannot find value `asdfghjkl` in this scope
 --> src/lib.rs:4:1
  |
4 | asdfghjkl
  | ^^^^^^^^^ not found in this scope

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0425`.
Couldn't compile the test.

failures:
    src/lib.rs - tests::foo (line 3)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s

all doctests ran in 0.11s; merged doctests compilation took 0.07s
error: doctest failed, to rerun pass `--doc`

Prior to the linked PR, rustdoc would not run a doctest behind cfg(any(doc, false)); however, it would run a test behind cfg(doc). The new behavior seems to fix this incosistency, thus breaking this crate which had invalid doctests behind cfg(any(doc, docsrs)).

[workingjubilee]

Seems correct to consider this acceptable breakage then?

[GuillaumeGomez]

I'd say yes as the previous behaviour was indeed buggy.