What is the proper way to use examples with model-based testing?

[Dannnno]

Normally when fast-check identifies an error in my code, I add the specific failing example to an examples array so that known failure case always gets covered.

I'm trying to do the same with some model-based testing, but I end up with type errors because it expects cmds to be Iterable<fc.Command<Model, Real>>. The output of my failing test doesn't provide a value for this property, just any other properties I gave. For example:

fc.assert(
  fc.property(
    fc.nat(),
    fc.integer(),
    fc.commands(MyCommandList),
    (v1, v2, cmds) => {
      const s = () => { model: whatever(v1, v2), real: whatever(v1, v2) };
      fc.modelRun(s, cmds);
    }
  )
)

Might give me an output like:

Error: Property failed after 1 tests
{ seed: 335168591, path: "0:0", endOnFailure: true }
Counterexample: [0,5, /*replayPath=":"*/]
Shrunk 1 time(s)

I know that I need to use the replayPath to re-run this using the seed, but how do I do it as one of several discrete examples? Should I just intercept it with the debugger and construct the array myself?

[gruhn]

Not sure how to do that "properly". But a simple alternative could be to put the test into a function, which takes seed, path, replayPath as arguments:

function myModelTest(seedEtc, replayPath) {
  fc.assert(
    fc.property(
      fc.nat(),
      fc.integer(),
      fc.commands(MyCommandList, { replayPath }),
      (v1, v2, cmds) => {
        const s = () => { model: whatever(v1, v2), real: whatever(v1, v2) };
        fc.modelRun(s, cmds);
      }
    ),
    seedEtc,
  )
}

And then create test for a list of example seed/replayPath combinations:

test('randomized', () => {
  myModelTest(undefined, undefined)
})

const examples = [
  [{ seed: 335168591, path: "0:0", replayPath: ":" }],
  // ...
]

// foreach-loop to create test for each example:
for (const { seed, path, replayPath } of examples) {
  test('example', () => {
    myModelTest({ seed, path, endOnFailure: true }, replayPath)
  })
}

// or with `test.each` if you are using Jest/Vitest/...
test.each(examples)('example', ({ seed, path, replayPath }) => {
  myModelTest({ seed, path, endOnFailure: true }, replayPath)
})

A downside is that the saved examples can become un-replayable if underlying arbitraries are changed.