[Broken task] Navidrome 97434c: requirements mandate nil transcoding but TestCore expects trc.ID

[J3m5]

Note

This issue was drafted by the AI model currently assisting me, reviewed by me before submission, and created at my request.

Summary

The public SWE-bench Pro instance instance_navidrome__navidrome-97434c1789a6444b30aae5ff5aa124a96a88f504 contains a contradiction between its visible requirements, gold patch, and official TestCore test.

An agent that follows the explicit requirement to return a nil transcoding value will fail the official test. Conversely, the gold patch passes by violating that visible requirement.

Affected instance

• Instance ID: instance_navidrome__navidrome-97434c1789a6444b30aae5ff5aa124a96a88f504
• Repository: navidrome/navidrome
• Base commit: f8ee6db72a015f33b41879d1ba253eeff8884bfd
• Gold commit: 97434c1789a6444b30aae5ff5aa124a96a88f504
• Required test: TestCore
• Dataset row: ScaleAI/SWE-bench_Pro, test row 87
• Evaluation metadata: instance_info.txt

Conflicting contracts

Source	Required behavior
Visible requirements field	Register must return a nil transcoding value
Gold patch	Loads and returns the configured transcoding
Official TestCore	Requires trc.ID to equal "1"

Visible requirement

The official dataset's requirements field states:

Register must return a nil transcoding value.

Gold patch behavior

The gold patch retains the existing transcoding lookup:

if plr.TranscodingId != "" {
    trc, err = p.ds.Transcoding(ctx).Get(plr.TranscodingId)
}
return plr, trc, err

It therefore does not satisfy the visible nil requirement when a player has a configured transcoding.

Official test behavior

The dataset's before_repo_set_cmd restores the complete test file from the gold commit:

git checkout 97434c1789a6444b30aae5ff5aa124a96a88f504 -- core/players_test.go

That test file still contains:

It("finds player by ID and return its transcoding", func() {
    plr := &model.Player{
        ID:            "123",
        Name:          "A Player",
        Client:        "client",
        LastSeen:      time.Time{},
        TranscodingId: "1",
    }

    repo.add(plr)
    p, trc, err := players.Register(
        ctx,
        "123",
        "client",
        "chrome",
        "1.2.3.4",
    )

    Expect(err).ToNot(HaveOccurred())
    Expect(p.ID).To(Equal("123"))
    Expect(trc.ID).To(Equal("1"))
})

Returning nil, as explicitly required by the prompt, necessarily makes this test fail.

Observed behavior

This was observed during an evaluation where the agent implemented the visible requirements, including the unconditional nil transcoding return.

The implementation and its local test suite completed successfully, but the official evaluator restored core players_test.go from the gold commit and reported:

Required tests failed or were skipped:
- TestCore: FAILED

The evaluation environment was healthy:

• agent exit code: 0
• test exit code: 1
• infrastructure failures: 0
• required test present and executed: TestCore

The failure is therefore not caused by a missing tool, Docker setup, or evaluation infrastructure problem.

Impact

This instance can produce a false comparison between agents:

• an agent that follows the explicit nil requirement is marked incorrect;
• an agent that ignores that requirement and follows the hidden test may pass;
• the gold patch passes while contradicting the visible contract.

This makes the instance unsuitable for comparing instruction-following or agent performance in its current form.

Suggested fix

The likely correction is to remove this requirement from the dataset:

Register must return a nil transcoding value.

That would align the visible task with both the gold patch and the official test.

If returning nil was actually intended, then the gold patch and core/players_test.go must instead be updated to enforce that behavior.

In either case, the visible requirements, gold patch, and verifier should express the same contract.

Provenance check

The issue was initially observed through a translated Harbor/Margin suite.
The official Hugging Face row was then inspected directly and contains the same nil requirement, gold patch, before_repo_set_cmd, and TestCore selection.