feat(studio): Remove code agent summary and improve agent evals by htolentino-nvidia · Pull Request #621 · NVIDIA-NeMo/nemo-platform

htolentino-nvidia · 2026-07-09T15:27:06Z

Summary by CodeRabbit

New Features
- Updated dataset/file selection examples to include .jsonl support alongside .csv and .parquet.
Bug Fixes
- Improved upload flow terminology to “Fileset” and broadened the fileset picker results by removing an unnecessary purpose restriction.
- Refined assistant dataset-selection and evaluation guidance, including interactive dataset choice and smoother handling for generated-column (gen_*) datasets.
- Updated Claude Code chat message shaping to better preserve completed tool activity in the conversation.

coderabbitai · 2026-07-09T15:31:04Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e12d470c-b1ec-4f5a-ad40-bb199470a4c3

📥 Commits

Reviewing files that changed from the base of the PR and between fa57bd4 and 9309172.

📒 Files selected for processing (9)

services/studio/src/nmp/studio/coding_agent_mcp_tools.py
services/studio/src/nmp/studio/coding_agents.py
services/studio/tests/unit/test_coding_agents.py
web/packages/common/src/components/UploadModal/DatasetUploader/Select.test.tsx
web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx
web/packages/studio/src/components/evaluation/Configurations/form/InputFile.test.tsx
web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/toolParts.ts
web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/useCustomAssistantChatRuntime.test.ts
web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/util.test.ts

✅ Files skipped from review due to trivial changes (1)

web/packages/studio/src/components/evaluation/Configurations/form/InputFile.test.tsx

🚧 Files skipped from review as they are similar to previous changes (8)

services/studio/src/nmp/studio/coding_agent_mcp_tools.py
web/packages/common/src/components/UploadModal/DatasetUploader/Select.test.tsx
web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/util.test.ts
web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/useCustomAssistantChatRuntime.test.ts
web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx
services/studio/tests/unit/test_coding_agents.py
web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/toolParts.ts
services/studio/src/nmp/studio/coding_agents.py

📝 Walkthrough

Walkthrough

Studio agent prompt instructions now target NAT agent evaluations, Claude Code fallback transcript shaping no longer synthesizes collapsed-thinking parts, and the upload modal fileset picker drops its dataset filter while renaming labels to Fileset.

Changes

Studio agent prompt and transcript shaping

Layer / File(s)	Summary
Prompt and tool contract `services/studio/src/nmp/studio/coding_agents.py`, `services/studio/src/nmp/studio/coding_agent_mcp_tools.py`, `services/studio/tests/unit/test_coding_agents.py`	System prompt evaluation instructions replace message-summary requirements; dataset-file tool example adds `.jsonl`; prompt test now asserts interactive-input contract strings.
Claude Code message-part fallback `web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/toolParts.ts`, `web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/useCustomAssistantChatRuntime.test.ts`, `web/packages/studio/src/routes/agents/ClaudeCodeChatRoute/util.test.ts`	Collapsed-thinking helper code is removed, fallback completion now returns original parts unchanged when no studio summary block is present, and runtime/transcript tests update expected assistant text accordingly.

Fileset upload modal labels

Layer / File(s)	Summary
Fileset listing and labels `web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx`, `web/packages/common/src/components/UploadModal/DatasetUploader/Select.test.tsx`, `web/packages/studio/src/components/evaluation/Configurations/form/InputFile.test.tsx`	`useFilesListFilesets` no longer receives a dataset purpose filter; slot label, section heading, and placeholder text now use Fileset wording, and the tests assert the updated label and absent filter.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: removing code agent summary behavior and updating agent evaluation flow.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch studio-remove-code-agent-summary/htolentino

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx (2)

120-135: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Incomplete rename: "Create Dataset"/"New Dataset" left unchanged next to "Existing Filesets".

Same dropdown now mixes "Fileset" (heading/placeholder) and "Dataset" (create-section heading/option) terminology, confusing users.

✏️ Suggested fix

                   {
-                    slotHeading: 'Create Dataset',
+                    slotHeading: 'Create Fileset',
                     attributes: {
                       MenuHeading: { className: 'hidden', 'aria-hidden': true },
                     },
                     items: [
                       {
-                        children: 'New Dataset',
+                        children: 'New Fileset',
                         value: 'new',
                       },
                     ],
                   },

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx`
around lines 120 - 135, The dropdown in Select.tsx still uses mixed “Dataset”
wording alongside “Fileset” labels, which makes the renamed UI inconsistent.
Update the allowNewDataset branch in the Select component so the section heading
and option text match the existing fileset terminology, and ensure any related
labels/aria text in this menu use the same term consistently.

38-46: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Restore the dataset-purpose filter Removing filter: { purpose: 'dataset' } broadens this selector to every fileset and breaks Select.test.tsx, which still expects dataset-only queries.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx`
around lines 38 - 46, The fileset query in Select should be constrained back to
datasets only, since the current useFilesListFilesets call is now fetching every
fileset and breaking the dataset selector behavior. Restore the missing purpose
filter in the options passed from Select.tsx so the query only returns filesets
with purpose set to dataset, keeping the existing page_size and sort settings
intact.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx`:
- Around line 120-135: The dropdown in Select.tsx still uses mixed “Dataset”
wording alongside “Fileset” labels, which makes the renamed UI inconsistent.
Update the allowNewDataset branch in the Select component so the section heading
and option text match the existing fileset terminology, and ensure any related
labels/aria text in this menu use the same term consistently.
- Around line 38-46: The fileset query in Select should be constrained back to
datasets only, since the current useFilesListFilesets call is now fetching every
fileset and breaking the dataset selector behavior. Restore the missing purpose
filter in the options passed from Select.tsx so the query only returns filesets
with purpose set to dataset, keeping the existing page_size and sort settings
intact.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 266b05a4-4fe9-4be0-81b6-6d527e25544e

📥 Commits

Reviewing files that changed from the base of the PR and between 84a3896 and 81b9d21.

📒 Files selected for processing (4)

services/studio/src/nmp/studio/coding_agent_mcp_tools.py
services/studio/src/nmp/studio/coding_agents.py
services/studio/tests/unit/test_coding_agents.py
web/packages/common/src/components/UploadModal/DatasetUploader/Select.tsx

github-actions · 2026-07-09T15:39:18Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	23427/30608	76.5%	61.3%
Integration Tests	13691/29288	46.8%	19.8%

Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>

htolentino-nvidia requested review from a team as code owners July 9, 2026 15:27

github-actions Bot added the feat label Jul 9, 2026

coderabbitai Bot reviewed Jul 9, 2026

View reviewed changes

htolentino-nvidia added 4 commits July 9, 2026 12:40

Remove studio agent summary

2a64184

Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>

Fix file selection for Evals

95c2d1f

Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>

Remove more collapsible content for now

4352fce

Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>

Fix lint and format

9309172

Signed-off-by: Henrique Tolentino <htolentino@nvidia.com>

htolentino-nvidia force-pushed the studio-remove-code-agent-summary/htolentino branch from fa57bd4 to 9309172 Compare July 9, 2026 16:40

Merge branch 'main' into studio-remove-code-agent-summary/htolentino

c1df155

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(studio): Remove code agent summary and improve agent evals#621

feat(studio): Remove code agent summary and improve agent evals#621
htolentino-nvidia wants to merge 5 commits into
mainfrom
studio-remove-code-agent-summary/htolentino

htolentino-nvidia commented Jul 9, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

github-actions Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

htolentino-nvidia commented Jul 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

htolentino-nvidia commented Jul 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 9, 2026 •

edited

Loading