Test Generation in Modular #3396

MaryGao · 2025-07-27T13:31:51Z

- Added modular test generation similar to samples - Test names use sample description logic for consistency - Clean test names without trailing dots - Proper vitest/mocha framework support based on module type - Azure Identity integration for ARM clients - Generated test utilities including recordedClient.ts - Support for both single and multiple client scenarios - Unique test names including function names for multiple examples Generated 7 operation test files with professional naming: - assign role to the data product for dataProductsAddUserRoleMaximumSetGen - create data product resource for dataProductsCreateMaximumSetGen - delete data product resource for dataProductsDeleteMaximumSetGen - generate sas token for storage account for dataProductsGenerateStorageAccountSasTokenMaximumSetGen - list data products by resource group for dataProductsListByResourceGroupMaximumSetGenGeneratedByMinimumSetRuleMinimumSetGen - initiate key rotation on Data Product for dataProductsRotateKeyMaximumSetGen - update data product resource for dataProductsUpdateMaximumSetGen

Enhanced test generation to include response assertions that validate response values against expected values from examples: - Added generateResponseAssertions() function - Added generateAssertionsForValue() for recursive validation - Support for string, number, boolean, array, model, dict, null, union types - Proper handling of TypeSpec SDK numeric response indices - Validates key response properties with strictEqual checks - Array length and item validation - Model/object property validation up to 3 levels deep Generated tests now include comprehensive response validation like: assert.strictEqual(result.properties.resourceGuid, expected_value) assert.strictEqual(result.properties.provisioningState, expected_value) assert.ok(Array.isArray(result.dataTypeScope)) This ensures tests validate both API success and response content.

- Added 4 passing unit test scenarios for test generation: - basicOperationTest: Tests simple GET operation with response assertions - pagingOperationTest: Tests paging operations with array response validation - moduleTypeTest: Tests different import styles for ESM vs CommonJS - complexResponseTest: Tests nested object response assertions - Added emitTestsFromTypeSpec function to test/util/emitUtil.ts - Enhanced scenarios.spec.ts with 'tests' output block type - Improved test output formatting with proper file separation - All test scenarios validate comprehensive response assertions - Tests follow same pattern as existing samples scenarios

- Fix formatting issues in emitTests.ts (whitespace and comment styles) - Update scenarios.spec.ts with better test file handling and formatting fixes - Refresh test scenario files with updated expected outputs: - basicOperationTest.md: Updated with improved test structure - complexResponseTest.md: Updated response assertion patterns - moduleTypeTest.md: Updated with proper module import handling - pagingOperationTest.md: Updated paging operation test patterns - voidOperationTest.md: Updated void operation test handling All tests passing with SCENARIOS_UPDATE=true refresh

### LRO (Long Running Operations) Support: - Add detection for LRO operations (method.kind === 'lro' || method.kind === 'lropaging') - Generate proper poller pattern: 'const poller = await client.method()' followed by 'const result = await poller.pollUntilDone()' - Add comprehensive LRO test scenario (lroOperationTest.md) with ArmResourceCreateOrReplaceAsync example - Include proper response assertions on the final LRO result ### Paging Test Improvements: - Create dedicated generatePagingResponseAssertions() function - Fix incorrect paging assertions that were checking resArray[0].value instead of resArray directly - Now correctly asserts on collected results length and individual item properties - Updated pagingOperationTest.md with proper assertions ### Test Coverage Enhancement: - Fixed void operation test (voidOperationTest.md) to properly handle LRO delete operations - All 6 test scenarios now passing: basic, complex, module, paging, void, and LRO operations ### Technical Improvements: - Better separation of concerns between regular, paging, and LRO test generation - Proper understanding of how paging iteration works vs response structure - Correct LRO poller pattern following Azure SDK conventions

…torest.typescript into generate-test-modular

- Created shared utility module sampleTestHelpers.ts to reduce code duplication - Extracted common functions: buildParameterValueMap, prepareCommonValue, getCredentialSampleValue, getCredentialTestValue, serializeExampleValue - Unified CommonValue interface for both samples and tests - Refactored emitSamples.ts and emitTests.ts to use shared utilities - Eliminated ~200+ lines of duplicate code while maintaining functionality - All 388 modular unit tests continue to pass

…tilities - Created comprehensive sampleTestHelpers.ts module with shared utilities - Extracted common functions: prepareCommonParameters, iterateClientsAndMethods, generateMethodCall, createSourceFile, generateAssertionsForValue, generateResponseAssertions - Reduced code duplication by ~200+ lines across emit files - Enhanced test vs sample differentiation (env variables for tests, example values for samples) - Improved maintainability with unified parameter preparation and assertion generation - All 388 modular unit tests passing after refactoring Breaking down the previous monolithic emit functions into: 1. Common parameter handling logic shared between samples and tests 2. Unified client/method iteration patterns 3. Shared assertion generation for response validation 4. Consistent source file creation and import handling 5. Better separation of concerns between sample and test generation This refactoring eliminates significant code duplication while maintaining 100% test compatibility.

…nto pr/3396

Copilot

Pull Request Overview

This PR introduces experimental test generation functionality for the TypeSpec TypeScript emitter. It adds the ability to automatically generate test files from TypeSpec examples, similar to the existing sample generation feature, with proper test recorder integration and response assertions.

Key Changes

Added new test generation infrastructure with emitTests.ts and shared helper utilities in exampleValueHelpers.ts
Introduced static test helper for recorder client setup (recordedClient.ts)
Added extensive unit test scenarios covering parameters, responses, and various operation types
Updated the binder to handle test-specific helper files

Reviewed Changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/typespec-ts/src/modular/emitTests.ts	New file implementing test generation logic with vitest integration
packages/typespec-ts/src/modular/helpers/exampleValueHelpers.ts	New shared utilities for both samples and tests, including parameter processing and assertion generation
packages/typespec-ts/src/modular/emitSamples.ts	Refactored to use shared helpers from exampleValueHelpers.ts
packages/typespec-ts/src/modular/external-dependencies.ts	Added AzureTestDependencies for vitest and test-recorder imports
packages/typespec-ts/src/modular/static-helpers-metadata.ts	Added CreateRecorderHelpers metadata
packages/typespec-ts/static/test-helpers/recordedClient.ts	New static helper for test recorder setup
packages/typespec-ts/src/framework/load-static-helpers.ts	Enhanced to load test helpers from test-helpers directory
packages/typespec-ts/src/framework/hooks/binder.ts	Updated to clean unused helper files in test directories
packages/typespec-ts/src/lib.ts	Added experimental-generate-test-files option to schema
packages/typespec-ts/src/index.ts	Integrated test generation into emit pipeline
packages/typespec-ts/test/util/testUtil.ts	Added test dependencies to test utilities
packages/typespec-ts/test/util/emitUtil.ts	Added emitTestsFromTypeSpec helper function
packages/typespec-ts/test/modularUnit/scenarios.spec.ts	Added test generation support to scenario testing
packages/typespec-ts/test/modularUnit/scenarios/test/*/.md	New test scenario files covering various parameter and operation types
packages/typespec-ts/test-next/integration/load-static-files.test.ts	Updated expected file count for additional helper file

Comments suppressed due to low confidence (2)

packages/typespec-ts/src/framework/load-static-helpers.ts:1

Typo in comment: 'normalizae' should be 'normalize' in the comment on line 324.

import { readdir, stat, readFile } from "fs/promises";

packages/typespec-ts/src/framework/load-static-helpers.ts:1

Typo in comment: 'normalizae' should be 'normalize'.

import { readdir, stat, readFile } from "fs/promises";

Copilot · 2025-10-29T08:25:09Z

packages/typespec-ts/src/modular/helpers/exampleValueHelpers.ts

+      return {
+        ...defaultSetting,
+        value: `{ getToken: async () => {
+          return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: now() }; } }`


The function now() is undefined. Should be Date.now() for the timestamp.

Suggested change

return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: now() }; } }`

return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: Date.now() }; } }`

Copilot · 2025-10-29T08:25:09Z

packages/typespec-ts/src/modular/helpers/exampleValueHelpers.ts

+        value: `{
+        getToken: async () => {
+            return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: Date.now() };
+        }
+    } `


[nitpick] Inconsistent formatting: this credential value has extra spaces and different indentation compared to the sample version on line 120-122. Consider using the same formatting for both.

Suggested change

value: `{

getToken: async () => {

return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: Date.now() };

}

} `

value: `{ getToken: async () => ({ token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: Date.now() }) }`

Copilot · 2025-10-29T08:25:10Z

packages/typespec-ts/src/modular/emitTests.ts

+
+  ${beforeEachType}(async function(ctx) {
+    recorder = await ${createRecorderHelper}(ctx);
+    ${clientParameterDefs.join("\n")}


The clientParameterDefs array elements are not being indented when joined. This will cause the generated code to have inconsistent indentation. Consider adding proper indentation: ${clientParameterDefs.join('\\n ')}

Suggested change

${clientParameterDefs.join("\n")}

${clientParameterDefs.join("\n ")}

Copilot · 2025-10-29T08:25:10Z

packages/typespec-ts/src/modular/helpers/exampleValueHelpers.ts

+  const assertions: string[] = [];
+
+  // Prevent infinite recursion for deeply nested objects
+  if (currentDepth >= maxDepth) {
+    return assertions;
+  }


The function generateAssertionsForValue has complex recursive logic with a maxDepth parameter, but lacks a JSDoc comment explaining the purpose of this parameter, the recursion depth limit, and when it would be hit. Consider adding comprehensive documentation.

MaryGao · 2025-10-29T08:43:23Z

@jiaodi Could you help resolve the conflicts?

…torest.typescript into pr/3396

Mary Gao and others added 8 commits July 27, 2025 16:58

Add more unit test cases

037d338

Fix the lro issue

93ca81f

Fix the issues in parameter test generation

c4b8cb7

MaryGao changed the title ~~Generate test modular~~ Test generation in Modular Jul 30, 2025

MaryGao and others added 18 commits July 31, 2025 16:52

Merge branch 'main' into generate-test-modular

7c216db

Merge branch 'main' into generate-test-modular

f4b4f34

Remove the useless generated codes

d7bb665

Regenerate the test cases for UTs

d0dc4bd

Fix the client-level parameter missing issues

caee79c

Fix the parameter without normalization issue

19d22ab

Fix the comma

054cbb4

Merge branch 'main' into generate-test-modular

123361e

Fix the bytes and additional properties issues in sample generation

049750a

Merge branch 'generate-test-modular' of https://github.com/marygao/au…

270dbbe

…torest.typescript into generate-test-modular

Fix the additional issues in testings

8752cc4

Format codes

ad54af1

Fix the response types in bytes

0405fd4

Merge branch 'main' into generate-test-modular

6983f01

Merge branch 'main' into generate-test-modular

636419c

Format code

4e36c0b

MaryGao changed the title ~~Test generation in Modular~~ Test Generation in Modular Aug 28, 2025

MaryGao and others added 2 commits August 28, 2025 13:51

Merge branch 'main' into generate-test-modular

b3a441c

Fix the lint issues

de93d3f

Mary Gao added 2 commits August 28, 2025 16:17

Revert the change in generated samples

f5b1b2a

Revert change in smoke testing

7ce7d01

MaryGao marked this pull request as ready for review September 8, 2025 05:12

MaryGao requested review from joheredi and qiaozha as code owners September 8, 2025 05:12

MaryGao and others added 6 commits September 8, 2025 14:59

Rename the interface a little and improve the import with binder

cdb25a7

Fix the import issues and regenerate in the UTs

5d0da7a

Refactor the recorder create helper

b107897

Fix the UTs

297835f

Fix the un-used files removed issue

0a31b02

Update the UTs

c486f60

qiaozha added the p0 priority 0 label Sep 10, 2025

v-jiaodi and others added 7 commits September 16, 2025 12:37

add ut for sample parameter order

eb9aebe

update

358c652

Merge branch 'main' into generate-test-modular

dbde475

Merge branch 'main' of https://github.com/Azure/autorest.typescript i…

0b31212

…nto pr/3396

update ut

64f73ac

Merge branch 'main' into pr/3396

c70d107

Merge branch 'main' into generate-test-modular

1ba08d6

lirenhe requested a review from Copilot October 29, 2025 08:19

Copilot AI reviewed Oct 29, 2025

View reviewed changes

kazrael2119 added 7 commits October 30, 2025 10:14

Merge branch 'main' into pr/3396

2203300

merge main

fb36e1c

Merge branch 'generate-test-modular' of https://github.com/MaryGao/au…

6cf79c6

…torest.typescript into pr/3396

remove testing smoke

428ca4f

Update load-static-helpers.ts

8eb84e4

update

711af79

update ut

a82017d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Test Generation in Modular #3396

Test Generation in Modular #3396

Uh oh!

MaryGao commented Jul 27, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

Copilot AI Oct 29, 2025

Uh oh!

MaryGao commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: now() }; } }`
	return { token: "INPUT_YOUR_TOKEN_HERE", expiresOnTimestamp: Date.now() }; } }`

	${clientParameterDefs.join("\n")}
	${clientParameterDefs.join("\n ")}

Test Generation in Modular #3396

Are you sure you want to change the base?

Test Generation in Modular #3396

Uh oh!

Conversation

MaryGao commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

MaryGao commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MaryGao commented Jul 27, 2025 •

edited

Loading