#51 Commit Submission in worker thread by leoraba · Pull Request #195 · overture-stack/lyric

leoraba · 2026-03-03T22:05:41Z

Summary

This PR improves the performance of the submission commit process.

Issues

Refactor Validation process using Worker threads #51

Description of Changes

Database changes

Added indices to tables submission, submitted_data and audit_submitted_data to improve data lookup
Added new Submission status COMMITTING, to indicate a submission is being processed from status VALID to COMMITTED

Worker threads

Added new dependency workerpool to manage a pool of workers.
Implemented an execution of worker thread to commit a submission. (CPU intensive process running out of the main thread) to keep the main thread available.

leoraba · 2026-03-10T20:31:43Z

packages/data-provider/src/services/submission/submissionProcessor.ts

+				const invalidRecordErrors = findInvalidRecordErrorsBySchemaName(resultValidation, entityName);
+				const hasErrorByIndex = groupErrorsByIndex(invalidRecordErrors);


Move calling these 2 functions out of the for loop to improve the performance.
During local testing, this change improve 15x the performance during commit operation.

leoraba · 2026-03-10T20:38:23Z

packages/data-provider/src/services/submission/submissionProcessor.ts

+			return {
+				submissionId: submission.id,
+				organization: submission.organization,
+				categoryId: submission.dictionaryCategory.id,
+				data: resultCommit,
+			};


Removed the execution of params.onFinishCommit from here, since this now runs on a worker thread and it doesn't have access to this function, instead this function just returns the data object

leoraba · 2026-03-10T20:39:49Z

packages/data-provider/src/services/submission/submissionService.ts

-			categoryId,
-			submission?.organization,
-		);
+		await submissionRepository.update(submissionId, { status: SUBMISSION_STATUS.COMMITTING, updatedBy: username });


submission is updated to COMMITTING status when the commit operation is called

leoraba · 2026-03-10T20:46:18Z

packages/data-provider/src/utils/submittedDataUtils.ts

refactor: updated this function to handle large arrays of submitted data more efficiently by replacing the spread operator, which previously created new arrays during each iteration and was using memory inefficiently.

leoraba · 2026-03-10T20:51:12Z

packages/data-provider/src/workers/types.ts

+export interface WorkerFunctions {
+	commitSubmission(input: CommitWorkerInput): Promise<void>;
+}


functions exposed are ran on a worker thread pool, more functions can be added later.

joneubank

Really great foundation for our worker setup. Love that this runs successfully. Most of my comments are for dealing with configuration and setup issue edge cases. There's a few changes to look at for server stability and completeness.

packages/data-provider/src/services/submission/submissionProcessor.ts

packages/data-provider/src/services/submission/submissionService.ts

packages/data-provider/src/utils/submittedDataUtils.ts

packages/data-provider/src/workers/workerContext.ts

packages/data-provider/src/workers/workerpool.ts

packages/data-provider/src/workers/workerPoolManager.ts

packages/data-provider/src/workers/commitSubmissionWorker.ts

packages/data-provider/src/workers/types.ts

leoraba · 2026-03-18T13:00:36Z

packages/data-provider/src/workers/commitSubmissionWorker.ts

+		// Reset the submission status back to VALID so it can be retried
+		await submissionRepo.update(submissionId, { status: SUBMISSION_STATUS.VALID, updatedBy: username });
+		throw error;


When an error occurs during committing a submission, it is back to VALID status.

leoraba · 2026-03-18T13:11:01Z

packages/data-provider/src/workers/workerPoolManager.ts

+ * @param configData The application configuration
+ * @returns The worker functions to execute tasks in the worker pool
+ */
+export const createWorkerPool = (configData: AppConfig): WorkerFunctions => {


This is the main function used to initialize the worker pool, it is used when the Lyric provider is created, returns the functions available in the baseDependency in the provider.

If any error occurs during worker pool initialization an error is thrown immediately that stops the application

joneubank

Amazing. Almost there. We added the shutdown mechanism, but now we need to use it in our sever and our docs.

joneubank · 2026-03-18T18:48:35Z

packages/data-provider/src/workers/types.ts

great documentation updates, will be very helpful. much appreciated.

joneubank · 2026-03-18T18:56:49Z

packages/data-provider/src/workers/workerPoolManager.ts

+		terminate: async (): Promise<void> => {
+			await pool.terminate();
+			logger.info(LOG_MODULE, 'Worker pool terminated');
+		},


We should make sure this is used in our lyric server code. We should also update the documentation in the lyric provider README so that we show how to perform clean up properly.

added in the Server a call to this function to gracefully terminate the workers.

Docs have been updated to explain the usage.

joneubank

Couple more comments on error handling. No requests for change, use your best judgement of whats necessary.

joneubank · 2026-03-18T19:17:40Z

packages/data-provider/src/services/submission/submissionService.ts

+			.then(() => logger.info(LOG_MODULE, `Worker pool execution succeeded for submission ${submissionId}`))
+			.catch((error) => {
+				logger.error(LOG_MODULE, `Worker pool execution failed for submission ${submissionId}: ${error}`);
+				throw new InternalServerError(`Commit submission failed: ${error}`);


How are errors thrown from the worker pool functions handled? Is there a built in handler for errors thrown from workers? At a glance it seems like this won't be caught or handled by anything (may even crash the server).

This throw indeed is crashing the server, will remove the then and catch from here as it's already handled by the worker when if fails or succeds.

Adittionally, The worker should not trhow any error.

joneubank · 2026-03-18T19:18:39Z

packages/data-provider/src/services/submission/submissionProcessor.ts

 				message,
 			);
 			logger.error(error);
+			throw new Error(`${message}`, { cause: error });


Any reason to throw a generic Error instead of just throwing the original error?

Suggested change

throw new Error(`${message}`, { cause: error });

throw error;

changed to re throw the original error!

leoraba · 2026-03-19T21:14:42Z

PR updated. ready for code review :)

joneubank

YESSSS LETS GO!

leoraba added 2 commits March 3, 2026 16:59

group errors outside loop

8613b31

combine data to validate improvement

a36d8ad

leoraba marked this pull request as draft March 5, 2026 16:19

leoraba added 4 commits March 5, 2026 15:37

commit worker thread

0159717

db index and new submission status

c37773d

setting up workerpool

b3ce751

revert baseDependencies db pool

7a4961b

leoraba commented Mar 10, 2026

View reviewed changes

leoraba marked this pull request as ready for review March 10, 2026 20:51

leoraba requested review from joneubank and justincorrigible March 10, 2026 20:51

leoraba added 2 commits March 11, 2026 17:09

Merge branch 'main' into commit_submission_worker

0eeed7c

fix inconsistencies

a5b9495

leoraba changed the title ~~#51 Submission Validation in worker thread~~ #51 Commit Submission in worker thread Mar 11, 2026

joneubank requested changes Mar 15, 2026

View reviewed changes

leoraba added 4 commits March 17, 2026 08:15

improving TSdocs and small code refactor

b79f975

refactor groupSchemaDataByEntityName

398b658

refactor worker pool

2ccc942

update comments and logs, revert unused code change

e9242ab

leoraba commented Mar 18, 2026

View reviewed changes

leoraba requested a review from joneubank March 18, 2026 13:11

joneubank requested changes Mar 18, 2026

View reviewed changes

joneubank reviewed Mar 18, 2026

View reviewed changes

leoraba added 2 commits March 19, 2026 14:43

terminate worker thread

ccdacc4

error handling

c0b7114

leoraba requested a review from joneubank March 19, 2026 21:14

joneubank approved these changes Mar 19, 2026

View reviewed changes

leoraba merged commit 14ddd0e into main Mar 20, 2026
2 checks passed

leoraba deleted the commit_submission_worker branch March 20, 2026 12:45

leoraba linked an issue Mar 24, 2026 that may be closed by this pull request

Prod Issue: Loading the submission details page takes several minutes imicroseq/roadmap#160

Closed

		const invalidRecordErrors = findInvalidRecordErrorsBySchemaName(resultValidation, entityName);
		const hasErrorByIndex = groupErrorsByIndex(invalidRecordErrors);

	throw new Error(`${message}`, { cause: error });
	throw error;

Conversation

leoraba commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Issues

Description of Changes

Database changes

Worker threads

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joneubank left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joneubank left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joneubank left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leoraba commented Mar 19, 2026

Uh oh!

joneubank left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leoraba commented Mar 3, 2026 •

edited

Loading