#1141 Set Up SearchClient :mag: by demariadaniel · Pull Request #1021 · overture-stack/arranger

demariadaniel · 2026-01-29T20:25:43Z

Summary

Base feature branch for supporting multiple Search Client options in Arranger

Issues

[Cloud Migration] Step 1: Setup OpenSearch Client to Work with Arranger nci-hcmi-catalog/portal#1141

Description of Changes

Creates a shared SearchClient function for determining configuration and returning appropriate client
Adds OpenSearch client library
Initial Type structure for SearchClient
Prettier & Typescript cleanup across multiple files

Setup

Requires Arranger npm install:

cd /arranger
npm i

Requires running ElasticSearch / OpenSearch instances
Basic OpenSearch setup:
https://docs.opensearch.org/latest/getting-started/quickstart/#option-1-try-opensearch-in-one-command
Requires mock Arranger configs
... then run npm run server

Successful connection to Arranger server with both OpenSearch and Elastic setups:

OpenSearch:

ElasticSearch:

Readiness Checklist

Self Review
- I have performed a self review of code
- I have run the application locally and manually tested the feature
- I have checked all updates to correct typos and misspellings
Formatting
- Code follows the project style guide
- Autmated code formatters (ie. Prettier) have been run
Local Testing
- Successfully built all packages locally
- Successfully ran all test suites, all unit and integration tests pass
Updated Tests
- Unit and integration tests have been added that describe the bug that was fixed or the features that were added
Documentation
- All new environment variables added to .env.schema file and documented in the README
- All changes to server HTTP endpoints have open-api documentation
- All new functions exported from their module have TSDoc comment documentation

demariadaniel · 2026-02-05T21:13:41Z

integration-tests/admin/test/index.test.js

@@ -1,9 +1,12 @@
+import { after, before, describe } from 'node:test';
+


Heads up that multiple files include simple prettier cleanup changes to clear up the repo (mostly order of imports, whitespace, & falsy handling) or Typescript updates to declutter the repo.

The main change is the creation of a SearchClient which replaces instances of ElasticSearch Client.

demariadaniel · 2026-02-05T21:14:57Z

modules/server/src/admin/schemas/AggsState/utils.ts

-    const aggsState: I_AggsState[] = mappingToAggsState(mapping);
-    return { timestamp: timestamp(), state: aggsState };
-  };
+	(es: SearchClientType) =>


All usage of @elastic Client is replaced by our SearchClientType

demariadaniel · 2026-02-09T23:55:07Z

modules/server/src/searchClient/index.ts

+const SearchClient = async (options: ESClientOptions | OSClientOptions) => {
+	const { ES_HOST } = ENV_CONFIG;
+	const searchConfig = await (await fetch(ES_HOST)).json();
+	const { distribution } = searchConfig.version;


See:
https://docs.opensearch.org/latest/api-reference/cluster-api/info/

https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-info

Cluster Info provides a distribution field which lets us clearly identify when an OpenSearch instance is running.

This will need to be expanded when investigating ElasticSearch 8. But for OpenSearch hopefully the distribution field will suffice.

demariadaniel · 2026-02-10T16:14:10Z

modules/server/src/admin/schemas/ExtendedMapping/utils.ts

-import { getProjectStorageMetadata, updateProjectIndexMetadata } from '../IndexSchema/utils';
+import { type AllClients } from '#searchClient/index.js';
+
+import { extendMapping } from '../../../mapping/index.js';


With types cleaned up I found extendMapping is undefined, not sure what this is actually meant to be

demariadaniel · 2026-02-11T19:57:13Z

modules/server/src/searchClient/index.ts

Main changes here in [modules/server/src/searchClient/index.ts]

joneubank

So great seeing this working end to end. Thanks for the big effort.

I saw quite a few places for improvement in how we are handling the configuration and setup for arranger server now that it supports multiple search services, so that's where this feedback focuses.

We also require some updates to documentation for arranger to communicate to operators about the search supports and how to configure it. @b-f-chan this may fit better into a separate ticket. @demariadaniel for this review, let's just make sure we document somewhere the valid options for the SEARCH_SERVICE_TYPE env var.

There is future work needed to make our SearchClient type strongly typed without all the anys. That will take some dedicated time and effort to get right so I couldn't fit it into this round of feedback comments.

Thanks again Dan!

joneubank · 2026-03-13T13:56:32Z

modules/server/src/searchClient/index.ts

+		// Determine which search client is being used
+		// Distribution field is specific to OpenSearch
+		// Else, if number field is a valid string, default to 'elasticSearch' as client type
+		const { distribution, number } = response.version;
+		const version =
+			typeof distribution === 'string' ? distribution : typeof number === 'string' ? 'elasticsearch' : undefined;


Should this check be case insensitive?

joneubank · 2026-03-13T13:57:44Z

modules/server/src/searchClient/types.ts

+
+export type SearchConfig = {
+	node: string;
+	clientType?: SupportedClientTypes | string;


Why can this type be string?

SupportedClientTypes are a subset of string so this union type is equivalent to string.

I believe this should just be typed as SupportedClientTypes.

joneubank · 2026-03-13T13:59:23Z

modules/server/src/server.ts

-export const buildEsClient = (esHost = '', esUser = '', esPass = '') => {
-	if (!esHost) {
-		console.error('no elasticsearch host was provided');
+export const createSearchConfig = (host = '', username = '', password = '', clientType = '') => {


Don't default clientType to an empty string. It should be undefined if not defined. The value is meant to be of type SupportedClientType

joneubank · 2026-03-13T14:03:16Z

modules/server/src/searchClient/index.ts

+export default async function getSearchClient(config: SearchConfig) {
+	try {
+		const configClientType = !config.clientType ? await getClientVersion(config) : config.clientType;
+		const clientType = supportedClientValues.find((key) => typeof key === 'string' && key === configClientType);


This typeof check is not doing anything.

I think we can ignore this change, because the client type should be checked and returned by getClientVersion().

joneubank · 2026-03-13T14:07:56Z

modules/server/src/server.ts

+	const searchConfig: SearchConfig = {
+		node: host,
+		auth,
+		clientType,


SEARCH_CLIENT is type string | undefined. If it is not undefined we should be confirming in this file that it is a valid client type, and therefore allows us to type this config as SupportedClientType and not the union with string, or we reject the user's configuration. Take a look at when host is not defined, we throw an error with logs indicating why their config is invalid, and it would be best to do a similar thing here. This prevents the server starting to let the user know that their config is invalid.

joneubank · 2026-03-13T15:04:46Z

modules/server/src/searchClient/createElasticSearchClient.ts

+export function createElasticSearchClient(options: ESClientOptions) {
+	const elasticSearchClient = new Client(options);
+
+	const searchClient: SearchClient = {
+		indices: {
+			create: async (input: any, options?: any) => {
+				const output = await elasticSearchClient.indices.create(input, options);
+				return output;
+			},
+			delete: async (input: any, options?: any) => {
+				const output = await elasticSearchClient.indices.delete(input, options);
+				return output;
+			},
+			exists: async (input: any, options?: any) => {
+				const output = await elasticSearchClient.indices.exists(input, options);
+				return output;
+			},
+			getMapping: async (input: any, options?: any) => {
+				const output = await elasticSearchClient.indices.getMapping(input, options);
+				return output;
+			},
+		},
+		cat: {
+			aliases: async (input: any, options?: any) => {
+				const output = await elasticSearchClient.cat.aliases(input, options);
+				return output;
+			},
+		},
+		bulk: async (input: any, options?: any) => {
+			const output = await elasticSearchClient.bulk(input, options);
+			return output;
+		},
+		index: async (input: any, options?: any) => {
+			const output = await elasticSearchClient.index(input, options);
+			return output;
+		},
+		search: async (input: any, options?: any) => {
+			const output = await elasticSearchClient.search(input, options);
+			return output;
+		},
+		update: async (input: any, options?: any) => {
+			const output = await elasticSearchClient.update(input, options);
+			return output;
+		},
+		create: async (input: any, options?: any) => {
+			const output = await elasticSearchClient.create(input, options);
+			return output;
+		},
+		delete: async (input: any, options?: any) => {
+			const output = await elasticSearchClient.delete(input, options);
+			return output;
+		},
+	};
+
+	return searchClient;
+}


Because our shared Client definition is so broad, this can currently be simplified to:

export function createElasticSearchClient(options: ESClientOptions) { const elasticSearchClient = new Client(options); return elasticSearchClient; }

This will compile without issue and still pass tests. This should not be surprising since all the functions here did no work.

Before updating like this though, the structure is still useful for when we actually enforce structure to the input and output arguments of our SearchClient interface.

Basically, this is an indication that the any usage in our SearchClient type is too broad.

joneubank · 2026-03-13T15:06:00Z

modules/server/src/searchClient/createOpenSearchClient.ts

+	clientType: 'opensearch';
+};
+
+export function createOpenSearchClient(options: OSClientOptions): SearchClient {


Same comment as for the ESClient, the shape of the SearchClient is compatible with the openSearchClient and with the current type we don't need all the boilerplate.

joneubank · 2026-03-13T15:06:30Z

modules/server/src/gqlServer.ts

 */
 type DefaultRoot = Root;
-export type Resolver<Root = DefaultRoot, QueryArgs = Object, ReturnValue = undefined> = (
+export type Resolver<Root = DefaultRoot, QueryArgs = object, ReturnValue = undefined> = (


joneubank · 2026-03-13T15:07:24Z

modules/server/src/mapping/utils/esSearch.ts

+export default (esClient: SearchClient) => async (params: SearchResponse) => {
+	return await esClient.search(params);


Nice type improvements here.

joneubank · 2026-03-13T15:11:19Z

modules/server/src/searchClient/types.ts

+export type SearchClient = {
+	indices: {
+		create: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+		delete: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+		exists: (input: any, options?: any) => SearchClientResponseHandler<boolean>;
+		getMapping: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	};
+	cat: {
+		aliases: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	};
+	bulk: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	index: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	search: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	update: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	create: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+	delete: (input: any, options?: any) => SearchClientResponseHandler<Record<string, any>>;
+};


The crux of this work is this type definition, and the use of any here is not sufficient for us to publish this. However, due to the similarity of the ES and OS libraries this works fine with our current code base, especially since a lot of the code relies on untyped JS files.

We need to get a clear view of what we want out inputs and outputs to look like, and then do work to make sure all clients adhere to that shared interface. Let's followup on this after addressing the other comments around the search service type configuration.

demariadaniel added 4 commits January 29, 2026 14:54

Basic SearchClient, w/ some type & import cleanup

63a21aa

Working Test Client setup

dbcce45

Additional Type cleanup

6529046

Basic check for version.distribution, remove console.log

ca8179a

demariadaniel changed the title ~~WIP: Basic SearchClient, w/ some type & import cleanup~~ WIP: Basic SearchClient Jan 30, 2026

demariadaniel commented Feb 5, 2026

View reviewed changes

demariadaniel self-assigned this Feb 9, 2026

demariadaniel commented Feb 9, 2026

View reviewed changes

Updated Client Init, with corrected use of Promises & Types

1acb1ef

demariadaniel commented Feb 10, 2026

View reviewed changes

Refine Types and Function definitions

a774721

demariadaniel changed the title ~~WIP: Basic SearchClient~~ WIP: #1141 Set Up Basic SearchClient Feb 11, 2026

demariadaniel commented Feb 11, 2026

View reviewed changes

Separate Types file & update imports

92b3f74

demariadaniel requested review from joneubank, justincorrigible and mistryrn February 11, 2026 20:19

demariadaniel added 7 commits February 23, 2026 11:12

Types Setup

f3ca922

Standardized ArrangerSearchClient draft

a89be80

Cleanup: add elastic options, move type definitions

0c25998

Major Config Update

eaf5f6e

Updated Config definitions

5903fd3

Fixed Client Version bug

19223d3

Fixed Testing Round 1, Revised EsSearch definition

9ad4669

demariadaniel force-pushed the 1141/feat-setup-opensearch-client branch from d611c86 to 9ad4669 Compare March 9, 2026 15:28

demariadaniel added 2 commits March 9, 2026 11:44

Fix aggregation integration test issues

745525b

Prettier + Test Config change

8da16c0

demariadaniel changed the title ~~WIP: #1141 Set Up Basic SearchClient~~ #1141 SearchClient Mar 9, 2026

demariadaniel changed the title ~~#1141 SearchClient~~ #1141 SearchClient 🔍 Mar 9, 2026

demariadaniel changed the title ~~#1141 SearchClient 🔍~~ #1141 Set Up SearchClient 🔍 Mar 9, 2026

demariadaniel marked this pull request as ready for review March 9, 2026 19:29

joneubank requested changes Mar 13, 2026

View reviewed changes

demariadaniel added 4 commits March 16, 2026 11:15

Export searchClient

59d1828

Additional setup for methods used in HCMI

1bc23e6

Corrected method definitions

4cb254e

Fix deleteByQuery

dcea216

		@@ -1,9 +1,12 @@
		import { after, before, describe } from 'node:test';

		export default (esClient: SearchClient) => async (params: SearchResponse) => {
		return await esClient.search(params);

Conversation

demariadaniel commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Issues

Description of Changes

Setup

Readiness Checklist

Uh oh!

demariadaniel Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

demariadaniel Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joneubank left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

demariadaniel commented Jan 29, 2026 •

edited

Loading

demariadaniel Feb 5, 2026 •

edited

Loading

demariadaniel Feb 11, 2026 •

edited

Loading