Add indexes

[Huber1]

Feature Request

Describe the feature you would like to see added:

Optional indexes on fields would reduce ram usage with large databases

Why do you believe this feature would be useful?

Would be more resource efficient

Have you tried implementing this feature yourself, or have you seen it implemented in other projects?

Not tried to implement myself. Most SQL Databases have Indexes (also for faster search)

Additional information:

A possible implementation could be like the following:

type User has an Index on id and email, because these fields are the ones mostly used for searching.
in Memory, only the fields with indexes are stored, the other ones are stored only on disk.
Searching is performed like before with objects in memory.
If an object matches, the additional fields are read from disk.
This would preserve the speed of in-memory while outsourcing most data to the disk

[IamSebastianDev]

Sounds like a good idea! Implementing optional indexes per collection could even be done without a breaking change, by defining all existing keys as indexes, when no index property exists.

An example implementation could look like this:

const users = await db.collect<User>('users', {
	validate: {
		name: [NotNull, IsText()],
		age: [NotNull, IsInt({ min: 18 })],
		email: [NotNull],
	},
	indexedFields: ['id', 'email'],
});

The type of the indexed field would then be indexedFields: Keyof T & { id: number }, this would mean that we have a singular level of indexable fields, which should not be an issue, as nested object indexing would be kind of a weird thing.

[IamSebastianDev]

This should be implemented after we have reliable Benchmarking tests, to check performance impact on Search and IO Read