Stemming with English preset

[amber-huo]

Hi,

I'm using the English preset from flexsearch/lang/en in my Encoder. I noticed that by itself, it will over-stem, for example, cut the '-ing' off of words like 'king', in which '-ing' is not actually a suffix. I don't know a lot about customizing the Encoder, but I still want to have stemming for appropriate words, so I was hoping for a built-in way to do this, while being able to identify words like 'king'.

Here is my current setup with the English preset:

const encoder = new Encoder(
  Charset.Default,
  EnglishPreset,   // import EnglishPreset from 'flexsearch/lang/en';
);

const index = new Document({
  document: {
    store: true,
    index: [{
      field: 'text', 
      tokenize: 'forward',
      encoder: encoder,
    }],
  },
});

Would this be possible? Or can you recommend some way to extend the English preset with custom options to make it intelligent to words where word endings are just arbitrary substrings?

I would be happy to provide more information about anything if needed.

[wcolts2000]

We hit the same over-stemming with the built-in English stemmer. It's a fixed suffix→replacement map, so it strips endings structurally without checking whether a real stem is left (king → k, etc.).

What worked for us was to drop the built-in stemmer and run a proper algorithmic stemmer instead - the stemmer package (Porter). Porter's measure rules keep short roots intact while still stemming genuine inflections:

king → king running → run
sing → sing meetings → meet
bring → bring kings → king

Our setup (flexsearch 0.7.43) - we're pinned to 0.7 by the RxDB premium FlexSearch plugin, where the stemmer option only accepts a suffix map (it compiles a regex from the keys), so we plug the function in through a custom encode. It runs over both documents and queries, so index/query stems line up:

import { stemmer } from 'stemmer';

const stopwords = new Set([/* … */]);
const SPLIT = /[\p{Z}\p{S}\p{P}\p{C}]+/u; // same word boundary flexsearch uses

const index = new Index({
tokenize: 'strict',
encode: (str) =>
String(str).toLowerCase().split(SPLIT)
.filter((w) => w && !stopwords.has(w))
.map(stemmer),
});
On 0.8 (what you're using) - the stemmer option is still coerced to a Map, but the Encoder has a finalize hook (custom function, array in → array out) that's the natural place for a real stemmer. One gotcha: run it over a charset that doesn't rewrite letters, or the stemmer gets corrupted input - Charset.Default/Normalize apply dedupe (collapsing double letters, so running → rune) before finalize. Charset.Exact (or dedupe: false) avoids that:

import { Encoder, Charset } from 'flexsearch';
import { stemmer } from 'stemmer';

const encoder = new Encoder(Charset.Exact, {
// add filter: [...stopwords] here for stop-word removal;
// don't also set a built-in stemmer, or you'll double-stem.
finalize: (terms) => terms.map(stemmer),
});
// encoder.encode('running king meetings') => ['run', 'king', 'meet']

Hope that helps.