Locale-aware sorting of output

[375gnu]

hunaftool should try setting locale so the output is sorted properly

[ssvb]

There are two factors here:

• First of all, the sorting is done in order to have some deterministic output, strictly based on the input files and without any influence of the other factors. This is useful for testing. If one user gets one output and the other user gets a different output from the same .aff+.dic files pair because the system locales of these users are different, then this may become a maintenance hell.
• And the second factor is that the standard Ruby library doesn't offer support for the Unicode Collation Algorithm (UCA) out of the box and an external gem needs to be used for that, such as https://github.com/twitter/twitter-cldr-rb#sorting-collation or some other alternative. So the Hunaftool's user will need to install an extra dependency and this is a hassle. Additionally, we would also need a similar Crystal language shard providing exactly the same functionality with exactly the same API, otherwise the "magic" of source level compatibility with both Ruby and Crystal will be gone.

[ssvb]

To address the first problem, we could possibly use the Hunspell's LANG directive:

       LANG langcode
              Set language code for language-specific functions of Hunspell. Use it to activate  special  casing
              of Azeri (LANG az), Turkish (LANG tr) and Crimean Tatar (LANG crh), also not generalized syllable-
              counting compounding rules of Hungarian (LANG hu).

Just set LANG be in the be_BY@tarask.aff file and get the Belarusian string collation algorithm, regardless of the user's default locale configured in their system. This way nobody gets discrepancies in their output and a lot of headache can be avoided.

To address the second problem, it's probably possible to have a minimalistic implementation of language-dependent collation algorithm in the Hunaftool's code itself. And then gradually add support for different languages, one at a time. Though I'm not very enthusiastic about this idea. It's just a few lines of code per one language, but this doesn't scale and reeks of NIH.

Why don't we just ask the users to do sorting using some external tool? Such as:

$ hunaftool be_BY@tarask.aff be_BY@tarask.dic be_BY@tarask.txt
$ LC_ALL=be_BY.UTF-8 sort be_BY@tarask.txt > be_BY@tarask_sorted.txt

I also see that there's https://sourceware.org/bugzilla//show_bug.cgi?id=1015

[375gnu]

sort will do the job with txt output, but not with csv. with csv it's hard to find new word forms when they are not ordered correctly

[ssvb]

This is a smallish stand-alone script to do the sorting via using https://github.com/twitter/twitter-cldr-rb. However it seems to be ridiculously slow:

#!/usr/bin/env ruby
require "twitter_cldr"
collator = TwitterCldr::Collation::Collator.new(:be)

results = []
File.open(ARGV[0]).each_line do |line|
  words = line.split(",").map {|word| word.strip }
  if words.size > 1
    results.push("#{words[0]}, #{collator.sort(words[1, words.size - 1]).join(", ")}")
  else
    results.push(words.join)
  end
end

collator.sort(results).each {|line| puts line }

Maybe some other Ruby gem can do a better job? Or maybe try Python or some other languages.

Edit: Here's Python:

#!/usr/bin/env python
import sys
from functools import cmp_to_key
from locale import *
setlocale(LC_ALL, "be_BY.UTF-8")

results = []
with open(sys.argv[1], 'r', encoding='UTF-8') as infile:
  for line in infile:
    line = line.strip()
    words = [word.strip() for word in line.split(',')]
    if len(words) > 1:
      line = f"{words[0]}, {', '.join(sorted(words[1:], key=cmp_to_key(strcoll)))}"
    results.append(line)

results.sort(key=cmp_to_key(strcoll))

if sys.argv[2]:
  with open(sys.argv[2], 'w', encoding='UTF-8') as outfile:
    for line in results:
      outfile.write(line + '\n')
else:
    for line in results:
      write(line + '\n')

It's interesting that these two programs produce different results because they handle apostrophes differently.

[ssvb]

@375gnu does the python script above satisfy your requirements? I release this primitive python code snippet under public domain or CC0 license.

But it doesn't feel right that the two alternative implementations using different backends even can't agree between themselves how to sort the Belarusian words correctly. And again, I still dislike the idea of pulling in an extra external dependency for hunaftool itself. Does anyone have any good suggestions?

[375gnu]

yes, it's good for me.

[ssvb]

That's great, I'm happy that it works.

I'm also considering to maybe still add some sort of custom non-standard sorting order with simple rules, so that at least ў/у, г/ґ and the other closely related Cyrillic and Latin letters are grouped together. I see that there was an initiative to have common European ordering rules for sorting strings. Not sure if it got any traction, but its ideas can be used as an inspiration. Hunaftool can opt to not exactly conform to the rules of any real locale, but at least provide a much more natural looking sorting order than LC_ALL=C. If anyone doesn't like that, then they can always sort the Hunaftool's output using any external tool.