Deprecate LSL String functions

[tapple]

• Part of https://feedback.secondlife.com/slua-alpha/p/remove-duplicated-luau-methods-from-ll-table

These have almost-direct replacements, but you need to switch from codepoint indices to byte offset indices:

• ll.GetSubString -> string.sub
• ll.SubStringIndex -> string.find
• ll.Ord -> utf8.codepoint
• ll.Char -> utf8.char

These have indirect replacements, and you need to switch from codepoint indices to byte offset indices:

• ll.DeleteSubString -> string.sub
• ll.InsertString -> string.sub

ll.ReplaceSubString can be usually be replaced with string.gsub:

1. Unless you need the 4th argument (replace some occurrences, not all)
2. But, you need to escape "%"

I didn't deprecate:

• ll.ParseString2List
• ll.ParseStringKeepNulls -> string.split
• ll.CSV2List -> string.split
• ll.List2CSV -> table.concat

Moved to #76 (MERGED):

• ll.Base64ToInteger -> llbase64.decode
• ll.Base64ToString -> llbase64.decode
• ll.StringToBase64 -> llbase64.encode
• ll.StringLength -> utf8.len

Moved to #77:

• ll.ToUpper
• ll.ToLower

[tapple]

I thought of deprecating ll.InsertString in favor of string interpolation, but, I don't think it's a good replacement

[Kristyku]

ll.Ord
ll.Char
ll.ToUpper
ll.ToLower

Cannot be deprecated, they support Unicode while Luau does not.

[Suzanna-Linn]

ll.ToLower()is useful with accents:

print(string.lower("CAFÉ"))  -- > cafÉ
print(ll.ToLower("CAFÉ"))    -- > café

[tapple]

print(string.lower("ДдAa")) -- Ддaa
print(ll.ToLower("ДдAa")) -- ддaa

Sounds like maybe string.lower should be deprecated instead

[tapple]

• reverted deprecation, and made a new PR: Deprecate string.lower in favor of ll.ToLower #77

[Suzanna-Linn]

ll.ToUpper()is useful with accents:

print(string.upper("café"))  -- > CAFé
print(ll.ToUpper("café"))    -- > CAFÉ

[tapple]

• reverted deprecation, and made a new PR: Deprecate string.lower in favor of ll.ToLower #77

[tapple]

print(string.upper("ДдAa")) -- ДдAA
print(ll.ToUpper("ДдAa")) --  ДДAA

sounds like maybe string.upper should be deprecated instead

[tapple]

• reverted deprecation, and made a new PR: Deprecate string.lower in favor of ll.ToLower #77

[Suzanna-Linn]

I think that most of the LL string functions are useful when working with extended characters.

The advantage of the SLua string library is that is faster (because they don't need to look at what kind of characters are in the string).

The best is to use ones or the others depending on what strings we have.

[Kristyku]

Here's code:

print("Дд")
print("Luau Lower: " .. string.lower("Дд"))
print("Luau Upper: " .. string.upper("Дд"))

print("SL Lower: " .. ll.ToLower("Дд"))
print("SL Upper: " .. ll.ToUpper("Дд"))

print("Luau Ord: " .. string.byte("Д"))
print("SL Ord: " .. ll.Ord("Д", 1))

local x = pcall(function() return string.char(1044) end)
print("Luau Char: " .. tostring(x))
print("SL Char: " .. ll.Char(1044))

and the output:

Object: Дд
Object: Luau Lower: Дд
Object: Luau Upper: Дд
Object: SL Lower: дд
Object: SL Upper: ДД
Object: Luau Ord: 208
Object: SL Ord: 1044
Object: Luau Char: false
Object: SL Char: Д

[tapple]

I can't find an issue with char:

print("Luau Char: " .. utf8.char(1044)) -- Д
print("SL Char: " .. ll.Char(1044)) -- Д

[Martin-Pitt]

It would be good to do a full spectrum check, as per the caveats I added on the wiki page for llChar

[tapple]

wrote a verification script:

for i = 0, 0x10FFFF do
--for i = 0, 0x10 do
--for i = 0xFFEE, 0x10FFFF do
--for i = 0xD800, 0xFFFF do
    if 0xD800 <= i and i < 0xE000 then
        -- surrogate pairs
        local luau_char = utf8.char(i)
        local utf8_raw = string.char(
            bit32.bor(0xE0, bit32.rshift(i, 12)),
            bit32.bor(0x80, bit32.band(0x3F, bit32.rshift(i, 6))),
            bit32.bor(0x80, bit32.band(0x3F, i))
        )
        assert(luau_char == utf8_raw)--, string.format('utf8.char(0x%x) incorrect', i))
        local luau_ord_valid, luau_ord = pcall(utf8.codepoint, luau_char)
        assert(not luau_ord_valid)--, string.format('"utf8.codepoint("\\u{%x}") did not error', i))
        assert(luau_ord == "invalid UTF-8 code") 
        -- lsl silliness --
        assert(ll.Char(i) == '\u{FFFD}')--, string.format('ll.Char(0x%x) incorrect', i))
        assert(ll.Ord(luau_char, 1) == 0x3F)--, string.format('ll.Ord("\\u{%x}", 1) incorrect', i))
    else
        local lsl_char = ll.Char(i)
        local luau_char = utf8.char(i)
        local lsl_ord = ll.Ord(luau_char, 1)
        local luau_ord = utf8.codepoint(luau_char)
        assert(luau_ord == i)--, string.format('utf8.codepoint("%s") = 0x%x ~= 0x%x', luau_char, luau_ord, i))
        if 0xFFFE <= i and i <= 0xFFFF then -- lsl silliness --
            assert(lsl_char == '\u{FFFD}')--, string.format('ll.Char(0x%x) incorrect', i))
        elseif i == 0 then -- lsl silliness --
            assert(lsl_char == '')--, string.format('ll.Char(0x%x) incorrect', i))
        else
            assert(lsl_char == luau_char)--, string.format('ll.Char(0x%x) = "%s" ~= utf8.char(0x%x) = "%s"', i, lsl_char, i, luau_char))
        end
        if 0xFFFE == i then -- lsl silliness --
            assert(lsl_ord == 0x3F)--, string.format('ll.Ord("\\u{%x}", 1) incorrect', i))
        else
            assert(lsl_ord == i)--, string.format('ll.Ord("%s") = 0x%x ~= 0x%x', luau_char, lsl_ord, i))
        end
    end
end
print("done")

• Added the new caveat about llOrd(<surrogate>) returning 0x3F to the wiki: https://wiki.secondlife.com/wiki/LlOrd
• Noted what utf8.char does on the wiki: https://wiki.secondlife.com/wiki/LlChar

[HaroldCindy]

I think it's worth thinking about the string functions and what their futures are, but it's pretty clear the internationalization situation is very hairy, even with the presence of the utf8 library. I'm not keen to deprecate any of the string functions until there's a clear plan of how to sanely handle internationalization under Luau.

Until then, the existing ll functions are at least no worse than what LSL already provides for internationalization, so let's not steer people away from them.