Case-sensitive validation of license expressions?

[choeppler]

While the latest SPDX standard specifies that license identifiers should (mostly) be treated case-insensitive (https://spdx.github.io/spdx-spec/v3.0.1/annexes/spdx-license-expressions/#case-sensitivity), I have a use case where we'd like to validate that all identifiers in an expression match with the case of the canonical identifier on the SPDX License List. Is there a way to do this with license-expression, e.g, by somehow accessing the original token from the parsed LicenseSymbol?

[DennisClark]

@choeppler thanks for creating the issue. Could you please explain the purpose and value of the use case that you are describing?

[pombredanne]

@choeppler there is not exactly what you need, but we could add that easily, or you can do that easily with the current code, if you want to be super strict:

See this example:

>>> import license_expression as lic
>>> ls = lic.get_spdx_licensing()

>>> ls.validate("gpl-2.0 OR Mit", strict=True)
ExpressionInfo(
    original_expression='gpl-2.0 OR Mit',
    normalized_expression='GPL-2.0-only OR MIT',
    errors=[],
    invalid_symbols=[]
)

>>> ls.validate("gpl-2.0-only OR Mit", strict=True)
ExpressionInfo(
    original_expression='gpl-2.0-only OR Mit',
    normalized_expression='GPL-2.0-only OR MIT',
    errors=[],
    invalid_symbols=[]
)

>>> exi = ls.validate("gpl-2.0-only OR Mit", strict=True)
>>> exi.original_expression == exi.normalized_expression
False
>>> assert exi.original_expression == exi.normalized_expression
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError


>>> exi = ls.validate("gpl-2.0+ with bison-exception-2.2", strict=True)
>>> exi
ExpressionInfo(
    original_expression='gpl-2.0+ with bison-exception-2.2',
    normalized_expression='GPL-2.0-or-later WITH Bison-exception-2.2',
    errors=[],
    invalid_symbols=[]
)
>>> assert exi.original_expression == exi.normalized_expression
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

So the trick is to validate the expression, and if valid, further check that the original expression is equal to the normalized expression.
Note that this would go beyond just validating the case, as it will also check preferred form vs. older deprecated form (with a trailing + for example).

[pombredanne]

@choeppler you can also get the symbols alright:

>>> exp = ls.parse("gpl-2.0+ with bison-exception-2.2")
>>> exp.get_symbols()
[LicenseWithExceptionSymbol(license_symbol=LicenseSymbol('GPL-2.0-or-later', aliases=('GPL-2.0+', 'GPL 2.0+'), is_exception=False), exception_symbol=LicenseSymbol('Bison-exception-2.2', is_exception=True))]

Or access the underlying parse tree of the boolean expression

>>> exp = ls.parse("(apache-2.0 and gpl-2.0+ with bison-exception-2.2) or (mit or gpl-2.0)")
>>> print(exp.pretty())
OR(
  AND(
    LicenseSymbol('Apache-2.0'),
    LicenseWithExceptionSymbol('GPL-2.0-or-later WITH Bison-exception-2.2')
  ),
  OR(
    LicenseSymbol('MIT'),
    LicenseSymbol('GPL-2.0-only')
  )
)

[choeppler]

@choeppler you can also get the symbols alright:

>>> exp = ls.parse("gpl-2.0+ with bison-exception-2.2")
>>> exp.get_symbols()
[LicenseWithExceptionSymbol(license_symbol=LicenseSymbol('GPL-2.0-or-later', aliases=('GPL-2.0+', 'GPL 2.0+'), is_exception=False), exception_symbol=LicenseSymbol('Bison-exception-2.2', is_exception=True))]

Or access the underlying parse tree of the boolean expression

>>> exp = ls.parse("(apache-2.0 and gpl-2.0+ with bison-exception-2.2) or (mit or gpl-2.0)")
>>> print(exp.pretty())
OR(
  AND(
    LicenseSymbol('Apache-2.0'),
    LicenseWithExceptionSymbol('GPL-2.0-or-later WITH Bison-exception-2.2')
  ),
  OR(
    LicenseSymbol('MIT'),
    LicenseSymbol('GPL-2.0-only')
  )
)

@pombredanne , thank you for these examples. Actually, I tried solving the problem by examining the symbols after parsing. However, I didn't find a way to get the original token (e.g., 'gpl-2.0+' in your example above) and compare that with the license symbol 'GPL-2.0-or-later' or its aliases ('GPL-2.0+', 'GPL 2.0+').

[choeppler]

@choeppler there is not exactly what you need, but we could add that easily, or you can do that easily with the current code, if you want to be super strict:

See this example:
So the trick is to validate the expression, and if valid, further check that the original expression is equal to the normalized expression. Note that this would go beyond just validating the case, as it will also check preferred form vs. older deprecated form (with a trailing + for example).

I wouldn't mind being super strict about expressions like GPL-2.0+. So I've tried your approach of checking original expression equals the normalized expression. However, that also catches cases which I'd deem okay, e.g., 'LicenseRef-scancode-libzip' becomes 'BSD-3-Clause' (since the license texts only exhibit minor, non-consequential differences, I guess):

>>> ls = license_expression.get_spdx_licensing()
>>> exi = ls.validate('LicenseRef-scancode-libzip', strict=True)
>>> assert not exi.errors
>>> assert exi.original_expression == exi.normalized_expression, f"{exi.original_expression} != {exi.normalized_expression}"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: LicenseRef-scancode-libzip != BSD-3-Clause
>>> ls.parse('LicenseRef-scancode-libzip')
LicenseSymbol('BSD-3-Clause', aliases=('LicenseRef-scancode-libzip',), is_exception=False)
>

@DennisClark, what I try to achieve is validating user input data to be a valid license expression and then link the parsed symbols to a database, where the URI contains the license name and where case is relevant.

I think that would be implementable with license-expression if there was a way to access the original token after parsing but I haven't found a way how to do that, yet...