7. Licenses Inventory

Disclaimer

This is quite experimental feature, with results which must be verified by a human. You must deal with platforms and APIs policies and fullfil them.

This is software is distributed on "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

Such caveats are about versions of components (not checked) and version names (not sure they are related to the good components)

Aim of this tool

The aim of this tool is to help open source referents and also curious developers to be able to geht the libre or open source licenses applied on the dependencies used by the project. Indeed, a lot of projects use package / dependency manager so as to list the librairies to use, but it could be a bit tricky to know what are the licenses in use. So this tool will parse specific files and request platforms like GitHub, Maven Central, pub.dev, crates.io or npmjs.org for details.

Special note

This tool has been entirely implemented by one of our colleagues who is visually impaired (implementation, unit tests and integration tests). His work is great and like all softwres can be updated. Thus if you want to improve sources or share suggestions, refer to our code of conduct first.

Supported environments

• Node.js / JavaScript with package.json file
• Go with go.mod file
• Dart / Flutter with pubspec.yaml file
• JVM / Kotlin with build.gradle and build.gradle.kts files
• Swift / Swift Package Manager with Package.swift file
• Rust with Cargo.lock file
• Cocoapods with Podfile file

How to use it

Fill the configuration file

Before to use the tools, the file 'config.ini' is at the root of the project, you have to personalize this file.

For example:

[dependencies]
# Where to find the package manager files above, must be defined, target must exist
path to parse = /absolute/path/to/project_to_test
# The name of the package manager file to process stored above, must be defined
the filenames = go.mod
# For outputs, must be defined, target must exists
path to store the licenses = /absolute/path/to/project_to_test-licences
# Erros maangement if requests failed
number of authorized successive errors = 1

where:

• path to parse contains the dependencies manager files
• the filenames contains the names of the dependencies manager files to process
• path to store the licenses points to a folder containing the result files prefixed by "licenses_" if license has been found or "errors_" if an error occured (e.g. requests limits in web site, etc)
• number of authorized successive errors is the number of succesive errors authorized before ignoring the next dependencies to treat

Run the tool

python3 sources/main.py

Run the tests

To run the tests (all must pass):

# Integration tests some user inputs
python3 -m pytest -s tests/integrationtests/*.py

# Unit tests 
python3 -m pytest tests/unittests/*.py

To run the unit tests, you must get the assets attached as artificats to the release you got. For integration tests, get the real_data folder in the integrationtests folder and move it to the same folder in your project. For unit tests, get the data folder in the unittests folder and move it to the same folder in your project.

Then you will have to update the configuration values defined in all the config files of the data sets. Indeed, absolute paths are used, so you must look for any "🥜" and replace by the path fragments leading to the folders.

Managed platforms

Go language

go.mod files are managed. Depending to the go.mod definitions implementation, some cases can be applied:

1. github.com will be requested if dependency starts by github.com
2. pkg.go.dev will be requested for other cases

For example:

module ...

go 1.15

require (
	emperror.dev/errors v0.4.2                                          // <--- Request pkg.go.dev
	github.com/antihax/optional v1.0.0                                  // <--- Request github.com
	golang.org/x/tools v0.0.0-20201014231627-1610a49f37af // indirect   // <--- Not managed
	k8s.io/api v0.20.2                                                  // <--- Request pkg.go.dev
	sigs.k8s.io/controller-runtime v0.7.2                               // <--- Request pkg.go.dev
)

Gradle environment

build.gradle and build.gradle.kts files are managed. Some platforms are requested like Maven Central (search.maven.org) and GitHub (through api.github.com).

*Warning: unstable feature with maybe Maven Central troubles, missing results sometimes

Managed (tested) keywords are:

    implementation 'ns_d:c_d:4.4.4'
    compile 'ns_e:c_e:5.5.5'
    api 'ns_f:c_f:6.6.6'
    testImplementation 'ns_g:c_g:7.7.7'
    androidTestImplementation 'ns_h:c_h:8.8.8'
    annotationProcessor 'ns_i:c_i:9.9.9'
    compileOnly 'ns_j:c_j:10.10.10'

But the following are not managed yet:

    implementation('...') {
        exclude module: '...'
    }

    androidTestImplementation('...') {
        exclude group: '...', module: '...'
    }

Rust environment

Cargo.lock files are also managed. Crates (crates.io) platform will be requested for each dependency found.

JavaScript / Node.js environment

package.json files can be parsed too. The platform npmjs.org_** wll be requested for each dependency found.

Swift / SPM environment

If you use Swift Package Manager, you can parse Package.swift file. The tool will extract the dependency URLs and request some forges, e.g. **_github.com.

Dart / Flutter environment

The pubspec.yaml files can also be processed. For each dependency found, the pub.dev_** platform will be requested.

CocoaPods case

The Podfile files can also be processed and the cocoapods.org website will be used.

Limits

The dependencies are always treated in the same order. The downloading can be aborted. For example, a website can limit the number of requests for a done duration. In this case, all the following dependencies will have the same error. For Gradle, we can limit the number of authorized errors to avoid to continue the unuseful downloadings.

Scenarios

With no error:

• the dependency is saved in the file "licenses_platform.txt"
• the dependency is not saved in the file "errors_platform.txt"

On error:

• the dependency is not saved in the file "licenses_platform.txt"
• the dependency is saved in the file "errors_platform.txt"

Example of use

The user executes the tools. If dependencies are on error, the tools displays, for each treated platform, the number of new dependencies to treat, the number of dependencies on error and the number of duplicated (dependencies on error which are in the new dependencies)/ The tools asks to the user to treat the dependencies on error or the new dependencies or to quit the program.

If they are only new dependencies, the tools does not display the number of dependencies. If they are only dependencies on error, the tools does not display the number of dependencies.

So, we can search licenses for dependencies which have not been treated following an error during the downloading.

Notes

The tool downloads a file for each dependency it found in the dependency manager file. These files containing the licenses are in directory like 'licenses/sub_folder', where 'sub_folder' is created for each platform: Gradle, Rust, etc.

A file 'licenses.txt' is created in the folder 'licenses'. This file contains the list of the licenses for each dependency. To personalize this folder, use 'config.ini'.

Beware of your proxys or public IP address to not be blocked by such platforms, and avoid flooding them.