Support for GitHub dependency graph

[Gedochao]

So I figure this might be a place like any other to raise this.

There currently is no automation for generating a GitHub dependency graph with Mill 0.12+

We do have @ckipp01's solution for up to 0.11, but it has been archived since:

• https://github.com/ckipp01/mill-dependency-submission (the GitHub action)
• https://github.com/ckipp01/mill-github-dependency-graph (the Mill plugin)

While it has been updated up to Mill 0.12.5, it seems it's borked with the Scala CLI build (currently on Mill 0.12.14).
We removed it in:

• Remove the github-dependency-graph CI workflow VirtusLab/scala-cli#3703

If anyone's interested in the bug, it's reproducible with

./mill -i --import ivy:io.chris-kipp::mill-github-dependency-graph::0.2.7 io.kipp.mill.github.dependency.graph.Graph/submit

in the https://github.com/VirtusLab/scala-cli repo.

[63/65] ================== io.kipp.mill.github.dependency.graph.Graph/submit ================= 35s
java.lang.ClassCastException: class scala.Tuple2 cannot be cast to class scala.Tuple3 (scala.Tuple2 and scala.Tuple3 are in unnamed module of loader 'app')
    mill.resolve.Resolve$.$anonfun$invokeCommand0$1(Resolve.scala:166)
    mill.resolve.Resolve$.$anonfun$invokeCommand0$1$adapted(Resolve.scala:166)
    scala.collection.MapOps$WithFilter.$anonfun$withFilter$1(Map.scala:402)
    scala.collection.MapOps$WithFilter.$anonfun$withFilter$1$adapted(Map.scala:402)
    scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:479)
    scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:601)
    scala.collection.immutable.List.prependedAll(List.scala:152)
    scala.collection.immutable.List$.from(List.scala:685)
    scala.collection.immutable.List$.from(List.scala:682)
    scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
    scala.collection.immutable.Iterable$.from(Iterable.scala:35)
    scala.collection.immutable.Iterable$.from(Iterable.scala:32)
    scala.collection.IterableOps$WithFilter.flatMap(Iterable.scala:903)
    mill.resolve.Resolve$.invokeCommand0(Resolve.scala:166)
    mill.resolve.Resolve$.$anonfun$instantiateCommand$1(Resolve.scala:151)

That being said, since the old solution has been archived, it is still in the air if this will be fixed.
Maintaining it would likely mean adding an implementation for Mill 1.0.0 once it is out (and a 1.0.0-RC1 one wouldn't hurt).

What is the GitHub dependency graph and why do we need it?

• https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph

So where I'm leading with it:

• are there any plans to fork the mill-github-dependency-graph plugin in the com-lihaoyi org?
• are there any alternatives for delivering the graph in repos on the newest Mill versions?

[lefou]

This is likely related to SBOM support

• Add ability to generate an SBOM for Java/Scala dependencies #4062
• Generation Software Bill of Materials (SBOM) #4757

Also, the mentioned and now archived plugin didn't support Mill's package.mill files. All modules where associated to the single build.sc file in GitHub's dependency view, which made them pretty unusable and uninformative for non-trivial projects.

[lefou]

In its current form, the GibHub dependency graph isn't very helpful, as it does not understand the notion of modules. In contrast to tools like Maven, where each module is associated to another buildfile (pom.xml), in Mill a single buildfile ( build.mill) may contain various modules and esp. a mix of library, application, test, integration-test, and other scopes. These are all mixed up in the GitHub dependency graph (they're only grouped by buildfile), which makes them useless for any serious dependency analysis. This is likely the main reason, there are no other attempts to integrate with it (or revive mill-github-dependency-graph).

[gamlerhart]

My current approach for SBOMs was that it is a 'per-module' opt in: So that you decide what is actually the 'artifact' you ship and what is dependencies of tooling, tests etc.

As for GitHub, I guess for smaller projects submitting the dependencies of 'production' modules in to the top level would be good enough for many projects.

[ckipp01]

In its current form, the GibHub dependency graph isn't very helpful, as it does not understand the notion of modules.

So just to provide a bit more context, this was my feeling as well. For the Mill projects I had, this ended up never really being that useful to me. Once the 0.12.x series started to come out it broke compat with a few of my plugins, and since I'm not actively using Mill anymore, my motivation to fix them was pretty low, which lead me to retire it. If you also look into the code, there are quite a few hacks that I included to account for some odd stuff I encountered with Coursier, for example this or this, but I never got any insight into those issues. So overall, I just didn't feel like it was worth pursuing further.

I do think having this would be great, but I also think it'd take some work on the GitHub side. A while back they did put out a call for people to provide feedback, which I did, but it was never responded to and I think they way they have implemented this for the JVM ecosystem is just fundamentally flawed. You can see some examples of this in here which show how often these links for Scala are simply incorrect.

Either way, if there is anything I can do to port this somewhere else, or help someone else take it over, I'd be happy to.