[FEAT] Add DINOV3 pre-training

[Alighieri1231]

Hello! I am playing with the library and while is pretty great I realized there is no Dinov3 pre-training added from scratch. The reason for this feature is while we can use distillation if the dataset is very different to the pre-trained dataset basically the distillation fails.

Example of how distillation of DINOV3 (using the pre-trained checkpoint) worked in lightly for my ultrasound dataset (the model collapses to only see brighness instead of global features):

Using the pre-training recipe for Dinov2 in lightly documentation gave me better results, but as reported in DINOV3 paper once we start training for many steps dense representation fails:

So adding gram anchoring and RoPE (DINOv3) would be amazing for overcoming this. Is this something planned in your milestones? Or maybe distillation is enough for your user base. Could you guide me on how could I pretrain dinov3 with your library? Thanks!

[liopeer]

Hi @Alighieri1231,
Awesome graphics that you produced, really nicely shows the limitations of both DINOv2 pretraining and knowledge distillation! For most use cases so far distillation (or DINOv2) were enough, but your example clearly shows that maybe we should again expand our efforts on the pretraining front. Since DINOv3's training pipeline involves several steps (pretraining, gram anchoring, high res adaptation), we will spend some time figuring out if it's possible to merge them into a single training run or if we're fine splitting the different steps.
We'll likely start looking into this starting mid next week!

[liopeer]

Btw, which version of distillation did you use? We've always had issue with getting distillationv2 to work properly with DINOv3 models (but worked well with DINOv2 models). For that reason we've also recently added "distillationv3", which is a hybrid between v1 and v2 and works well with DINOv3 models. It's not officially released yet (will happen early next week), but it's already usable if you install LightlyTrain from the main branch.

[Alighieri1231]

Thanks a lot for the response!

So some things here:

For pre-training of the DINOV2 model I am using the vits14 model.
For distillation of the DINOV3 model I am using the VIT-b/16 model (with the pre-trained checkpoint and distillationv2 - I will try to run today a smaller model with distillationv3).

• I know both are not necessary comparable due the model size but it was the first proposed approach.

Now, today I tried implementing the pretraining using lighthly with DINOV3 and VITs16 adding the gram loss as regularizer, but my results not were as good as DINOV2 (probably I am not considering something in the training process).

I will wait for some updates from your team side. Thanks again for the effort!

Best,

Emilio Ochoa
BME PhD Student

[liopeer]

Feel free to drop your exact training settings here (for all of the runs mentioned so far). Or alternatively you can also join our Discord (link in the README) and send them to me there. Also: Is the dataset that you're training on open-source?