What is the difference between the datasets in HuggingFace channel and anaconda channel?

[horizon86]

https://anaconda.org/conda-forge/datasets is 2.4.0
https://anaconda.org/huggingface/datasets is 2.4.0
but https://anaconda.org/anaconda/datasets is 1.12.1

These datasets are all claimed to be from huggingface datasets.

I can not use load_datasets('imagefolder', data_dir='path_to_my_files') with datasets 1.12.1, but it works with datasets 2.4.0.
So why?

Thanks!

[mariosasko]

Hi! The channels conda-forge and huggingface store identical releases of datasets and are maintained by us. On the other hand, the anaconda channel represents the default package index maintained by Anaconda folks. It's unclear what inclusion criteria they use to determine whether a package should be added to the index, but we can request the version update here: https://github.com/ContinuumIO/anaconda-issues.

@LysandreJik Is there something we can do to keep our packages up-to-date at all times in the default index?

I can not use load_datasets('imagefolder', data_dir='path_to_my_files') with datasets 1.12.1, but it works with datasets 2.4.0.
So why?

imagefolder was added in ˙datasets 2.0.0, so 2.0.0 is the minimal version required to use this feature.

[LysandreJik]

I am not aware; @adrinjalali, would you know by any chance?

[adrinjalali]

I have my take on conda-forge vs main channel here.

For this particular case, you can think of the main channel as kind of a Linux distro which does releases every now and then like Debian, and of course until the next release packages are going to be out of date compared to the upstream.

I've even noticed scikit-learn being about a year out of date on the main channel.

This is partly why many communities have switched all their documentation to recommend installing from conda-forge, since maintainers don't have pretty much any control on what goes on the main channel. Also note that the main channel can have packages with patches w/o the consent of the maintainers.

[msapaydin]

conda-forge channel and huggingface channel have different datasets versions (4.3.0 vs 4.4.1, respectively) and the one using huggingface datasets version works with the datasets documentation.
`

mehmetserkanapaydin@fedora:~$ conda install -c huggingface -c conda-forge datasets
2 channel Terms of Service accepted
Channels:

• huggingface
• conda-forge
• defaults
  Platform: linux-64
  Collecting package metadata (repodata.json): done
  Solving environment: done

Package Plan

environment location: /home/mehmetserkanapaydin/miniconda3/envs/coding

added / updated specs:
- datasets

The following NEW packages will be INSTALLED:

dataclasses conda-forge/noarch::dataclasses-0.8-pyhc8e2a94_3

The following packages will be UPDATED:

datasets conda-forge::datasets-4.3.0-pyhcf101f~ --> huggingface::datasets-4.4.1-py_0
`