Hi all,
I recently ran into an unexpected memory explosion running SCimplify_for_Seurat on an object with 500k cells:
Error in table(clusters, supercell_membership) :
attempt to make a table with >= 2^31 elements
It seems like the error arises when calculating purity metrics on metadata fields with high cardinality. In my case there was a Cell_ID column for which each cell had a unique value, resulting in a huge supercell_membership table.
The simple workaround of course is to just remove the problematic metadata fields from the original object before running SCimplify_for_Seurat -- these can always be replaced afterwards. My only suggestion would be to add a check for metadata with many unique values and exclude these from purity calculations.
Thanks for the package!
Tim
Hi all,
I recently ran into an unexpected memory explosion running SCimplify_for_Seurat on an object with 500k cells:
It seems like the error arises when calculating purity metrics on metadata fields with high cardinality. In my case there was a Cell_ID column for which each cell had a unique value, resulting in a huge supercell_membership table.
The simple workaround of course is to just remove the problematic metadata fields from the original object before running SCimplify_for_Seurat -- these can always be replaced afterwards. My only suggestion would be to add a check for metadata with many unique values and exclude these from purity calculations.
Thanks for the package!
Tim