Interop with casatasks/casatools

[AlecThomson]

Hi there,

Apologies if this is wrong stream to report this issue. Myself and @tjgalvin have had adventures trying to install both python-casacore and casatasks/casatools on Python 3.10. I've been testing on MacOS Apple Silicon, but @tjgalvin has had similar results on linux.

First, installation with pip alone does not seem to work. I needed to resort to using a conda install. There are also issues with casatasks/casatools and how it handles matplotlib - it seems to be have been built against matplotlib<=3.5.1 and newer versions break.

Here is summary of my first stages of installation trials with minimal conda installs:

#!/usr/bin/env bash

# Create a conda environment for CASA in x86_64 architecture
CONDA_SUBDIR=osx-64 conda create -n casa_py310 python=3.10 pip ipython -y
conda activate casa_py310

# Install CASA
pip install casatasks casatools casadata python-casacore

# Test CASA
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"
# Fails - Matplotlib is called incorrectly
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/__init__.py", line 249, in <module>
#     from .plotbandpass import plotbandpass
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/plotbandpass.py", line 9, in <module>
#     from .private.task_plotbandpass import plotbandpass as _plotbandpass_t
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/private/task_plotbandpass.py", line 36, in <module>
#     warnings.filterwarnings("ignore",category=matplotlib.cbook.MatplotlibDeprecationWarning)
# AttributeError: module 'matplotlib.cbook' has no attribute 'MatplotlibDeprecationWarning'. Did you mean: 'VisibleDeprecationWarning'?

# Fix matplotlib
pip install "matplotlib<=3.5.1"

# Test CASA
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"

# Fails - boost is not installed
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/__init__.py", line 60, in <module>
#     from .msutil import *
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/msutil.py", line 29, in <module>
#     from casacore.tables.table import (table, taql,
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/table.py", line 40, in <module>
#     from ._tables import (Table,
# ImportError: dlopen(/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/_tables.cpython-310-darwin.so, 0x0002): Library not loaded: @rpath/libboost_python310.dylib
#   Referenced from: <E240941C-9D5F-3C5E-BF5D-93D9C4629B0E> /Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/_tables.cpython-310-darwin.so
#   Reason: tried: '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tho822/mambaforge/envs/x86/lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/casa_py310/bin/../lib/libboost_python310.dylib' (no such file), '/Users/tho822/mambaforge/envs/casa_py310/bin/../lib/libboost_python310.dylib' (no such file), '/usr/local/lib/libboost_python310.dylib' (no such file), '/usr/lib/libboost_python310.dylib' (no such file, not in dyld cache)

# Install boost
conda install boost -y

# Test CASA
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"

# casacore fails
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/__init__.py", line 60, in <module>
#     from .msutil import *
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/msutil.py", line 29, in <module>
#     from casacore.tables.table import (table, taql,
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/table.py", line 40, in <module>
#     from ._tables import (Table,
# ImportError: dlopen(/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/_tables.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__PyTraceMalloc_NewReference'

# But casatasks works
python -c "from casatasks import tclean; print(tclean.__doc__)"
# Works

python -c "from casacore.tables import table; print(table.__doc__)"
# Fails

Next I resorted to the conda-forge build. This can seemingly work, but the order of imports matters with casatasks/casatools:

#!/usr/bin/env bash

# Create a conda environment for CASA in x86_64 architecture
CONDA_SUBDIR=osx-64 conda create -n casa_py310 python=3.10 pip ipython python-casacore -y
conda activate casa_py310
pip install "matplotlib<=3.5.1" casatasks casatools
python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"
# Works
python -c "from casacore.tables import table; from casatasks import tclean; print(tclean.__doc__); print(table.__doc__)"
# Fails
# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/__init__.py", line 5, in <module>
#     from casatools import logsink as _logsink
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__init__.py", line 43, in <module>
#     from .image import image
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/image.py", line 4, in <module>
#     from .__casac__.image import image as _image
#   File "/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 10, in <module>
#     from . import _image
# ImportError: dlopen(/Users/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/_image.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN8casacore11ArrayColumnINSt3__17complexIdEEE8putSliceEyRKNS_6SlicerERKNS_5ArrayIS3_EE'

[AlecThomson]

On linux, using that last script, I get these errors:

~> python -c "from casatasks import tclean; from casacore.tables import table; print(tclean.__doc__); print(table.__doc__)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/__init__.py", line 60, in <module>
    from .msutil import *
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/msutil.py", line 29, in <module>
    from casacore.tables.table import (table, taql,
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/table.py", line 40, in <module>
    from ._tables import (Table,
ImportError: /home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casacore/tables/../../../../libcasa_python3.so.7: undefined symbol: _ZN8casacore6VectorIxSaIxEE6resizeERKNS_9IPositionEb

and

~> python -c "from casacore.tables import table; from casatasks import tclean; print(tclean.__doc__); print(table.__doc__)"
Traceback (most recent call last):
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/lib/libcasa_fits.so.7: undefined symbol: _ZTIN8casacore5ArrayIhEE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatasks/__init__.py", line 5, in <module>
    from casatools import logsink as _logsink
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__init__.py", line 43, in <module>
    from .image import image
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/image.py", line 4, in <module>
    from .__casac__.image import image as _image
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 17, in <module>
    _image = swig_import_helper()
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/site-packages/casatools/__casac__/image.py", line 16, in swig_import_helper
    return importlib.import_module('_image')
  File "/home/tho822/mambaforge/envs/casa_py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_image'

[AlecThomson]

Hey all - just wondering if anyone has had a chance to look at this?

Tagging @gijzelaerr @tammojan for some visibility.

[tammojan]

Hi, thanks for the ping. I think trying to have both the CASA python bindings and the python-casacore python bindings to casacore is near impossible. That is due to the underlying casacore. If you load two python bindings, two casacores will be loaded, but the symbols of these two will shadow each other. If the versions are binary incompatible (which is probably the case) this will in the best case lead to segfaults, in a good case you get the error on importing.

In theory this should be solvable, but on the short term we have no plans to work on this. So if you use casatools, you'd have to access the table also through casatools - or use a trick with multiple venvs or so.

[AlecThomson]

Thanks so much @tammojan. Really appreciate the feedback on this. I'll give the casatools.table functionality some testing to see if we can get things working.