Upgrade dependency package constraint for pyarrow

[nazmallik]

Is your feature request related to a problem? Please describe.
I am unable to install arcgis via my work's artifactory due to hitting a HTTP error 403 error when trying to install dependency pyarrow<17,>=16. It seems the only pyarrow versions fitting that constraint is pyarrow==16.0 and pyarrow==16.1, both flagging a security vulnerability.

I believe this is the dependency list for the most recent arcgis version.

dependencies = ['pillow',
 'urllib3<3,>=2.1.0',
 'cachetools',
 'lxml',
 'cryptography',
 'pandas<2.3.0,>=2.0.0',
 'numpy<2,>=1.21.6',
 'matplotlib',
 'keyring>=23.3.0',
 'pylerc',
 'ujson>=3',
 'truststore>=0.10.0',
 'pywin32>=223; platform_system == "Windows"',
 'geomet',
 'requests<3,>=2.32.3',
 'requests-oauthlib',
 'requests_toolbelt',
 'pyspnego>=0.8.0',
 'dask[dataframe]<2025.1.0,>=2024.12.1',
 'matplotlib-inline',
 'pyarrow<17,>=16',
 'puremagic<2,>=1.15',
 'pydantic<3,>=2.8.2',
 'networkx<4,>=3.3',
 'websocket-client<2.0.0,>=1.2.3',
 'dill; extra == "gp"',
 'gdal<4,>=3.9.2; extra == "gdal"',
 'requests-kerberos; extra == "kerberos"',
 'requests-gssapi; extra == "kerberos"']

Describe the solution you'd like
Could we please relax the pyarrow constraint to also include versions of pyarrow>=17?

Describe alternatives you've considered
I have tried to downgrade my arcgis version to 2.4.0 so that I use pyarrow<=15 but then other dependencies (such as pandas) are downgraded to versions released almost two years ago. Unfortunately my code at work relies on pandas>=2.2.2 which is incompatible with earlier arcgis versions.

I believe the most effective fix would be to update the dependencies so the most up-to-date packages are used by arcgis.

Additional context
This was the weakness in pyarrow==16.1 identified during our vulnerability checks: https://cwe.mitre.org/data/definitions/502.html

[achapkowski]

For this issue you are referring to, the issue occurs from version 14 and below:
https://security.snyk.io/vuln/SNYK-PYTHON-PYARROW-6052811

I would provide this information to your security expert and inform them that we are using a version that falls outside of the CVE version range.

The team does not want to relax the version, we would be actually introducing the issue that you are concerned about.

Also check out: https://security.snyk.io/package/pip/pyarrow

[nazmallik]

Thanks for the quick reply! I will follow up with our security expert but may I ask why not allow pyarrow>=17 too in that case?

Update:
A colleague and I did some digging and we learned that our artifactory could be flagging the apache arrow repo vulnerable to CVE-2024-52338 per
https://lists.apache.org/thread/0rcbvj1gdp15lvm23zm601tjpq0k25vt and
https://nvd.nist.gov/vuln/detail/CVE-2024-52338
Due to the Apache Arrow R versions 4-16 containing vulnerabilities.

This does not influence the pyarrow 14+ packages themselves so I will reach out to our security experts to unblock pyarrow==16.1. I still request if we could please set the dependency to pyarrow>=16 (as opposed to the current pyarrow<17,>=16)