Dockerfile-default-rocm split into two separate files mainly aimed at different users. #272
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| ARG BASE_IMAGE | ||
| FROM ${BASE_IMAGE} | ||
|
|
||
| # MAY NOT BE IMPORTANT ANYMORE | ||
|
Contributor
Author
There was a problem hiding this comment. I guess we can remove it and if something breaks then we add it back in. |
||
| RUN apt install rocm-libs | ||
|
|
||
|
|
||
| # THIS FIX IS FOR SAWMILL, UNCLEAR IF NECESSARY FOR GENERAL USERS | ||
| #TODO: is this necessary? | ||
|
|
||
| RUN apt remove -y openmpi ucx | ||
| #Let's remove existing /opt/ompi; and, link to our version. | ||
| RUN rm -rf /opt/ompi | ||
| RUN ln -s /container/ompi /opt | ||
| COPY dockerfile_scripts /tmp/det_dockerfile_scripts | ||
|
|
||
| # SHOULDN'T NEED TO SET SOME OF THESE VARIABLES | ||
| #USING OFI | ||
| #TODO: up until line 63 should be a separate shell script | ||
| ARG WITH_MPI=1 | ||
| ARG WITH_OFI=1 | ||
| ARG WITH_MPICH | ||
| ARG UCX_INSTALL_DIR=/container/ucx | ||
| ARG OMPI_INSTALL_DIR=/container/ompi | ||
| ARG MPICH_INSTALL_DIR=/container/mpich | ||
| ARG OFI_INSTALL_DIR=/container/ofi | ||
| ARG OMPI_WITH_CUDA=0 | ||
| ARG OMPI_WITH_ROCM=1 | ||
| RUN if [ "$WITH_MPI" = "1" ]; then /tmp/det_dockerfile_scripts/ompi_rocm.sh "$UBUNTU_VERSION" "$WITH_OFI" "$OMPI_WITH_ROCM" "$WITH_MPICH"; fi | ||
|
|
||
| # Make sure OMPI/UCX show up in the right paths | ||
| ARG VERBS_LIB_DIR=/usr/lib/libibverbs | ||
| ARG UCX_LIB_DIR=${UCX_INSTALL_DIR}/lib:${UCX_INSTALL_DIR}/lib64 | ||
| ARG UCX_PATH_DIR=${UCX_INSTALL_DIR}/bin | ||
| ARG OFI_LIB_DIR=${OFI_INSTALL_DIR}/lib:${OFI_INSTALL_DIR}/lib64 | ||
| ARG OFI_PATH_DIR=${OFI_INSTALL_DIR}/bin | ||
| ARG OMPI_LIB_DIR=${OMPI_INSTALL_DIR}/lib | ||
| ARG OMPI_PATH_DIR=${OMPI_INSTALL_DIR}/bin | ||
| ARG MPICH_LIB_DIR=${MPICH_INSTALL_DIR}/lib | ||
| ARG MPICH_PATH_DIR=${MPICH_INSTALL_DIR}/bin | ||
|
|
||
| # Set up UCX_LIBS and OFI_LIBS | ||
| ENV UCX_LIBS="${VERBS_LIB_DIR}:${UCX_LIB_DIR}:${OMPI_LIB_DIR}:" | ||
| ENV OFI_LIBS="${VERBS_LIB_DIR}:${OFI_LIB_DIR}:${MPICH_LIB_DIR}:" | ||
|
|
||
| # If WITH_OFI is true, then set EXTRA_LIBS to OFI libs, else set to empty string | ||
| ENV EXTRA_LIBS="${WITH_OFI:+${OFI_LIBS}}" | ||
|
|
||
| # If EXTRA_LIBS is empty, set to UCX libs, else leave as OFI libs | ||
| ENV EXTRA_LIBS="${EXTRA_LIBS:-${UCX_LIBS}}" | ||
|
|
||
| # But, only add them if WITH_MPI | ||
| ENV LD_LIBRARY_PATH=${WITH_MPI:+$EXTRA_LIBS}$LD_LIBRARY_PATH | ||
|
|
||
| #USING OFI | ||
| ENV PATH=${WITH_OFI:+$PATH:${WITH_MPI:+$OFI_PATH_DIR:$MPICH_PATH_DIR}} | ||
|
|
||
| #USING UCX | ||
| ENV PATH=${PATH:-$CONDA:${WITH_MPI:+$UCX_PATH_DIR:$OMPI_PATH_DIR}} | ||
|
|
||
| ENV PATH=$OMPI_PATH_DIR:$OFI_INSTALL_DIR:$PATH | ||
|
|
||
| # Enable running OMPI as root | ||
| ENV OMPI_ALLOW_RUN_AS_ROOT ${WITH_MPI:+1} | ||
| ENV OMPI_ALLOW_RUN_AS_ROOT_CONFIRM ${WITH_MPI:+1} | ||
|
|
||
|
|
||
| ARG AWS_PLUGIN_INSTALL_DIR=/container/aws | ||
| ARG WITH_AWS_TRACE | ||
| ARG INTERNAL_AWS_DS | ||
| ARG INTERNAL_AWS_PATH | ||
| ARG ROCM_DIR=/opt/rocm | ||
| ENV ROCM_DIR $ROCM_DIR | ||
| RUN if [ "$WITH_OFI" = "1" ]; then /tmp/det_dockerfile_scripts/build_aws_rocm.sh "$WITH_OFI" "$WITH_AWS_TRACE" "$WITH_MPICH"; fi | ||
| ENV LD_LIBRARY_PATH=${WITH_OFI:+$AWS_PLUGIN_INSTALL_DIR:}$LD_LIBRARY_PATH | ||
|
|
||
| # Set an entrypoint that can scrape up the host libfabric.so and then | ||
| # run the user command. This is intended to enable performant execution | ||
| # on non-IB systems that have a proprietary libfabric. | ||
|
|
||
| ARG WITH_RCCL=1 | ||
| ENV WITH_RCCL=$WITH_RCCL | ||
| ARG WITH_NFS_WORKAROUND=1 | ||
| ENV WITH_NFS_WORKAROUND=$WITH_NFS_WORKAROUND | ||
|
|
||
| RUN mkdir -p /container/bin && cp /tmp/det_dockerfile_scripts/scrape_libs.sh /container/bin | ||
| ENTRYPOINT ["/container/bin/scrape_libs.sh"] | ||
|
|
||
| RUN rm -r /tmp/* | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| ARG BASE_IMAGE | ||
| FROM ${BASE_IMAGE} | ||
| #why no highlighting? | ||
|
|
||
| ENV PYTHONUNBUFFERED=1 PYTHONFAULTHANDLER=1 PYTHONHASHSEED=0 TT=0 | ||
|
|
||
| RUN mkdir -p /var/run/sshd | ||
| RUN rm /etc/apt/sources.list.d/rocm.list | ||
| RUN pip install --upgrade pip | ||
|
|
||
| COPY dockerfile_scripts /tmp/det_dockerfile_scripts | ||
|
|
||
| RUN apt-get update && /tmp/det_dockerfile_scripts/install_deb_packages.sh | ||
| RUN python --version | ||
| #RUN /tmp/det_dockerfile_scripts/install_deb_packages.sh | ||
|
|
||
| # LIBFABRIC ISSUE | ||
| # USE CONDA FOR WORKAROUND | ||
| #TODO: MAY NOT BE A PROBLEM ANYMORE? | ||
|
|
||
| # protect this image from slurm | ||
| #ENV PATH="/opt/conda/envs/py_3.8/bin:${PATH}" | ||
| ARG CONDA="${PATH}" | ||
|
|
||
| #RUN exit 1 | ||
| # Install fixed version of FFI package for Ubuntu 20.04. | ||
| # This is done after above stuff to make sure we get right version. | ||
| RUN /tmp/det_dockerfile_scripts/install_package_fixes.sh | ||
| RUN /tmp/det_dockerfile_scripts/add_det_nobody_user.sh | ||
| RUN /tmp/det_dockerfile_scripts/install_libnss_determined.sh | ||
|
|
||
|
|
||
| RUN pip install determined && pip uninstall -y determined | ||
| RUN apt update | ||
| RUN DEBIAN_FRONTEND=noninteractive apt-get install -y cargo | ||
|
|
||
| RUN python -m pip install -r /tmp/det_dockerfile_scripts/additional-requirements-rocm.txt | ||
|
|
||
| RUN python -m pip install -r /tmp/det_dockerfile_scripts/notebook-requirements.txt && \ | ||
| jupyter labextension disable "@jupyterlab/apputils-extension:announcements" | ||
|
|
||
| ENV JUPYTER_CONFIG_DIR=/run/determined/jupyter/config | ||
| ENV JUPYTER_DATA_DIR=/run/determined/jupyter/data | ||
| ENV JUPYTER_RUNTIME_DIR=/run/determined/jupyter/runtime | ||
|
|
||
| #ENV HSA_FORCE_FINE_GRAIN_PCIE=1 #TODO: check if this is necessary | ||
|
|
||
| #RUN ldconfig #TODO: check if this is necessary | ||
| RUN echo A | ||
| #TODO: finish iterating here, preferably turn it into a shell script. | ||
| ARG DEEPSPEED_PIP | ||
| ARG DS_BUILD_CUTLASS_OPS=0 | ||
| ENV DS_BUILD_CUTLASS_OPS=0 | ||
|
|
||
| #RUN if [ -n "$DEEPSPEED_PIP" ]; then DEBIAN_FRONTEND=noninteractive apt-get install -y pdsh libaio-dev&& git clone https://github.com/ROCmSoftwarePlatform/triton.git && cd triton && git checkout triton-mlir && cd python && pip3 install ninja cmake && python setup.py install;fi | ||
| #RUN if [ -n "$DEEPSPEED_PIP" ]; then DEBIAN_FRONTEND=noninteractive apt-get install -y pdsh libaio-dev&& git clone https://github.com/ROCmSoftwarePlatform/triton.git && cd triton && git checkout triton-mlir && cd python && pip3 install ninja cmake && python setup.py install;fi | ||
| RUN if [ -n "$DEEPSPEED_PIP" ]; then DEBIAN_FRONTEND=noninteractive apt-get install -y pdsh libaio-dev&&pip3 install ninja cmake;pip3 install triton==2.3.1;fi | ||
| #RUN if [ -n "$DEEPSPEED_PIP" ]; then DEBIAN_FRONTEND=noninteractive apt-get install -y pdsh libaio-dev&& python -m pip install pydantic==1.10.11 && git clone https://github.com/microsoft/DeepSpeed.git && cd DeepSpeed && DS_BUILD_OPS=1 DS_BUILD_SPARSE_ATTN=0 DS_BUILD_EVOFORMER_ATTN=0 python3 setup.py build && python3 setup.py install && python -m deepspeed.env_report; fi | ||
| RUN if [ -n "$DEEPSPEED_PIP" ]; then DEBIAN_FRONTEND=noninteractive apt-get install -y pdsh libaio-dev&& python -m pip install pydantic==1.10.11 && git clone https://github.com/microsoft/DeepSpeed.git && cd DeepSpeed && DS_BUILD_OPS=1 DS_BUILD_RAGGED_DEVICE_OPS=0 DS_BUILD_CUTLASS_OPS=0 DS_BUILD_SPARSE_ATTN=0 DS_BUILD_EVOFORMER_ATTN=0 DS_BUILD_RANDOM_LTD=0 DS_BUILD_FUSED_ADAM=0 DS_BUILD_CCL_COMM=0 python3 setup.py build && python3 setup.py install && python -m deepspeed.env_report; fi | ||
| #RUN if [ -n "$DEEPSPEED_PIP" ]; then DEBIAN_FRONTEND=noninteractive apt-get install -y pdsh libaio-dev&& python -m pip install pydantic==1.10.11 && git clone https://github.com/ROCmSoftwarePlatform/DeepSpeed.git && cd DeepSpeed && python3 setup.py build && python3 setup.py install && python -m deepspeed.env_report; fi | ||
| RUN if [ -n "$DEEPSPEED_PIP" ]; then python -m deepspeed.env_report ; fi | ||
|
There was a problem hiding this comment. This deepspeed section definitely needs cleanup |
||
| RUN pip list | grep -i deepspeed | ||
| RUN echo "$DEEPSPEED_PIP" | ||
|
|
||
| RUN pip install tokenizers>=0.19 | ||
| RUN pip install transformers==4.43.3 | ||
| RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash | ||
| #RUN exit 1 | ||
| # MIOPEN_DEBUG_SAVE_TEMP_DIR is required to prevent | ||
| # PAD-133 | ||
| ENV MIOPEN_DEBUG_SAVE_TEMP_DIR=1 | ||
|
|
||
| CMD ["/bin/bash"] | ||
| USER root | ||
|
|
||
| RUN rm -r /tmp/* | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 0.33.2 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do we test these images?