Merge pull request #27 from llvm-ml/lp/pypi-patch

Complete the GitHub Actions & Publish Package

Author: ludgerpaehler

.github/workflows/package_publish.yml

@@ -0,0 +1,32 @@
+name: Upload Package to PyPi
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  package-and-deploy:
+    name: ${{ matrix.python_version }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python_version: ["3.9", "3.10"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: "{{ matrix.python_version }}"
+    - name: Install Poetry
+      run: |
+        python -m pip install --upgrade pip
+        pip install poetry
+        poetry config virtualenvs.in-project false
+        poetry config virtualenvs.path ~/.virtualenvs
+        poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
+    - name: Build & publish package
+        poetry publish --build

.github/workflows/poetry_build.yml

@@ -0,0 +1,25 @@
+name: Test Build of Package
+
+on: [push, pull_request]
+
+permissions:
+  contents: read
+
+jobs:
+  package-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: 3.9
+    - name: Install Poetry
+      run: |
+        python -m pip install --upgrade pip
+        pip install poetry
+        poetry config virtualenvs.in-project false
+        poetry config virtualenvs.path ~/.virtualenvs
+    - name: Build package
+      run: |
+        poetry install

README.md

@@ -5,7 +5,13 @@ multiple sources.
 
 ## Getting Started
 
-To get started with the dataset construction utilities, we'd suggest to use the packaged [pipenv](https://pipenv.pypa.io) to isolate the Python from your system isolation or other environments. To install, you then have to
+To get started with the dataset construction utilities, we'd suggest to use the
+packaged [pipenv](https://pipenv.pypa.io), or the packaged [poetry](https://python-poetry.org/)
+to isolate the Python from your system isolation or other environments.
+
+### Pipenv
+
+To get started with pipenv, you then have to
 
 ```bash
 pipenv install
@@ -29,6 +35,27 @@ In case you want to develop the package, this becomes
 pipenv shell && pip install -e .
 ```
 
+### Poetry
+
+To get started with poetry, you then have to
+
+```bash
+poetry install
+```
+
+which will draw the exact software version from the packaged lockfile, and install the editable version of the
+dataset construction utilities into the environment. To only install the dependencies, you can run
+
+```bash
+poetry install --no-root
+```
+
+To then develop inside of poetry's virtual environment, we can launch a shell with
+
+```bash
+poetry shell
+```
+
 ## Creating First Data
 
 To create your first small batch of IR data you then have to run from the root directory of the package

llvm_ir_dataset_utils/tools/upload_dataset_hf.py

@@ -53,8 +53,8 @@ def main(_):
   for language_folder in os.listdir(FLAGS.dataset_dir):
     for file_name in os.listdir(
         os.path.join(FLAGS.dataset_dir, language_folder)):
-      if FLAGS.start_after and file_to_upload <= FLAGS.start_after:
-        logging.info(f'Skipping uploading {file_to_upload}')
+      if FLAGS.start_after and file_name <= FLAGS.start_after:
+        logging.info(f'Skipping uploading {file_name}')
         continue
 
       full_file_path = os.path.join(FLAGS.dataset_dir, language_folder,