diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 7b82e31b..3990ac3a 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -25,3 +25,4 @@ Check out the following pages for conceptual information on ADBC fundamentals: There's also a concept page for dbc's driver list functionality: - [Driver List](./driver_list.md) +- [Driver Registry](./driver_registry.md) diff --git a/docs/guides/continuous_integration.md b/docs/guides/continuous_integration.md new file mode 100644 index 00000000..6732df65 --- /dev/null +++ b/docs/guides/continuous_integration.md @@ -0,0 +1,92 @@ + + +# Continuous Integration + +dbc works well in non-interactive environments such as on continuous integration (CI) platforms. You may also want to read through our [Version Control](./version_control.md) guide as these two concepts are related. + +## GitHub Actions + +We recommend using the [columnar-tech/setup-dbc](https://github.com/columnar-tech/setup-dbc) action if you're using [GitHub Actions](https://docs.github.com/en/actions) for CI. + +As an example, here's a workflow that automatically installs all drivers listed in your [driver list](../concepts/driver_list.md) before running your tests: + +```yaml +name: Test +on: [push] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + + # Note: Automatically installs drivers specified in dbc.toml + - uses: columnar-tech/setup-dbc@v1 + + - name: Run tests + run: pytest ... +``` + +See the [columnar-tech/setup-dbc README](https://github.com/columnar-tech/setup-dbc) for usage information and more examples. + +## Other CI Systems + +To use dbc with other CI systems, we recommend using our command line installers because they will always install the latest version of dbc for whatever platform you run them on. + +As an example for you to adapt to your system, here's a GitHub Actions workflow that installs and makes dbc available without using [columnar-tech/setup-dbc](https://github.com/columnar-tech/setup-dbc): + +{% raw %} +```yaml +name: Test +on: [push] + +jobs: + test: + runs-on: ${{ matrix.os }} + + strategy: + matrix: + os: [ubuntu-latest, windows-latest, macos-latest] + + steps: + - uses: actions/checkout@v6 + + - name: Install dbc (Linux, macOS) + if: runner.os != 'Windows' + run: | + curl -LsSf https://dbc.columnar.tech/install.sh | sh + + - name: Install dbc (Windows) + if: runner.os == 'Windows' + run: | + powershell -ExecutionPolicy ByPass -c "irm https://dbc.columnar.tech/install.ps1 | iex" + + - name: Add dbc to PATH (Linux, macOS) + if: runner.os != 'Windows' + shell: bash + run: echo "$HOME/.local/bin" >> $GITHUB_PATH + + - name: Add dbc to PATH (Windows) + if: runner.os == 'Windows' + shell: pwsh + run: | + Join-Path $env:USERPROFILE ".local\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append + + - name: Run tests + run: pytest ... +``` +{% endraw %} diff --git a/docs/guides/index.md b/docs/guides/index.md index 633d5ba4..fe04a871 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -23,3 +23,6 @@ For detailed guides on using dbc, see the following pages: - [Using a Driver List](./driver_list.md) - [Installing a Driver Manager](./driver_manager.md) - [Python Notebooks](./python_notebooks.md) +- [Private Drivers](./private_drivers.md) +- [Continuous Integration](./continuous_integration.md) +- [Version Control](./version_control.md) diff --git a/docs/guides/installing.md b/docs/guides/installing.md index cf18d7da..a0e30249 100644 --- a/docs/guides/installing.md +++ b/docs/guides/installing.md @@ -270,6 +270,10 @@ Installed some_driver 1.0.0 to /Users/user/Library/Application Support/ADBC/Driv Make note of the name "some_driver" printed above as this will be the name to use when loading the driver with a [Driver Manager](../concepts/driver_manager.md). i.e., `dbapi.connect(driver="some_driver")`. +## GitHub Actions + +To use dbc with [GitHub Actions](https://docs.github.com/en/actions), we recommend the official [`columnar-tech/setup-dbc`](https://github.com/columnar-tech/setup-dbc) action. See the [Continuous Integration](./continuous_integration.md) guide for more detail and examples. + ## Uninstalling Drivers You can uninstall a driver with the `dbc uninstall` subcommand. diff --git a/docs/guides/version_control.md b/docs/guides/version_control.md new file mode 100644 index 00000000..7aae2b95 --- /dev/null +++ b/docs/guides/version_control.md @@ -0,0 +1,68 @@ + + +# Version Control + +When using dbc in projects where version control software such as [git](https://git-scm.com) is being used, we recommend the following: + +- Use a [driver list](../concepts/driver_list.md) to record drivers and their version constraints instead of installing drivers manually with [`dbc install`](./installing.md) +- Track `dbc.toml` with version control and always use `dbc sync` to install drivers after checkout +- To maximize reproducibility, also track [`dbc.lock`](./driver_list.md#lockfile) +- Don't track installed driver directories with version control, use `dbc.toml` instead + +## Example Workflow + +To help illustrate how this works in practice, see the example below for how to use dbc when collaborating with git. This assumes both developers have already [installed](../getting_started/installation.md) dbc. + +Developer 1 sets up dbc with the drivers their project needs: + +```console +# Create a driver list file +$ dbc init + +# Add the mysql and sqlite drivers to it. Constrain sqlite's version. +$ dbc add mysql "sqlite<2" +added mysql to driver list +added sqlite to driver list with constraint <2 +use `dbc sync` to install the drivers in the list + +# Install the drivers from dbc.toml +$ dbc sync +✓ mysql-0.1.0 +✓ sqlite-1.11.0 +Done! + +# Start tracking dbc.toml with git +$ git add dbc.toml + +# Commit and push +$ git commit -m "Create dbc.toml" +$ git push +``` + +Developer 2 then clones the repository and uses `dbc sync`: + +```console +$ git clone example/repo + +# Install the drivers from dbc.toml +$ dbc sync +✓ mysql-0.1.0 +✓ sqlite-1.11.0 +Done! +``` + +Now, at this point, both Developer 1 and Developer 2 have the same set of drivers available on their systems. diff --git a/docs/index.md b/docs/index.md index a1831f49..d83b9e6f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -923,7 +923,7 @@ dbc is the command-line tool for installing and managing [ADBC](https://arrow.ap - Create reproducible environments with [driver list](concepts/driver_list.md) files - Cross-platform: Runs on macOS, Linux, and Windows - Installable with pip, Docker, and more (See [Installation](./getting_started/installation.md)) -- Works great in CI/CD environments +- Works great in CI/CD environments (See [Continuous Integration](./guides/continuous_integration.md)) ## Help diff --git a/docs/reference/index.md b/docs/reference/index.md index 158dcc25..2f8dd764 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -20,3 +20,4 @@ limitations under the License. - [Config level](./config_level.md) - [Driver List](./driver_list.md) - [Supported Platforms](./supported_platforms.md) +- [Analytics](./analytics.md) diff --git a/mkdocs.yml b/mkdocs.yml index 3689ae66..5483dc00 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -47,6 +47,8 @@ nav: - Driver Managers: guides/driver_manager.md - Python Notebooks: guides/python_notebooks.md - Private Drivers: guides/private_drivers.md + - Continuous Integration: guides/continuous_integration.md + - Version Control: guides/version_control.md - Concepts: - concepts/index.md - Driver: concepts/driver.md @@ -147,6 +149,9 @@ plugins: - guides/driver_list.md: Using driver lists for reproducible setups - guides/driver_manager.md: Working with driver managers - guides/python_notebooks.md: Using dbc in Python notebooks + - guides/private_drivers.md: Using private drivers + - guides/continuous_integration.md: Using dbc in CI/CD pipelines + - guides/version_control.md: Using dbc with version control Concepts: - concepts/driver.md: What is an ADBC driver - concepts/driver_manager.md: Understanding driver managers