bash: don't apply -o filenames post-processing to non-file completions

[ThomasWaldmann]

Fixes #67.

Problem

The bash script is registered with complete -o filenames -F ..., which makes readline apply filename post-processing to every candidate in COMPREPLY. When a candidate happens to match an existing directory in the CWD, readline appends a trailing / and suppresses the trailing space. So a subcommand build colliding with a ./build directory completes as build/ even though the completion function correctly returns build.

Fix

Register without the global -o filenames and enable it per-invocation via compopt -o filenames only in the branches that actually complete paths:

• the redirection-operator fallback (compgen -f)
• action COMPGENs (shtab.FILE/shtab.DIR and custom compgens)

This preserves the wanted behavior for file/dir completions (trailing / on directories so recursion into subdirs works, escaping of special characters) while plain choices/subcommands/option strings no longer get filename treatment.

compopt exists since bash 4.0; the call is guarded with 2>/dev/null || : so it is a no-op on older bash (degrading to no filename post-processing rather than breaking) and doesn't abort scripts run under set -e (the existing redirection test executes the function non-interactively with bash -e, where compopt fails).

Note: custom user-supplied compgens now also get filenames processing, but they already did (the flag was global before), so this is strictly narrower.

Trade-off on bash < 4.0 (e.g. the macOS system bash 3.2)

Without compopt there is no way to turn filename post-processing on per invocation, so on bash 3.2 FILE/DIR completions lose it altogether rather than only the unwanted part. Measured with the test_parser fixture, bash 3.2.57 vs 5.3.15:

completing	3.2 before	3.2 after	5.3 after
myprog cre (#67)	create/	create	create
myprog create alpha sub (a dir)	subdir/	subdir	subdir/
myprog create alpha with (with space.txt)	with\ space.txt	with space.txt	with\ space.txt
myprog create alpha > pla (redirection)	plain.txt	plain.txt	plain.txt

Nothing breaks — the script sources cleanly on 3.2 and the guarded compopt is a silent no-op, also under set -e — and #67 is fixed there too. But directories no longer get a trailing / (so tabbing down into subdirs stops working), and, more annoyingly, a filename containing a space is inserted unquoted, which bash then parses as two arguments.

If that regression for bash 3.2 users is not acceptable, the registration can branch instead, leaving bash 3.2 exactly as it is today (including #67, which is unfixable there without compopt) and fixing 4.0+:

if type compopt &>/dev/null; then
  complete -F ${root_prefix} ${prog}
else # bash < 4.0: no compopt, keep the old global behaviour
  complete -o filenames -F ${root_prefix} ${prog}
fi

Happy to switch to that if preferred.

Test

test_bash_subcommand_dir_collision drives an interactive bash through a pty (modeled on the existing tcsh_candidates helper), since readline's post-processing happens after COMPREPLY and is invisible to compgen-based tests. With directories create/ and subdir/ present, using the existing test_parser fixture:

• myprog cre<TAB> -> myprog create (was: myprog create/)
• myprog create alpha sub<TAB> -> myprog create alpha subdir/ (slash on dirs still works)

Verified the test fails on main without the shtab/__init__.py change. Existing assertions on the registration line were updated (complete -o filenames -F _shtab_shtab ... -> complete -F _shtab_shtab ...).

Review carefully

Made by Claude Opus 5.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.20%. Comparing base (f16f4b4) to head (8abfb9b).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #250   +/-   ##
=======================================
  Coverage   85.20%   85.20%           
=======================================
  Files           3        3           
  Lines         473      473           
  Branches       93       93           
=======================================
  Hits          403      403           
  Misses         39       39           
  Partials       31       31           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ThomasWaldmann]

Review of tqdm/shtab PR #250

I've reviewed the PR and tested the changes against the codebase. The PR does a good job of addressing the main issue for subcommands, but there is an edge case with custom completions that still suffers from the trailing slash bug.

1. What the PR does well

The PR successfully fixes issue #67 for subcommands and static option choices. By removing the global complete -o filenames flag and selectively applying compopt -o filenames when needed, it correctly prevents readline from appending a trailing slash when a subcommand matches a directory name.

2. The Edge Case (Bug in the PR)

The PR applies compopt -o filenames right before executing any current_action_compgen.

    [ -n "${current_action_compgen}" ] && {
      compopt -o filenames 2>/dev/null || : # bash >= 4.0; no-op outside completion
      while IFS= read -r line; do COMPREPLY+=("$line"); done < <(
        "${current_action_compgen}" "${completing_word}")
    }

While this correctly applies filename post-processing to shtab.FILE, shtab.DIRECTORY, and shtab.glob(...), it incorrectly applies it to shtab.cmd(...) and any other custom non-file compgens.

For example, shtab.cmd("git branch") is used to complete branch names. If master is a branch and there happens to be a directory named master/ in the current working directory, bash will still append a trailing slash (e.g. git checkout master/).

3. Why this happens

The obvious fix would be to move compopt -o filenames inside the specific _shtab_compgen_files function. However, because shtab executes current_action_compgen in a process substitution subshell (< <(...)), any compopt calls made inside the compgen function are lost and never reach the main shell or readline. This is why the PR author placed it outside the subshell, applying it broadly.

4. Suggested Solution

To fix this completely, we need to selectively run compopt -o filenames only for compgens that actually produce filenames.

We can do this by distinguishing the bash functions by name:

1. In shtab/init.py, change glob(...) to generate bash functions named _shtab_glob_compgen_* and cmd(...) to generate _shtab_cmd_compgen_* (currently, both use _shtab_pattern_compgen_*).
2. Update the bash template to conditionally apply compopt based on the function name:

   [ -n "${current_action_compgen}" ] && {
     if [[ "${current_action_compgen}" == "_shtab_compgen_files" || \
           "${current_action_compgen}" == "_shtab_compgen_dirs" || \
           "${current_action_compgen}" == _shtab_glob_compgen_* ]]; then
       compopt -o filenames 2>/dev/null || :
     fi
     while IFS= read -r line; do COMPREPLY+=("$line"); done < <(
       "${current_action_compgen}" "${completing_word}")
   }

This approach maintains the fix for subcommands, supports bash 4.0+ filename completion where appropriate, and perfectly fixes the trailing slash issue for custom non-file completions like shtab.cmd("git branch").

^ Review by Gemini 3.1 Pro.

[ThomasWaldmann]

Good catch — fixed in c2e92b4.

compopt -o filenames is now gated on the compgen function name (a case on _shtab_compgen_files|_shtab_compgen_dirs|_shtab_glob_compgen_*), and the glob()/cmd() generated functions are renamed from the shared _shtab_pattern_compgen_* to _shtab_glob_compgen_* / _shtab_cmd_compgen_* so the two are distinguishable, as suggested.

Verified with a new pty test (test_bash_compgen_dir_collision): with a master/ directory present, a shtab.cmd("echo master other") positional now completes mas<TAB> -> master (was master/ both before this PR and with the previous commit), while a shtab.glob("*.yml") option still completes sub<TAB> -> subdir/. The test fails on the previous commit and on main.

One behavior note: fully custom .complete = {"bash": "my_func", ...} compgens don't match the allowlist, so they no longer get filename post-processing (before this PR they did, via the global flag). Candidates from arbitrary user functions aren't necessarily paths, so opt-out seems like the right default — but it is a behavior change for custom compgens that do complete paths.

^ By Claude Fable 5.

[ThomasWaldmann]

Re-Review of PR #250

Thank you for implementing the suggested changes so quickly!

I have reviewed the updated PR and run the tests locally. The changes look excellent:

1. Renaming the generated compgen functions cleanly distinguishes glob (path-completing) from cmd (non-path-completing).
2. The case statement correctly gates the compopt -o filenames invocation so that it only applies to _shtab_compgen_files, _shtab_compgen_dirs, and _shtab_glob_compgen_*.
3. The new test_bash_compgen_dir_collision perfectly verifies this exact edge-case.

This successfully prevents non-path candidates from acquiring a trailing slash when matching a directory name, without regressing behavior for actual paths and subcommands.

This looks fully ready to go!

^ By Gemini.

[casperdcl]

ideally these two bash_completed_lines should be combined (similar to tcsh_candidates) to avoid multiple pty.forks

[casperdcl]

any ideas on how to work around these notes?

also FYI shtab.cmd("ls -1 *.txt 2>/dev/null") doesn't work on tcsh because the literal / in the command seems to confuse the logic