Add generic plugin test runner script by ehelms · Pull Request #10963 · theforeman/foreman

ehelms · 2026-04-27T12:57:23Z

Plugins like Katello and foreman_puppet have their own wrapper scripts or Rakefile customizations to make it easier to run individual test files without manually switching to the Foreman directory and setting up cross-project load paths. This script generalizes that pattern for all plugins.

Supports running from the Foreman checkout, a plugin checkout, or anywhere on PATH with automatic Foreman/plugin discovery. Single-file runs use direct ruby invocation for faster feedback; full-suite runs delegate to the plugin's existing rake tasks.

I am submitting this as a replacement for ktest to be available for any plugin, and support different paths of usage. And instead of this living inside our development deployment tooling, to live as a script in the main repository and thus allow it's usage from more places.

In the use case of foremanctl, the development deployer would copy the script to the users local bin directory to make it broadly available within the environment.

adamruzicka · 2026-04-27T13:07:55Z

Every now and then, I need to run an ad-hoc plugin rake task (usually foreman_theme_satellite:validate_docs), could this be extended to be able to run arbitrary rake tasks or would that be out of scope?

Plugins like Katello and foreman_puppet have their own wrapper scripts or Rakefile customizations to make it easier to run individual test files without manually switching to the Foreman directory and setting up cross-project load paths. This script generalizes that pattern for all plugins. Supports running from the Foreman checkout, a plugin checkout, or anywhere on PATH with automatic Foreman/plugin discovery. Single-file runs use direct ruby invocation for faster feedback; full-suite runs delegate to the plugin's existing rake tasks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ehelms · 2026-04-27T13:26:04Z

Every now and then, I need to run an ad-hoc plugin rake task (usually foreman_theme_satellite:validate_docs), could this be extended to be able to run arbitrary rake tasks or would that be out of scope?

I think that's fair. Do you think we can count on the pattern always including : to differentiate rake tasks from test files? Or do you think it will need an option like --rake ?

adamruzicka · 2026-04-27T13:43:05Z

Or do you think it will need an option like --rake ?

I'd prefer this to also cover cases like string extraction where the plugin-name-prefix is not there (it is plugin:gettext[$plugin])

adamruzicka · 2026-04-28T08:58:50Z

+  gemspec=$(compgen -G "$dir"/*.gemspec | head -1) || true
+  if [[ -n "$gemspec" ]]; then
+    basename "$gemspec" .gemspec
+  fi


Here we'll run into foreman-tasks being silly with spaces and underscores. The gemspec is foreman-tasks.gemspec, but the rake task prefix is foreman_tasks.

adamruzicka · 2026-04-28T08:59:44Z

+Usage: plugin-test [PLUGIN_NAME] [TEST_FILES|RAKE_TASKS...] [OPTIONS...]
+
+Run Foreman plugin tests or rake tasks from any directory.
+
+Arguments:
+  PLUGIN_NAME   Plugin gem name (e.g. katello, foreman_puppet).
+                Optional when running from a plugin checkout or
+                when test file paths include the plugin directory.
+  TEST_FILES    One or more test files to run. Paths are resolved
+                relative to the current directory or the plugin root.
+  RAKE_TASKS    One or more rake tasks to run. Any argument containing
+                ":" is treated as a rake task (e.g. katello:reindex).
+  OPTIONS       Extra arguments passed through to the test runner
+                (e.g. --name test_create).
+
+Examples:
+  plugin-test katello                                     # full suite via rake
+  plugin-test katello test/models/repo_test.rb            # single file
+  plugin-test test/models/repo_test.rb                    # from plugin checkout
+  plugin-test ../katello/test/models/repo_test.rb         # direct path from foreman
+  plugin-test test/models/repo_test.rb --name test_create
+  plugin-test foreman_theme_satellite:validate_docs       # run a rake task


Could we s/plugin-test/\$0 everywhere so the script can be freely renamed? Alternatively, use a more descriptive name like foreman-plugin-test?

foreman-plugin-test sounds fine. Be great if we had some prefix that wasn't "foreman" since a hundred things share that bash-completion and name 😁

adamruzicka · 2026-04-28T09:00:45Z

+  local dir="$1"
+  for candidate in "$dir" "$dir/foreman" "$dir/../foreman" "$dir/../../foreman"; do
+    if [[ -f "$candidate/config/application.rb" ]]; then
+      (cd "$candidate" && pwd)


Personal preference, feel free to ignore

Suggested change

(cd "$candidate" && pwd)

readlink -f "$candidate"

adamruzicka · 2026-04-28T09:01:24Z

+detect_plugin_name() {
+  local dir="$1"
+  local gemspec
+  gemspec=$(compgen -G "$dir"/*.gemspec | head -1) || true


Personal preference, feel free to ignore
I had to look up what compgen does, wouldn't this work as well?

Suggested change

gemspec=$(compgen -G "$dir"/*.gemspec | head -1) || true

gemspec=$(ls -1 "$dir"/*.gemspec | head -1) || true

adamruzicka · 2026-04-28T09:03:55Z

+  local depth=0
+  while [[ "$dir" != "/" ]] && [[ $depth -lt 10 ]]; do
+    if compgen -G "$dir"/*.gemspec &>/dev/null; then
+      echo "$dir"
+      return 0
+    fi
+    dir=$(dirname "$dir")
+    depth=$((depth + 1))
+  done


Suggested change

local depth=0

while [[ "$dir" != "/" ]] && [[ $depth -lt 10 ]]; do

if compgen -G "$dir"/*.gemspec &>/dev/null; then

echo "$dir"

return 0

fi

dir=$(dirname "$dir")

depth=$((depth + 1))

done

git -C "$dir" rev-parse --show-toplevel 2>/dev/null

adamruzicka · 2026-04-28T09:09:53Z

+  INCLUDE_PATHS="lib:test"
+  if [[ -n "$PLUGIN_PATH" ]]; then
+    [[ -d "$PLUGIN_PATH/test" ]] && INCLUDE_PATHS="$INCLUDE_PATHS:$PLUGIN_PATH/test"
+    [[ -d "$PLUGIN_PATH/spec" ]] && INCLUDE_PATHS="$INCLUDE_PATHS:$PLUGIN_PATH/spec"
+  fi
+
+  RAKE_PATH=$(bundle info rake --path) || {
+    echo "Error: Could not find rake in the bundle." >&2
+    exit 1
+  }
+
+  exec bundle exec ruby \
+    -I"$INCLUDE_PATHS" \
+    -I"${RAKE_PATH}/lib" \
+    "${TEST_FILES[@]}" \
+    "${OTHER_OPTS[@]}"


I can't say I'm exactly happy about inventing yet another way of running the tests by modifying the include path. Considering we're limited to a single plugin, wouldn't this sort of work

exec bundle exec rake test:"$PLUGIN_NAME" "${OTHER_OPTS[@]}" TEST="$TEST_FILES[@]"

I'm a bit rusty with bashisms, so this might need some tweaking, but the point should be clear.

I don't follow what the issue is, or what you are requesting a change for. Is it the use of ruby vs rake ?

Test rake tasks can declare additional paths to be put on the load path. By invoking plugin tests without rake, we might be missing those additional paths. It is an edge case, true, but still.

Those can fall back to the supply a rake task method. The original ktest tried to solve two problems:

make it easy to run Katello test's from it's home directory

speed up execution

The use of rake loads more of the environment than is needed. One option I have not explored is to try to have a special Rakefile for testing that's as fast as running ruby directly. Similar to how Rakefile.dist is much quicker for the tasks it handles because it avoids loading all of Rails.

Those can fall back to the supply a rake task method.

Fair

One option I have not explored is to try to have a special Rakefile for testing

That's tempting

The syntax (if it works) would be:

rake -f Rakefile.test test:single TEST=path/to/test.rb PLUGIN_PATH=/path/to/plugin

vs. this script:

# 1. From the Foreman checkout: plugin-test katello test/models/foo_test.rb # 2. From a plugin checkout: plugin-test test/models/foo_test.rb # 3. From anywhere (on PATH): plugin-test katello test/models/foo_test.rb # 4. With direct paths: plugin-test ../katello/test/models/foo_test.rb

When you put it like that, the script feels more ergonomic

pr-processor Bot added Not yet reviewed Waiting on contributor labels Apr 27, 2026

ehelms mentioned this pull request Apr 27, 2026

ktest not present for running Katello tests theforeman/foremanctl#474

Closed

ehelms force-pushed the add-plugin-test-script branch from 7717fdf to 00ebd76 Compare April 27, 2026 14:23

adamruzicka reviewed Apr 28, 2026

View reviewed changes

pr-processor Bot removed the Not yet reviewed label Apr 28, 2026

	gemspec=$(compgen -G "$dir"/*.gemspec \| head -1) \|\| true
	gemspec=$(ls -1 "$dir"/*.gemspec \| head -1) \|\| true

Conversation

ehelms commented Apr 27, 2026

Uh oh!

adamruzicka commented Apr 27, 2026

Uh oh!

ehelms commented Apr 27, 2026

Uh oh!

adamruzicka commented Apr 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants