Skip to content

LCORE-2882: make e2e fail on error or test failure in konflux #2190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
radofuchs wants to merge 6 commits into
base: main
Choose a base branch
from
Open

LCORE-2882: make e2e fail on error or test failure in konflux #2190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
330b66e
LCORE-2882: make e2e fail on error or test failure in konflux
Jul 21, 2026
1ac8916
disable tests temporarily
Jul 21, 2026
5d5c774
increase number of tests
Jul 22, 2026
e1362de
restore complete test list
Jul 22, 2026
ac5c9e6
print the actual value on assert
Jul 23, 2026
91bdc45
add skills mounts to konflux env
Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .tekton/integration-tests/pipeline/lightspeed-stack-integration-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,10 @@ spec:
- name: credentials
value: credentials
- name: run-e2e-tests
onError: continue
#onError: continue
# Do not use onError: continue here unless a later step like
# fail-if-any-step-failed is enabled; otherwise the PipelineRun stays green
# even when pipeline-konflux.sh / behave exit non-zero.
resources:
requests:
cpu: '1'
Expand Down
35 changes: 35 additions & 0 deletions tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,32 @@ spec:
securityContext:
seccompProfile:
type: RuntimeDefault
# Seed agent-skills fixtures (docker-compose: ./tests/e2e/skills -> /app-root/skills).
# ConfigMap cannot preserve nested skill dirs; extract tarball into emptyDir.
initContainers:
- name: seed-skills
image: busybox:latest
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
command:
- /bin/sh
- -c
- |
set -e
tar xzf /skills-seed/skills.tgz -C /skills-dest
ls -la /skills-dest
volumeMounts:
- name: skills-seed
mountPath: /skills-seed
readOnly: true
- name: skills
mountPath: /skills-dest
containers:
- name: lightspeed-stack-container
securityContext:
Expand Down Expand Up @@ -58,6 +84,10 @@ spec:
- name: config
mountPath: /app-root/lightspeed-stack.yaml
subPath: lightspeed-stack.yaml
# Agent skills E2E (skills.feature): matches docker-compose mount path
- name: skills
mountPath: /app-root/skills
readOnly: true
# MCPFileAuth E2E: must match authorization_headers paths in lightspeed-stack-mcp-file-auth.yaml (/tmp/mcp-token)
- name: mcp-file-auth-token
mountPath: /tmp/mcp-token
Expand All @@ -72,6 +102,11 @@ spec:
- name: config
configMap:
name: lightspeed-stack-config
- name: skills-seed
configMap:
name: e2e-skills
- name: skills
emptyDir: {}
- name: mcp-file-auth-token
secret:
secretName: mcp-file-auth-token
Expand Down
13 changes: 13 additions & 0 deletions tests/e2e-prow/rhoai/pipeline-konflux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,19 @@ else
log "⚠️ No kv_store.db found at $RAG_DB_PATH"
fi

# Agent skills E2E: same fixture docker-compose mounts at /app-root/skills
SKILLS_DIR="$REPO_ROOT/tests/e2e/skills"
if [ -d "$SKILLS_DIR" ]; then
tar czf /tmp/e2e-skills.tgz -C "$SKILLS_DIR" .
oc create configmap e2e-skills -n "$NAMESPACE" \
--from-file=skills.tgz=/tmp/e2e-skills.tgz \
--dry-run=client -o yaml | oc apply -f -
rm -f /tmp/e2e-skills.tgz
log "✅ e2e-skills ConfigMap created from $SKILLS_DIR"
else
log "⚠️ No skills directory at $SKILLS_DIR — skills.feature will fail"
fi


# ConfigMap for Llama Stack run-from-source (init container clones this repo @ this revision)
REPO_URL="${REPO_URL:-$(cd "$REPO_ROOT" && git config --get remote.origin.url 2>/dev/null)}"
Expand Down
13 changes: 13 additions & 0 deletions tests/e2e-prow/rhoai/pipeline.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,19 @@ else
echo "⚠️ No kv_store.db found at $RAG_DB_PATH"
fi

# Agent skills E2E: same fixture docker-compose mounts at /app-root/skills
SKILLS_DIR="$REPO_ROOT/tests/e2e/skills"
if [ -d "$SKILLS_DIR" ]; then
tar czf /tmp/e2e-skills.tgz -C "$SKILLS_DIR" .
oc create configmap e2e-skills -n "$NAMESPACE" \
--from-file=skills.tgz=/tmp/e2e-skills.tgz \
--dry-run=client -o yaml | oc apply -f -
rm -f /tmp/e2e-skills.tgz
echo "✅ e2e-skills ConfigMap created from $SKILLS_DIR"
else
echo "⚠️ No skills directory at $SKILLS_DIR — skills.feature will fail"
fi

./pipeline-services.sh

echo "--> Final wait for both lightspeed-stack-service and llama-stack-service pods..."
Expand Down
2 changes: 1 addition & 1 deletion tests/e2e/test_list.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ features/unified-mode-legacy.feature
features/unified-mode-validation.feature
features/unified-mode-migration.feature
features/unified-mode-synthesis.feature
features/skills.feature
features/skills.feature
2 changes: 1 addition & 1 deletion tests/e2e/utils/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ def validate_json_partially(actual: Any, expected: Any) -> None:
continue
assert (
matched
), f"No matching element found in list for schema item {schema_item}"
), f"No matching element found in list for schema item {schema_item}, got {actual}"

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Avoid logging the full response value in assertion failures.

actual comes from HTTP response data, so embedding the entire list in the error message can leak PII, tokens, or other sensitive fields into CI logs and produce excessively large failure output. Include a bounded, sanitized representation or only the failing index/item instead.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/utils/utils.py` at line 268, Update the assertion message in the
schema-item matching logic to avoid interpolating the full HTTP response value
from actual. Use a bounded, sanitized summary or report only the failing
index/item, while retaining enough context to identify the mismatch without
exposing sensitive data or producing oversized CI logs.

Source: Coding guidelines


else:
assert actual == expected, f"Value mismatch: expected {expected}, got {actual}"
Expand Down
Loading