[SPARK-56115] Support crds.install in Helm Chart

[TQJADE]

What changes were proposed in this pull request?

This PR introduces a crds.install configuration option to the Helm chart. CRD files are moved from the crds/ directory
(unconditionally installed by Helm) to templates/crds/ with a {{- if .Values.crds.install }} guard, allowing users to
disable CRD installation by setting crds.install: false. Similar to:
https://github.com/argoproj/argo-helm/blob/f7aee451eb788abd682d187fb055668bbfe28091/charts/argo-cd/values.yaml#L31-L40

Why are the changes needed?

In environments where CRDs are managed separately, users need the ability to skip CRD installation at the chart level. Previously the only option was the helm install --skip-crds flag, which is not declarative and cannot be expressed in a values file.

Does this PR introduce any user-facing change?

Yes.

• crds.install (default: true) — Controls whether CRDs are installed with the chart. Set to false when CRDs are managed externally or when deploying a second operator instance.
• crds.annotations (default: {"helm.sh/resource-policy": "keep"}) — Annotations applied to CRD resources. The default prevents CRDs from being deleted on helm uninstall.
• Migration note: Users running multiple operator releases in different namespaces must set crds.install: false on
  releases.

How was this patch tested?

helm template --set crds.install=true renders both CRDs
helm template --set crds.install=false renders no CRDs                                                                    

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.7

[dongjoon-hyun]

I'm not sure if this PR is safe because templates/crds seems to be not protected during uninstallation. Could you double-check the CRD remains in the system during helm uninstall operation like before?

Does this PR introduce any user-facing change?

No

[dongjoon-hyun]

Is this safe?

[dongjoon-hyun]

Why do we need to change spark-operator-api/build.gradle in this PR which adds crds.install?

[TQJADE]

I'm not sure if this PR is safe because templates/crds seems to be not protected during uninstallation. Could you double-check the CRD remains in the system during helm uninstall operation like before?

Does this PR introduce any user-facing change?

No

Thanks for catching this! The move from crds/ to templates/crds/ did lose Helm's built-in uninstall protection. I've addressed this by adding a configurable crds.annotations value, defaulting to "helm.sh/resource-policy": keep, which tells Helm to skip deleting CRDs on uninstall.

[TQJADE]

Why do we need to change spark-operator-api/build.gradle in this PR which adds crds.install?

1. The compared base file path change: The CRD files were moved from crds/ to templates/crds/
2. Crd contents contains the {{- if .Values.crds.install }} etc, yq cmd will failed with sed

[dongjoon-hyun]

Could you check the CI failures because it fails 3 times consistently in our CI? Does it pass locally?

[TQJADE]

Could you check the CI failures because it fails 3 times consistently in our CI? Does it pass locally?

Fix by commit: 4bfbb74

[dongjoon-hyun]

I'm wondering if you verify helm upgrade manually, too?