Add custom server certificate support by ehelms · Pull Request #462 · theforeman/foremanctl

ehelms · 2026-04-21T18:56:18Z

Why are you introducing these changes? (Problem description, related links)

Allow users to provide their own server certificates via --certificates-server-certificate, --certificates-server-key, and --certificates-server-ca-certificate flags on foremanctl deploy. The user must then specify the use of custom server certificates with --certificate-source=custom_server. Custom certificates are copied into the canonical /root/certificates/ structure, while client certificates and localhost certificates continue to be managed by the internal CA. The custom certificates do not need to persist on disk after the first time they are deployed.

What are the changes introduced in this pull request?

Adds new user parameters for providing server certificate, key and CA
Includes custom certificates in the test matrix

How to test this pull request

Steps to reproduce:

Perform a standard setup for testing and stop before foremanctl deploy
Run ./forge custom-certs to generate custom certs on the host
Deploy with custom certs: ./foremanctl --certificate-source=custom_server --certificate-server-certificate /root/custom-certificates/certs/quadlet.example.com.crt --certificate-server-key /root/custom-certificates/private/quadlet.example.com.key --certificate-server-ca-certificate /root/custom-certificates/certs/server-ca.crt

Checklist

Tests added/updated (if applicable)
Documentation updated (if applicable)

evgeni · 2026-04-22T08:25:11Z

@ehelms I tried to make minimal changes to the tests so they match the docs (no "custom" source, just "default" with additional params) and picked the one change I had in #421 for "don't re-gen the server cert", please have a look if you agree

ekohl

Zooming out: what is the server CA certificate? I know we're in a messy place, but can we at least better define this?

Today we really mix a few concerns so I'm going to start with what certbot creates because I think it's a really great setup. For every certificate it creates files /etc/letsencrypt/live/$domain:

cert.pem only the issued certificate
chain.pem the intermediate certificates between the issued certificate and the root
fullchain.pem the issued certificate AND the intermediate certificates (the same as cert.pem and chain.pem combined)
privkey.pem the private key

Note it doesn't write the actual root CA anywhere and it assumes that's already in the trust stores somewhere.

You can configure Apache in 2 ways correctly serve to clients. What we do today:

SSLCertificateFile /etc/letsencrypt/live/$domain/cert.pem
SSLCertificateChainFile /etc/letsencrypt/live/$domain/chain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/$domain/privkey.pem

But quoting https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslcertificatechainfile

SSLCertificateChainFile became obsolete with version 2.4.8, when SSLCertificateFile was extended to also load intermediate CA certificates from the server certificate file.

These days it's recommended to do:

SSLCertificateFile /etc/letsencrypt/live/$domain/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/$domain/privkey.pem

Note that if you use multiple algorithms (typically RSA, DSA, ECC, but with PQC also ML-DSA) you need to repeat the SSLCertificateFile statement. How it exactly matches certs and keys is unclear to me. This also implies the foremanctl --tls-server-certificate parameters needs to be repeatable.

In Foreman we also need the actual root CA because we want to distribute that to clients. It's also used to only allow certificates signed by a specific CA to do client authentication. If you use a third party CA that is already well known (like Let's Encrypt) then is there really a point to specifying it? Right now our tooling expects it so I can accept we'll do that for now.

As we're moving to a whole new installer, I think this is the moment to properly define the inputs and what we expect. As we're also looking at PQC then I think the fullchain.pem approach is really the only sane approach but not 100% sure all of our tooling supports that today.

evgeni · 2026-04-22T09:16:07Z

We only use the server_ca for Apache as a "server" thing, so that part should work.
We also pass it to Foreman Proxy, and Hammer to validate the connection to the server, but there it's already used as a "CA file" which can contain multiple certs, and one more cert in that file won't hurt.

ehelms · 2026-04-24T13:16:00Z

As we're moving to a whole new installer, I think this is the moment to properly define the inputs and what we expect. As we're also looking at PQC then I think the fullchain.pem approach is really the only sane approach but not 100% sure all of our tooling supports that today.

Yea, we'd have to audit and potentially update any component to work this same way that uses public facing certificates. And we'd be putting the onus on the user to concatenate their CA and server certificate together. They are capable of this, but we'd have to do error checking and what not to ensure the user does it correctly. I'm hesitant to go that route, at least right now. Especially with not having resolved our installer certificate strategy. I am hopeful #421 is the right approach and that would allow us to then consider this route.

ekohl · 2026-04-24T15:04:42Z

I'm wondering what we should do. I'd like to have a GA quality with Foreman 3.19, but we can "migrate" in Foreman 3.20. But at that point I'd really want to have finalized the meaning of parameters for at least some time.

Looking at https://community.theforeman.org/t/foreman-3-19-schedule-and-planning/45747 we have prep week next week so need to decide soon. Not saying we can't merge this in 3.19, but then we also need to plan follow up issues.

ehelms · 2026-04-24T16:07:17Z

I'm wondering what we should do. I'd like to have a GA quality with Foreman 3.19, but we can "migrate" in Foreman 3.20. But at that point I'd really want to have finalized the meaning of parameters for at least some time.

Does GA quality meaning supporting upgrades from release to release?

ekohl · 2026-04-24T16:25:29Z

IMHO yes

ShimShtein

I would prefer generating the bundle internally, and not exposing it to the user.
Something along the lines of this pseudocode:

ca_sources = ([client_ca_path] + [server_ca_path]).unique

ca_bundle = File.combine(ca_sources)

Then you will just need to set the server_ca parameter to whatever value you want - it can be internally generated, installer generated or custom.

ShimShtein · 2026-04-27T10:47:07Z

    choices:
      - default
      - installer
+      - custom_server


Do we really need this option explicitly? Can't we use the other two options, and if certificates_custom_server* options are specified, they will take precedence over the default paths specified by certificate-source?
This will also address my concern about combining installer client certs with custom server ones.

I do, for this design to work where the user does not have to persist the certificates used as input on disk. We need some indicator of the mode they are operating in. I'll update the PR description as this design has deviated since I first opened it.

evgeni · 2026-04-28T11:25:36Z

Welp, this fails with No smart proxy server found on ["quadlet.example.com"] and is not in trusted_hosts as the proxy has no Report-enabled features 🙈

evgeni · 2026-04-28T13:14:07Z

needs a rebase after I just merged #456

ShimShtein · 2026-04-28T16:09:55Z

-    mode: '0600'
+- name: Issue server certificate
+  when:
+    - (certificates_source != 'custom_server') or (certificates_hostname == 'localhost')


It means we will issue a new key and server cert if the hostname is localhost regardless of the certificate source. Why would you like to do it in case the user had specified certificate_source = custom_server and hostname=localhost

localhost is for internal communications: Candlepin and IoP. The user cannot supply these certificates. They can only provide custom server certificates for public facing interfaces.

evgeni · 2026-04-29T06:33:55Z

+  required_if:
+    - [certificates_source, custom_server, [certificates_custom_server_certificate, certificates_custom_server_key, certificates_custom_server_ca_certificate]]


this seems not to work, as running ./foremanctl deploy --certificate-source custom_server starts a deployment.

(also, it would explode on subsequent calls to deploy, where you don't pass the certs anymore)

theforeman/obsah#116

Good find, thanks!

This PR got released as 1.8.1, mind bumping the minimum obsah version in this PR, so we ensure we run with the right constraints

finally, its failing CI as predicted :D

Except that it failed in the wrong way. I dropped this line for now. I think this could be cleaner for the user, but this feels like a fundamental problem.

I want to guard against user input. This is failing because it's reading from parameters.yaml and treating that as part of the constraints. The problem is: with this design parameters.yaml should have only this value. But if the user supplies --certificate_source=custom_server then they should be providing those other three values.

Allow users to provide their own server certificates via --certificate-server-certificate, --certificate-server-key, and --certificate-server-ca-certificate flags on foremanctl deploy. Custom certificates are copied into the canonical /root/certificates/ structure, while client certificates and localhost certificates continue to be managed by the internal CA. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pr-processor Bot added the Not yet reviewed label Apr 21, 2026

ehelms mentioned this pull request Apr 21, 2026

Auto-detect and normalize installer certificates #421

Draft

evgeni requested changes Apr 22, 2026

View reviewed changes

Comment thread tests/conftest.py Outdated

Comment thread .github/workflows/test.yml Outdated

Comment thread docs/user/certificates.md Outdated

Comment thread docs/user/certificates.md Outdated

pr-processor Bot added Waiting on contributor and removed Not yet reviewed labels Apr 22, 2026

evgeni reviewed Apr 22, 2026

View reviewed changes

Comment thread src/playbooks/deploy/metadata.obsah.yaml Outdated

pr-processor Bot added Needs re-review and removed Waiting on contributor labels Apr 22, 2026

evgeni mentioned this pull request Apr 22, 2026

Add certs expiry params #456

Merged

2 tasks

ekohl reviewed Apr 22, 2026

View reviewed changes

Comment thread docs/user/certificates.md Outdated

pr-processor Bot removed the Needs re-review label Apr 22, 2026

shubhamsg199 reviewed Apr 22, 2026

View reviewed changes

Comment thread src/roles/certificates/tasks/ca.yml

evgeni reviewed Apr 24, 2026

View reviewed changes

Comment thread src/roles/certificates/defaults/main.yml Outdated

evgeni reviewed Apr 24, 2026

View reviewed changes

Comment thread src/playbooks/deploy/metadata.obsah.yaml Outdated

ehelms force-pushed the add-custom-certificate branch from 84b3878 to cbcc962 Compare April 24, 2026 13:00

ehelms force-pushed the add-custom-certificate branch from cbcc962 to 883e3fa Compare April 25, 2026 18:52

evgeni reviewed Apr 27, 2026

View reviewed changes

Comment thread docs/user/certificates.md Outdated

evgeni reviewed Apr 27, 2026

View reviewed changes

Comment thread tests/foreman_proxy_test.py Outdated

ShimShtein reviewed Apr 27, 2026

View reviewed changes

ehelms force-pushed the add-custom-certificate branch 3 times, most recently from 7c403c2 to 743bd11 Compare April 27, 2026 18:35

evgeni reviewed Apr 28, 2026

View reviewed changes

Comment thread tests/foreman_proxy_test.py Outdated

evgeni reviewed Apr 28, 2026

View reviewed changes

Comment thread tests/foreman_proxy_test.py Outdated

evgeni force-pushed the add-custom-certificate branch from b22e543 to 45b087d Compare April 28, 2026 10:22

ehelms force-pushed the add-custom-certificate branch 2 times, most recently from 22c2061 to 3fed8f2 Compare April 28, 2026 12:36

ehelms force-pushed the add-custom-certificate branch from 3fed8f2 to 66fb5f7 Compare April 28, 2026 13:41

evgeni reviewed Apr 28, 2026

View reviewed changes

Comment thread src/roles/certificates/tasks/ca.yml Outdated

ehelms force-pushed the add-custom-certificate branch from 66fb5f7 to ac3cc7e Compare April 28, 2026 14:02

ShimShtein reviewed Apr 28, 2026

View reviewed changes

evgeni reviewed Apr 29, 2026

View reviewed changes

Comment thread src/playbooks/deploy/metadata.obsah.yaml Outdated

ehelms force-pushed the add-custom-certificate branch 3 times, most recently from 512f1ff to 9672ace Compare April 30, 2026 15:39

ehelms force-pushed the add-custom-certificate branch from 9672ace to e3e5e8c Compare May 1, 2026 13:25

		required_if:
		- [certificates_source, custom_server, [certificates_custom_server_certificate, certificates_custom_server_key, certificates_custom_server_ca_certificate]]

Conversation

ehelms commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are you introducing these changes? (Problem description, related links)

What are the changes introduced in this pull request?

How to test this pull request

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evgeni commented Apr 22, 2026

Uh oh!

ekohl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

evgeni commented Apr 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ehelms commented Apr 24, 2026

Uh oh!

ekohl commented Apr 24, 2026

Uh oh!

ehelms commented Apr 24, 2026

Uh oh!

ekohl commented Apr 24, 2026

Uh oh!

Uh oh!

Uh oh!

ShimShtein left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

evgeni commented Apr 28, 2026

Uh oh!

evgeni commented Apr 28, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ehelms commented Apr 21, 2026 •

edited

Loading