Skip to content

Ejabberd is not able to serve the appropriate cert #4102

New issue
New issue
Closed
Closed
Ejabberd is not able to serve the appropriate cert#4102
Labels
Kind:Bug
Milestone
ejabberd 26.xx

Description

@lokesh411
lokesh411
opened on Oct 17, 2023

Environment

  • ejabberd version: 22.05
  • Erlang version: 24
  • OS: Linux (Debian)
  • Installed from: official deb/rpm

Configuration:

certfiles:
 - abc.com # given by godaddy
 - xyz.com # given by lets encrypt for *.xyz.com

Bug description

Ejabberd is not able to serve the appropriate cert if multiple certs. If I am making a request to ejabberd with r0.xyz.com as the SNI, then it serves cert given toabc.com

Activity

  1. licaon-kter commented on Oct 17, 2023

    @licaon-kter
    licaon-kter
    on Oct 17, 2023
    Contributor

    ejabberd.log has any info?

    You made sure ejabberd can access the cert files?

  2. lokesh411 commented on Oct 17, 2023

    @lokesh411
    lokesh411
    on Oct 17, 2023 via email
    Author
  3. mremond
    added
    Note:Needs more info
    on Oct 23, 2023

  4. EOUpSL93Y commented on Mar 24, 2024

    @EOUpSL93Y
    EOUpSL93Y
    on Mar 24, 2024

    May be it's related to this issue, so - i have the same problem. Debian (10-12), ejabberd (few versions from Debian repo, latest - 24.02 - official deb; so - at least all 2* is affected); 3 domains (JW is the first domain in hosts). Nothing in logs; usually it's happened after certificates update (permissions are ok) - just after ejabberdctl reload_config SOMETIMES sending wrong certificate for SOME domain and SOME port. Usually i can fix it with one more reload_config.
    Just for example (same certs, nothing changed in config file, just launched ejabberdctl reload_config ~10 times):

    C2S TLS OK:

    rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5223 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = linuxoid.in
jabber.name: subject=CN = jabber.name

    S2S TLS WRONG:

    rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5270 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = jabberworld.info
jabber.name: subject=CN = jabberworld.info
    • First reload_config - same result:
    rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5223 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = linuxoid.in
jabber.name: subject=CN = jabber.name
rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5270 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = jabberworld.info
jabber.name: subject=CN = jabberworld.info
    • Second - now all ok:
    rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5223 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = linuxoid.in
jabber.name: subject=CN = jabber.name
rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5270 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = linuxoid.in
jabber.name: subject=CN = jabber.name
    • 3-10 reload_config - all ok.
    • Ok, let's change something. Added a space to /etc/ejabberd/certs/jabber.name.fullchain.pem in between lines:
    -----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

    Permissions are ok:

    ls -l /etc/ejabberd/certs/jabber.name.fullchain.pem 
-rw-r----- 1 root ejabberd 5852 Mar 24 20:16 /etc/ejabberd/certs/jabber.name.fullchain.pem

    Tried reload_config 7 times - all ok.

    • Ok, let's change something else. Added a "#" to config file (to already commented line at the beginning of file - so now it's just a '##').
      1st reload_config - all ok (!)
      2nd - got same problem again:
    rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5223 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = linuxoid.in
jabber.name: subject=CN = jabber.name
rain@walkbook:~$ for i in jabberworld.info linuxoid.in jabber.name ; do echo ${i}: $(openssl s_client -connect $i:5270 </dev/null  2>&1 | grep ^subject=CN) ; done
jabberworld.info: subject=CN = jabberworld.info
linuxoid.in: subject=CN = jabberworld.info
jabber.name: subject=CN = jabberworld.info

    3..7 - all ok.

    • Removed that comment.
      1st reload_config - got a problem.
      2nd - ok
      3rd - ok
      ...

  5. licaon-kter commented on Mar 24, 2024

    @licaon-kter
    licaon-kter
    on Mar 24, 2024
    Contributor

    @EOUpSL93Y are you the admin of linuxoid.in ?

  6. EOUpSL93Y commented on Mar 24, 2024

    @EOUpSL93Y
    EOUpSL93Y
    on Mar 24, 2024

    Yes

  7. weiss commented on Aug 2, 2024

    @weiss
    weiss
    on Aug 2, 2024
    Member

    Next time you run into the issue, could you check whether running the following call in an ejabberdctl debug shell fixes the issue:

    fast_tls:clear_cache().

    (Including the trailing .. Press Ctrl+g and then q to exit the shell.)

    As an alternative, you could use a script such as the following (if escript is in the path, and assuming the node name is ejabberd@localhost):

    #!/usr/bin/env escript
%%! -sname fix-cert@localhost

-define(NODE, 'ejabberd@localhost').

-spec main([string()]) -> any().
main(_Args) ->
  try ok = erpc:call(?NODE, fast_tls, clear_cache, [])
  catch error:{erpc, Reason} ->
      io:fwrite(standard_error, "Cannot query ejabberd: ~p~n", [Reason]),
      halt(1)
  end.

  8. mremond commented on Aug 5, 2024

    @mremond
    mremond
    on Aug 5, 2024
    Member

    Maybe having a clear cache ejabberdctl command would be handy ?
    It could a hook that module could register to.
    It could take a specific parameter to only clear the cache of a single module.

    @weiss Is this a good idea ?

  9. prefiks commented on Aug 5, 2024

    @prefiks
    prefiks
    on Aug 5, 2024
    Member

    We already do that in reload_config, so probably no need to add separate command just for that (and we call that clear cache function from reload_config). Now the question is why there are sometimes stale results...

  10. weiss commented on Aug 6, 2024

    @weiss
    weiss
    on Aug 6, 2024
    Member

    The problem being related to fast_tlss caching was just a blind guess of mine. I suggested that test to make sure nothing else is reloaded, just to track things down.

  11. EOUpSL93Y commented on Aug 11, 2024

    @EOUpSL93Y
    EOUpSL93Y
    on Aug 11, 2024

    Got this problem again just after upgrade to 24.07, then tried

    fast_tls:clear_cache().

    and it fixed the problem!

  12. badlop
    closed this as not plannedon Mar 20, 2025
  13. badlop
    reopened this on Apr 29, 2025

  14. badlop commented on May 2, 2025

    @badlop
    badlop
    on May 2, 2025
    Member

    To facilitate the investigation and hopefully finding a solution (or at least an acceptable workaround), it would help to have a reproduction guide, even if it doesn't offer 100% reproducibility.

    Following the explanations in the comments, I tried to reproduce the problem... but ejabberd always provides the proper certificate. This is how I setup the test environment, let's hope somebody points how to tweak it to reproduce the problem:

    Reproduction steps

    1. Download and install ejabberd 24.07. Run and Deb installers contain exactly the same files, so let's use the Run one which allows a regular user to manage ejabberd:

      wget https://github.com/processone/ejabberd/releases/download/24.07/ejabberd-24.07-1-linux-x64.run
chmod +x ejabberd-24.07-1-linux-x64.run
./ejabberd-24.07-1-linux-x64.run

    2. Generate self-signed certs and copy them

      make-certs.sh
cp *.pem $HOME/opt/ejabberd/conf/

    3. Configure ejabberd in file $HOME/opt/ejabberd/conf/ejabberd.yml

      hosts:
  - example.com
  - example.im
  - example.net
  - example.org

acme:
  auto: false

certfiles:
  - /opt/ejabberd/conf/example.com.pem
  - /opt/ejabberd/conf/example.im.pem
  - /opt/ejabberd/conf/example.net.pem
  - /opt/ejabberd/conf/example.org.pem

    4. Start ejabberd:

      $HOME/opt/ejabberd-24.07/bin/ejabberdctl live

    5. Run the tests:

      ./test.sh

    When asking a domain, if ejabberd offered a certificate with mismatched CN, it would return FAIL.
    However, no matter how many times the script runs, it always returns OK. Any clue how to reproduce the problem?

    make-certs.sh 
    makecert()
{
    DOMAIN=$1
    PASS=somerandompass
    openssl genrsa -des3 -out $DOMAIN-ca.key -passout pass:$PASS 4096
    openssl req -new -x509 -days 1826 -key $DOMAIN-ca.key -out $DOMAIN-ca.crt -passin pass:$PASS -subj "/CN=$DOMAIN"
    openssl genrsa -out $DOMAIN.key 4096
    openssl req -new -out $DOMAIN.csr -key $DOMAIN.key -passin pass:$PASS -subj "/CN=$DOMAIN"
    openssl x509 -req -in $DOMAIN.csr -CA $DOMAIN-ca.crt -CAkey $DOMAIN-ca.key -CAcreateserial -out $DOMAIN.crt -days 360 -passin pass:$PASS
    cat $DOMAIN.crt $DOMAIN.key > $DOMAIN.pem
}

for D in example.com example.im example.net example.org ; do
    makecert $D
done
    test.sh 
    test_port()
{
    PORT=$1
    echo ""
    echo "  Port $PORT"
    for DOMAIN in example.com example.im example.net example.org ; do
        RESULT=$(openssl s_client -connect $DOMAIN:$PORT </dev/null  2>&1 | grep ^subject= | sed 's|.*CN = ||g')
        [ "${DOMAIN}" = "${RESULT}" ] \
            && echo "    ${DOMAIN} = ${RESULT} -> OK" \
            || echo "    ${DOMAIN} / ${RESULT} -> FAIL" ;
    done
}

test_all()
{
    #for PORT in 5222 5223 5269 5443 ; do
    for PORT in 5223 5443 ; do
        test_port $PORT
    done
}

reload()
{
    $HOME/opt/ejabberd-24.07/bin/ejabberdctl reload_config
}

modify()
{
    echo "# aaa ">> $HOME/opt/ejabberd/conf/ejabberd.yml
    echo "">> $HOME/opt/ejabberd/conf/example.com.pem
    echo "">> $HOME/opt/ejabberd/conf/example.im.pem
    echo "">> $HOME/opt/ejabberd/conf/example.org.pem
}

echo "First run:"
test_all

echo ""
echo "Reloading..."
reload
echo "Second run:"
test_all

echo ""
echo "Modifying files..."
modify
echo "Reloading..."
reload
echo "Third run:"
test_all

  15. weiss commented on May 3, 2025

    @weiss
    weiss
    on May 3, 2025
    Member

    Any clue how to reproduce the problem?

    I don't know, so far I wasn't able to reproduce it on a test system either, that's precisely my problem.

    no matter how many times the script runs, it always returns OK.

    How often did you try? E.g., one of my production servers accepted a few millions of connections since midnight today, and monitoring detected the issue 5 times since then. So it's certainly not trivial to reproduce. And I can't rule out that the issue is only triggered by some condition that is missing when brute-forcing an otherwise idle test system with connection attempts.

    let's hope somebody points how to tweak it to reproduce the problem

    Your test looks similar to my monitoring script (I'll include that below). My only suggestion would be to not reload ejabberd between your attempts, as reloading hides the problem. (The issue is most certainly related to fast_tls's certs_map, as clearing that fixes the issue.)

    Just BTW, I hadn't looked at my script in quite a while and am now surprised that it only detects the problem if ejabberd incorrectly offers the certificate for the 'main' domain, rather than checking whether ejabberd actually offers the correct certificate. This server has a large number of virtual hosts, and from what I remember, the failure case has indeed always been that the certificate for the 'main' domain (the first domain in the hosts list) was offered. Rather than the certificate for another 'secondary' domain.

    #!/bin/sh
set -u

prefix='/home/xmpp'
main_domain='example.org'
c2s_domain='example.net'
s2s_domain='example.com'
notify_file='/dev/shm/cert-check-notified'
me=${0##*/}

notify()
{
        local result=$1
        local subject="$result certificate offered"

        printf '%s %s\n' "$(date '+%F %T')" \
            "The server currently offers a(n) $result TLS certificate"
}

openssl s_client                                                        \
    -servername "$c2s_domain"                                           \
    -connect 'xmpps.conversations.im:5223' <'/dev/null' 2>'/dev/null'   \
    | openssl x509 -noout -subject | tail -1 | grep -Fq "$main_domain"
c2s_status=$?

openssl s_client                                                        \
    -servername "$s2s_domain"                                           \
    -connect 'xmpps.conversations.im:5269' <'/dev/null' 2>'/dev/null'   \
    | openssl x509 -noout -subject | tail -1 | grep -Fq "$main_domain"
s2s_status=$?

if [ $c2s_status -eq 0 ] || [ $s2s_status -eq 0 ]
then
        if ! [ -e "$notify_file" ]
        then
                
                touch "$notify_file"
                notify 'INCORRECT'
        fi
else
        if [ -e "$notify_file" ]
        then
                rm -f "$notify_file"
                notify 'CORRECT'
        fi
fi

  16. poVoq commented on May 3, 2025

    @poVoq
    poVoq
    on May 3, 2025

    This server has a large number of virtual hosts, and from what I remember, the failure case has indeed always been that the certificate for the 'main' domain (the first domain in the hosts list) was offered.

    This is indeed also what happens on my server.

  17. weiss commented on Oct 18, 2026

    @weiss
    weiss
    on Oct 18, 2026
    Member

    Is anyone still running into this?

    Our monitoring detected the issue several times per day until August 18, 2025, and not once since then. On that day, we updated the server from commit 1a9b147 to ff3d33d. However, I haven’t spotted a change in ejabberd or fast_tls that looks related.

  18. weiss commented on Oct 18, 2026

    @weiss
    weiss
    on Oct 18, 2026
    Member

    On that day, we updated the server from commit 1a9b147 to ff3d33d. However, I haven’t spotted a change in ejabberd or fast_tls that looks related.

    Ah, looks like we hadn’t fully restarted ejabberd since June 2025, so we were running an older fast_tls.

    I think I got it: The cache key in fast_tls doesn’t include whether a TLS context is used for an incoming or outgoing connection, but the SNI callback is only installed for the former. So if an outgoing s2s connection is the first to create a context (using the default domain's certificate), that context is then reused for incoming connections, and SNI is ignored until the next cache flush.

    The changes for accepting s2s client certificates without the client purpose flag (processone/fast_tls@f1e55d6, 72bc9b6) hide the problem for s2s, as the new flag ends up in the cache key, so incoming and outgoing s2s connections no longer share contexts. It could still be triggered for c2s, though, if ciphers, protocol options, DH file, and CA file match the s2s settings.

    processone/fast_tls#68 should fix this.

  19. prefiks
    closed this as completedin 3667381on Oct 19, 2026
  20. badlop
    added this to the ejabberd 26.xx milestone on Oct 21, 2026
  21. badlop
    added
    Kind:Bug
    and removed
    Note:Needs more info
    on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Kind:Bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions