[os_must_gather] Add OOMKill log#3776
[os_must_gather] Add OOMKill log#3776danpawlik wants to merge 1 commit intoopenstack-k8s-operators:mainfrom
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
danpawlik
force-pushed
the
add-oom-kill-info
branch
2 times, most recently
from
2c71749 to
fdad102
Compare
|
Looks good to me. Did we found any occurrence of this? |
|
@averdagu in some cases, we see that tempest tests can not pass, where we wondering what can be root cause of it. Maybe it is oomkill, maybe bad health check? Something to discover. We just want to more aware whats going on. |
|
/lgtm |
| oc get events | ||
| --all-namespaces | ||
| --sort-by=.lastTimestamp | | ||
| grep -i -E 'OOMKill|Killing.*out of memory|Pressure' > |
There was a problem hiding this comment.
(blocking) concenr: I'm unsure here but it seems that if we don't have any OOMKill, this might fail.
There was a problem hiding this comment.
Good catch, I think we should add ignore_errors on this task.
I've checked on my machine and if grep don't find anything it will return 1
There was a problem hiding this comment.
Also, I don't understand why we sort by timestamp.
You're interessted in OOMKill related events, so you should perform a server side pre-filtering with something like:
oc get events --all-namespaces --field-selector type=Warning --sort-by=.reason
You can get the json and use jq to filter the "Reason" you're interested to, but to me that kind of post- processing should happen later, as an additional task, on the gathered file.
There was a problem hiding this comment.
we don't have pipefail so would be good.
Adding ignore error in case if cluster utilization is high and oc will get an error.
There was a problem hiding this comment.
Re-opening, afaik redirection are not taken into consideration for pipefail.
ALso I see here a @fmount comment that you might skipped.
danpawlik
force-pushed
the
add-oom-kill-info
branch
from
fdad102 to
5436872
Compare
|
New changes are detected. LGTM label has been removed. |
It might happen that there would be an OOMKill of the container. We would like to be aware about that issue. Signed-off-by: Daniel Pawlik <dpawlik@redhat.com>
danpawlik
force-pushed
the
add-oom-kill-info
branch
from
5436872 to
afa5997
Compare
fmount
left a comment
fmount
left a comment
There was a problem hiding this comment.
I'm not against the idea of being aware of any OOMKill that might happen in the cluster, but I think just getting the events is not enough to have a clear idea of the nodes status.
Instead considering to stores the output of oc describe nodes that contains more data about Conditions and Capacity and can drive through a resolution of the problem.
| - name: Collect pod usage | ||
| ansible.builtin.include_tasks: get_top.yml | ||
|
|
||
| - name: Check if there were some OOMKill |
There was a problem hiding this comment.
-1: events are already collected within the must-gather, so unless you're interested in a specific workload (if you give me an example I might understand better what you have in mind).
I think just reviewing the collected nodes that report already memory or disk pressure, or the existing events, might give you an idea of what's going on.
| - name: Check if there were some OOMKill | ||
| ansible.builtin.shell: > | ||
| oc get events | ||
| --all-namespaces |
There was a problem hiding this comment.
--all-namespaces in a large environment might take a bit, and I'm not sure you need to check which Pod is oomikilled all over the cluster.
Did you try in a DC or DZ environment?
Perhaps you can consider running the regular openshift must gather along with the openstack one if you're looking for this kind of data. [1]
[1] e.g.
oc adm must-gather \
--image-stream=openshift/must-gather \
--image=quay.io/openstack-k8s-operators/openstack-must-gather:latestIf you get OOMKill all over the cluster I expect must-gather does not work and you've probably lost access to the OCP API at this point.
| oc get events | ||
| --all-namespaces | ||
| --sort-by=.lastTimestamp | | ||
| grep -i -E 'OOMKill|Killing.*out of memory|Pressure' > |
There was a problem hiding this comment.
Also, I don't understand why we sort by timestamp.
You're interessted in OOMKill related events, so you should perform a server side pre-filtering with something like:
oc get events --all-namespaces --field-selector type=Warning --sort-by=.reason
You can get the json and use jq to filter the "Reason" you're interested to, but to me that kind of post- processing should happen later, as an additional task, on the gathered file.
4 participants
It might happen that there would be an OOMKill of the container. We would like to be aware about that issue.