[action] [PR:703] [SmartSwitch] Add graceful shutdown and startup handling in platform daemons #726
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…daemons
<!-- Provide a general summary of your changes in the Title above -->
#### Description
<!--
Describe your changes in detail
-->
HLD: https://github.com/sonic-net/SONiC/blob/master/doc/smart-switch/graceful-shutdown/graceful-shutdown.md
These changes build upon enhancements in [`sonic-platform-daemons#667`](sonic-net#667)
This PR introduces **graceful shutdown and startup orchestration** across SONiC platform daemons to ensure safe DPU and peripheral module transitions during reboot or administrative state changes.
Key updates include:
- Integration of `ModuleBase` lifecycle methods (`module_pre_shutdown`, `module_post_startup`, and `set_admin_state_gracefully`) into platform daemons.
- Move graceful handling of PCIe detach/reattach and sensor reload sequences into set_admin_state_gracefully.
- State tracking in `CHASSIS_MODULE_TABLE` via `STATE_DB` to synchronize transition state across processes.
- File-based operation locks to prevent concurrent access to shared hardware resources.
#### Motivation and Context
<!--
Why is this change required? What problem does it solve?
If this pull request closes/resolves an open Issue, make sure you
include the text "fixes #xxxx", "closes #xxxx" or "resolves #xxxx" here
-->
Platform daemons currently perform shutdown and startup independently, leading to:
- Race conditions during DPU detachment.
- Inconsistent Redis state across PMON daemons.
- Uncoordinated sensor and PCIe transitions during reboot.
This change introduces a unified **graceful shutdown framework** for SmartSwitch modules.
It ensures predictable module transitions, preserves hardware health, and supports orchestrated restarts without transient hardware errors.
#### How Has This Been Tested?
<!--
Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc.
-->
Testing performed on both **DPU-enabled (SmartSwitch)**.
**Functional validation**
- Verified end-to-end reboot flow with DPU detach/reattach sequence.
- PCIe state (`detaching/attaching`) reflected in `STATE_DB`.
- `pcied` daemon logs confirm ordered detach before reboot and reattach after startup.
- Confirmed no stale Redis entries or orphaned locks post-reboot.
**Unit tests executed**
- tests/test_DaemonPcied.py
- tests/test_chassisd_graceful.py
Coverage includes:
- Transition flag handling
- Timeout behavior
- DB write/read operations
- Graceful admin state flow
**Manual validation**
#### Additional Information (Optional)
Collaborator
Author
|
Original PR: #703 |
Collaborator
Author
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
HLD: https://github.com/sonic-net/SONiC/blob/master/doc/smart-switch/graceful-shutdown/graceful-shutdown.md
These changes build upon enhancements in
sonic-platform-daemons#667This PR introduces graceful shutdown and startup orchestration across SONiC platform daemons to ensure safe DPU and peripheral module transitions during reboot or administrative state changes.
Key updates include:
ModuleBaselifecycle methods (module_pre_shutdown,module_post_startup, andset_admin_state_gracefully) into platform daemons.CHASSIS_MODULE_TABLEviaSTATE_DBto synchronize transition state across processes.Motivation and Context
Platform daemons currently perform shutdown and startup independently, leading to:
This change introduces a unified graceful shutdown framework for SmartSwitch modules.
It ensures predictable module transitions, preserves hardware health, and supports orchestrated restarts without transient hardware errors.
How Has This Been Tested?
Testing performed on both DPU-enabled (SmartSwitch).
Functional validation
detaching/attaching) reflected inSTATE_DB.pcieddaemon logs confirm ordered detach before reboot and reattach after startup.Unit tests executed
Coverage includes:
Manual validation
Additional Information (Optional)