rtos: string: Add memory barrier to memset_s for security

[tmleman]

Add Xtensa memory barrier (memw) instruction after memset() in memset_s() implementation to prevent compiler dead store elimination (DSE) optimization from removing the memory clearing operation.

When optimization flags like -O2 are enabled, compilers may perform dead store elimination and incorrectly remove memset() calls used for security purposes to scrub sensitive data from memory. This is critical for confidential data handling where memory must be reliably cleared after use.

The memory barrier ensures the memset operation completes and cannot be optimized away, satisfying secure memory scrubbing requirements for cryptographic operations and sensitive data processing.

Additionally, the patch removes the check for the return value of memset. The standard C library memset always returns the pointer passed as its first argument and does not indicate errors through its return value. Error handling for a NULL destination is already performed earlier in the function, so the return value check is unnecessary and can be safely omitted.

[Copilot]

Pull request overview

This PR enhances the security of the memset_s() function by preventing compiler optimizations from eliminating critical memory clearing operations. The change adds a memory barrier instruction for Xtensa architectures and removes unnecessary error checking.

Changes:

• Added Xtensa-specific memory barrier (memw) after memset() to prevent dead store elimination
• Removed unnecessary return value check from memset() call since it always returns the destination pointer

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The memory barrier is only applied for Xtensa architecture, leaving other architectures vulnerable to dead store elimination. For security-critical memset_s() function, consider using a portable solution like a volatile function pointer or compiler builtins (e.g., memset_explicit() in C23, or explicit_bzero()) that work across all architectures to prevent compiler optimizations from removing the memory clearing operation.

Suggested change

	memset(dest, data, count);
	/*
	* Prevent compiler from optimizing away the memset.
	* Memory barrier prevents dead store elimination.
	*/
	#if defined(CONFIG_XTENSA)
	__asm__ __volatile__("memw" ::: "memory");
	#endif
	/*
	* Prevent compiler from optimizing away the memset.
	* Use a volatile function pointer so the call has
	* observable side effects on all architectures.
	*/
	void *(*volatile memset_ptr)(void *, int, size_t) = memset;
	memset_ptr(dest, data, count);

[Copilot]

The memory barrier implementation creates architecture-specific code paths in a security function. Consider extracting this into a macro (e.g., SECURE_MEMORY_BARRIER()) defined in an architecture header file. This would improve maintainability and make it easier to add barriers for other architectures in the future.

[lyakh]

I understand the motivation, but I don't think this is the right place to fix this. I'd say, that memset_s() should have the same semantics as memset() plus a couple of argument checks. I don't think adding a memory barrier here is a good idea. It needlessly punishes most callers, while I think it's those "special" callers, where such an optimisation can take place and can be harmful. Also not sure it's the memw that prevents the compiler from optimising the code, I'd rather suspect that it's the __volatile__ attribute. Maybe just __asm__ __volatile__("" ::: "memory") would be enough? But again - not here, but in actual sensitive code sections, and maybe as a meaningful macro.

[tmleman]

@lyakh Thank you for your feedback. I appreciate your point about the potential unnecessary overhead for most use cases. My initial thought was that adding the barrier directly to memset_s would be the safest approach, ensuring that memory scrubbing is always protected against compiler optimizations.

Alternatively, introducing a dedicated function or macro (e.g., secure_memzero()) for sensitive memory scrubbing is a good idea. My concern was that, during development/code reviews, developers might still use memset_s for scrubbing sensitive data and overlook the secure variant. However, if the team agrees, I’m open to moving the barrier to a specialized function and keeping memset_s lightweight.

Regarding the use of memw, it’s based on the Xtensa ISA documentation, which specifies that it enforces memory ordering at the hardware level.

Let me know your thoughts, and I’m happy to adjust the implementation as needed.