Windows Server Backup fails with "semaphore timeout" (0x8004245f) or unknown error"

Windows Server Backup fails with "semaphore timeout" (0x8004245f) I'm using Windows Server 2022. At the beginning, full backups were no problem.

Now, the first backup attempt fails with "Unknown error (0x8004245f)" / "The semaphore timeout period has expired." After it fails, if I run the backup again, it completes without any issues. But if I try to run another backup right after that success, it fails again.

If I try to back up multiple drives, or do a full server backup, it fails. It only succeeds if I back up one drive at a time and even then, only after a failed attempt in between.

I've restarted the server, and vssadmin list writers always shows everything stable. I've also tried backing up to both an external drive and a shared folder same issue either way.

Does any one has the same issue?