It’s worth noting that patchelf isn’t blindly modifying binaries. It’s just rewriting some metadata, namely rpath and the path to the dynamic linker.
It’s actually common for other distros to use patchelf in their packages too. People are just much less aware of it. That’s because even among FHS distros, authors of different software make different assumptions about paths. Here’s a search result for “patchelf” in the Debian repositories, for example.
https://codesearch.debian.net/search?q=patchelf&literal=1
Personally, I’m not aware of any downside to using patchelf. Patched binaries work just fine. It’s a common criticism, but it might be a knee-jerk reaction.