A discussion on Reddit (https://redd.it/4znftc) made me want to use Rest Mode after reading that so many people are using it without having any issues.
What kept me bothering was that my PS4 fan was ALWAYS ON, even in Rest Mode. Thought this can't be right.
Dug a little bit deeper and found out that you can turn off the power of the USB ports and that will turn off the fan as well. It will also cause the PS4 to use 3 watts per hour instead of 10 watts in Rest Mode.
Here is the step-by-step procedure:
Turn off power to USB ports in Rest Mode to make PS4 silent