drm/amd/display: redesign scaling rotation math

Change the math to work in viewport rotation when calculating
viewport and viewport adjustment. This simplifies the math
for viewport calculation and makes viewport adjustment easier to
understand.
Signed-off-by: Dmytro Laktyushkin <Dmytro.Laktyushkin@amd.com>
Reviewed-by: Tony Cheng <Tony.Cheng@amd.com>
Acked-by: Bhawanpreet Lakha <Bhawanpreet.Lakha@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>