This reverts commit 27da370e.

Sudip Mukherjee reports that this broke pulseaudio with a NULL pointer
dereference in vc4_hdmi_audio_prepare(), bisected it to this commit, and
confirmed that a revert fixed the problem.

Revert the problematic commit until fixed.

Link: https://lore.kernel.org/all/CADVatmPB9-oKd=ypvj25UYysVo6EZhQ6bCM7EvztQBMyiZfAyw@mail.gmail.com/
Link: https://lore.kernel.org/all/CADVatmN5EpRshGEPS_JozbFQRXg5w_8LFB3OMP1Ai-ghxd3w4g@mail.gmail.com/Reported-and-tested-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
Cc: Maxime Ripard <maxime@cerno.tech>
Cc: Emma Anholt <emma@anholt.net>
Cc: Dave Airlie <airlied@gmail.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>