This reverts commit affdb62b.

The commit introduced a regression with AD codecs where the stream is
always clean up.  Since the patch is just a minor optimization and
reverting the commit fixes the issue, let's just revert it.
Reported-and-tested-by: Michael Burian <michael.burian@sbg.at>
Cc: <stable@vger.kernel.org> [v3.9+]
Signed-off-by: Takashi Iwai <tiwai@suse.de>