when our ->disable() method is called, we must
make sure to teardown all our resources, including
endpoints.
Signed-off-by: Felipe Balbi <balbi@ti.com>