This reverts commit 7e7c059c.

I was wrong about trying to do this, as it breaks the
orthogonality between gpiochips and irqchips.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>