This reverts commit f4caa6ee.

The same can be expressed better by dropping the
SH_PFC_PIN_CFG_PULL_DOWN flag from the GPIO description, as it includes
returning an error to the caller when trying to configure the pin for
pull-down, causing:

    sh-pfc e6060000.pin-controller: pin_config_set op failed for pin 201
    sh-pfc e6060000.pin-controller: Error applying setting, reverse things back
    sh-pfc e6060000.pin-controller: failed to select default state
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: Simon Horman <horms+renesas@verge.net.au>