Remove checking if a unsigned is less than zero

This was found using a static code analysis program called cppcheck.
Signed-off-by: Rickard Strandqvist <rickard_strandqvist@spectrumdigital.se>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>