Built-in PHY ports are still being polled, avoid generating spurious
and duplicate events which the PHY library resolves through polling
anyways.

Fixes: 0e01491d ("net: dsa: b53: Add SerDes support")
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>