All ports types can have a fixed PHY associated with it. Remove the
check which limits removal to only CPU and DSA ports.
Signed-off-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>