Although an ipsec SA was established, kernel couldn't seem to find it.

I think since we are now using "x->sel.family" instead of "family" in
the xfrm_selector_match() called in xfrm_state_find(), af_key needs to
set this field too, just as xfrm_user.

In af_key.c, x->sel.family only gets set when there's an
ext_hdrs[SADB_EXT_ADDRESS_PROXY-1] which I think is for tunnel.

I think pfkey needs to also set the x->sel.family field when it is 0.

Tested with below patch, and ipsec worked when using pfkey.
Signed-off-by: David S. Miller <davem@davemloft.net>