Remove the invalid key factories

They're only used once each, and they're easy to build in-place.

See merge request !1766