device_remove_group() was called on any cleanup, even if the
device attrs had not been added yet.  That can occur in certain
error scenarios, so add a flag to know if it has been added.

Also make sure we remove the dev if we added it ourselves.
Signed-off-by: Corey Minyard <cminyard@mvista.com>
Cc: stable@vger.kernel.org # 4.15
Cc: Laura Abbott <labbott@redhat.com>
Tested-by: Bill Perkins <wmp@grnwood.net>