Addresses #2172 refs[t:2172] Bugfix of commit r15040 (Addresses #2037 refs[t:2037])

Was not holding cachetable lock to access rwlock (whose mutex was cachetable lock).
This caused a race condition that you would sometimes grab the cachetable lock on exiting the lock but didn't have it before.

We believe this caused the deadlock

git-svn-id: file:///svn/toku/tokudb@15802 c7de825b-a66e-492c-adef-691d508d4ae1

newbrt/cachetable.c

@@ -1975,10 +1975,14 @@ toku_cachetable_end_checkpoint(CACHETABLE ct, TOKULOGGER logger,
         //cachefiles_in_checkpoint is protected by the checkpoint_safe_lock
 	for (cf = ct->cachefiles_in_checkpoint; cf; cf=cf->next_in_checkpoint) {
 	    if (cf->end_checkpoint_userdata) {
+                cachetable_lock(ct);
                 rwlock_prefer_read_lock(&cf->fdlock, ct->mutex);
+                cachetable_unlock(ct);
                 //end_checkpoint fsyncs the fd, which needs the fdlock
 		int r = cf->end_checkpoint_userdata(cf, cf->userdata);
+                cachetable_lock(ct);
                 rwlock_read_unlock(&cf->fdlock);
+                cachetable_unlock(ct);
 		assert(r==0);
 	    }
 	}