We need to cleanup after each execution so we can still re-use the same
paths and prevent collision with the on disk validation