Add debug logging and sync troubleshooting information to documentation [ci skip]

doc/administration/auth/ldap-ee.md

@@ -172,6 +172,8 @@ network and LDAP server response time will affect these metrics.
 
 ## Troubleshooting
 
+### Referral Error
+
 If you see `LDAP search error: Referral` in the logs, or when troubleshooting
 LDAP Group Sync, this error may indicate a configuration problem. The LDAP
 configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source)
@@ -196,3 +198,178 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server
       account control attribute (`userAccountControl:1.2.840.113556.1.4.803`)
       has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/
       for more information.
+
+### User DN has changed
+
+When an LDAP user is created in GitLab, their LDAP DN is stored for later reference.
+If a user's DN changes, it can cause problems for LDAP sync. Administrators can
+manually update a user's stored DN in this case.
+
+> **Note:** If GitLab cannot find a user by their DN, it will attempt to fallback
+  to finding the user by their email. If the lookup is successful, GitLab will
+  update the stored DN to the new value.
+
+1. Sign in to GitLab as an administrator user.
+1. Navigate to **Admin area -> Users**.
+1. Search for the user
+1. Open the user, by clicking on their name. Do not click 'Edit'.
+1. Navigate to the **Identities** tab.
+1. Click 'Edit' next to the LDAP identity.
+1. Change the 'Identifier' to match the user's new LDAP DN.
+1. Save the identity.
+
+Now the user should sync correctly.
+
+### User is not being added to a group
+
+Sometimes you may think a particular user should be added to a GitLab group via
+LDAP group sync, but for some reason it's not happening. There are several
+things to check to debug the situation.
+
+- Ensure LDAP configuration has a `group_base` specified. This configuration is
+  required for group sync to work properly.
+- Ensure the correct LDAP group link is added to the GitLab group. Check group
+  links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**.
+- Check that the user has an LDAP identity
+  1. Sign in to GitLab as an administrator user.
+  1. Navigate to **Admin area -> Users**.
+  1. Search for the user
+  1. Open the user, by clicking on their name. Do not click 'Edit'.
+  1. Navigate to the **Identities** tab. There should be an LDAP identity with
+     an LDAP DN as the 'Identifier'.
+
+If all of the above looks good, jump in to a little more advanced debugging.
+Often, the best way to learn more about why group sync is behaving a certain
+way is to enable debug logging. There is verbose output that details every
+step of the sync.
+
+1. Start a Rails console
+
+    ```bash
+    # For Omnibus installations
+    sudo gitlab-rails console
+
+    # For installations from source
+    sudo -u git -H bundle exec rails console production
+    ```
+1. Set the log level to debug (only for this session):
+
+    ```ruby
+    Rails.logger.level = Logger::DEBUG
+    ```
+1. Choose a GitLab group to test with. This group should have an LDAP group link
+   already configured. If the output is `nil`, the group could not be found.
+   If a bunch of group attributes are output, your group was found successfully.
+
+    ```ruby
+    group = Group.find_by(name: 'my_group')
+
+    # Output
+    => #<Group:0x007fe825196558 id: 1234, name: "my_group"...>
+    ```
+1. Run a group sync for this particular group.
+
+    ```ruby
+    EE::Gitlab::LDAP::Sync::Group.execute_all_providers(group)
+    ```
+1. Look through the output of the sync. See [example log output](#example-log-output)
+   below for more information about the output.
+1. If you still aren't able to see why the user isn't being added, query the
+   LDAP group directly to see what members are listed. Still in the Rails console,
+   run the following query:
+
+    ```ruby
+    adapter = Gitlab::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider
+    ldap_group = EE::Gitlab::LDAP::Group.find_by_cn('group_cn_here', adapter)
+
+    # Output
+    => #<EE::Gitlab::LDAP::Group:0x007fcbdd0bb6d8
+    ```
+1. Query the LDAP group's member DNs and see if the user's DN is in the list.
+   One of the DNs here should match the 'Identifier' from the LDAP identity
+   checked earlier. If it doesn't, the user does not appear to be in the LDAP
+   group.
+
+    ```ruby
+    ldap_group.member_dns
+
+    # Output
+    => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"]
+    ```
+1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead.
+   If you didn't see results from the last query, try querying by UIDs instead.
+
+    ```ruby
+    ldap_group.member_uids
+
+    # Output
+    => ['john','mary']
+    ```
+
+#### Example log output
+
+The output of the last command will be very verbose, but contains lots of
+helpful information. For the most part you can ignore log entries that are SQL
+statements.
+
+Indicates the point where syncing actually begins:
+
+```bash
+Started syncing all providers for 'my_group' group
+```
+
+The follow entry shows an array of all user DNs GitLab sees in the LDAP server.
+Note that these are the users for a single LDAP group, not a GitLab group. If
+you have multiple LDAP groups linked to this GitLab group, you will see multiple
+log entries like this - one for each LDAP group. If you don't see an LDAP user
+DN in this log entry, LDAP is not returning the user when we do the lookup.
+Verify the user is actually in the LDAP group.
+
+```bash
+Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com",
+"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com",
+"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com",
+"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com",
+"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com",
+"uid=mary4,ou=people,dc=example,dc=com"]
+```
+
+Shortly after each of the above entries, you will see a hash of resolved member
+access levels. This hash represents all user DNs GitLab thinks should have
+access to this group, and at which access level (role). This hash is additive,
+and more DNs may be added, or existing entries modified, based on additional
+LDAP group lookups. The very last occurrence of this entry should indicate
+exactly which users GitLab believes should be added to the group.
+
+> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Master'
+  and 50 is 'Owner'
+
+```bash
+Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30,
+"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30,
+"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30,
+"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30,
+"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30,
+"uid=mary4,ou=people,dc=example,dc=com"=>30}
+```
+
+It's not uncommon to see warnings like the following. These indicate that GitLab
+would have added the user to a group, but the user could not be found in GitLab.
+Usually this is not a cause for concern.
+
+If you think a particular user should already exist in GitLab, but you're seeing
+this entry, it could be due to a mismatched DN stored in GitLab. See
+[User DN has changed](#User-DN-has-changed) to update the user's LDAP identity.
+
+```bash
+User with DN `uid=john0,ou=people,dc=example,dc=com` should have access
+to 'my_group' group but there is no user in GitLab with that
+identity. Membership will be updated once the user signs in for
+the first time.
+```
+
+Finally, the following entry says syncing has finished for this group:
+
+```bash
+Finished syncing all providers for 'my_group' group
+```