Use the approach presented here: https://her.esy.fun/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html
to try to reduce the log caused by crawlers.
The idea is make users visiting the website with a JS-enabled browser register a cookie, and then redirect them transparently to the website. If the cookie is set, they can access it. If not, they loop on the simple webpage served by nginx asking them to set the cookie.
There is an exception when the user agent is the git command line.
This approach is completely transparent for the users.
There will probably be other impacts on our infrastructure, for exemple code retrieving the JSON schemas. Let's address this after we can prove this change has any effect on the bot traffic.