Use single data directory

Now all data is stored under DATA_DIR and Weblate manages subdirs on
it's own. Based on forseeable increase in directories we need to manage,
this is the only way forward.

Fixes #604
Signed-off-by: Michal Čihař <michal@cihar.com>

.gitignore

@@ -18,3 +18,5 @@ weblate-*.tar.*
 /Weblate.egg-info/
 /build/
 /pyuca
+/data/
+/data-test/

docs/admin/config.rst

@@ -102,6 +102,21 @@ For example you can enable only few of them:
 
 .. seealso:: :ref:`checks`, :ref:`custom-checks`
 
+.. setting:: DATA_DIR
+
+DATA_DIR
+--------
+
+Directory where Weblate stores all data. This consists of VCS repositories,
+fulltext index and various configuration files for external tools.
+
+Following subdirectories usually exist:
+
+:file:`vcs`
+    Version control repositories.
+:file:`whoosh`
+    Fulltext search index using Whoosh engine.
+
 .. setting:: ENABLE_AVATARS
 
 ENABLE_AVATARS
@@ -145,6 +160,9 @@ Whether to show links to share translation progress on social networks.
 GIT_ROOT
 --------
 
+.. deprecated:: 2.1
+   This setting is no longer used, use :setting:`DATA_DIR` instead.
+
 Path where Weblate will store cloned VCS repositories. Defaults to
 :file:`repos` subdirectory.
 
@@ -463,4 +481,7 @@ achieved by setting ``WSGIScriptAlias``.
 WHOOSH_INDEX
 ------------
 
+.. deprecated:: 2.1
+   This setting is no longer used, use :setting:`DATA_DIR` instead.
+
 Directory where Whoosh fulltext indices will be stored. Defaults to :file:`whoosh-index` subdirectory.

docs/admin/install.rst

@@ -185,9 +185,8 @@ On openSUSE or SLES you can install them using:
 Filesystem permissions
 ----------------------
 
-Weblate process needs to be able to read and write to two directories where it
-keeps data. The :setting:`GIT_ROOT` is used for storing VCS repositories and
-:setting:`WHOOSH_INDEX` is used for fulltext search data.
+Weblate process needs to be able to read and write to the directory where it
+keeps data - :setting:`DATA_DIR`.
 
 The default configuration places them in same tree as Weblate sources, however
 you might prefer to move these to better location such as
@@ -755,7 +754,7 @@ After installation on OpenShift Weblate is ready to use and preconfigured as fol
 * Committing of pending changes if the cron cartridge is installed (commit_pending)
 * Weblate machine translations for suggestions bases on previous translations (MACHINE_TRANSLATION_SERVICES)
 * Source language for machine translations set to "en-us" (SOURCE_LANGUAGE)
-* Weblate directories (STATIC_ROOT, GIT_ROOT, TTF_PATH, WHOOSH_INDEX, HOME, Avatar cache) set according to OpenShift requirements/conventions
+* Weblate directories (STATIC_ROOT, DATA_DIR, TTF_PATH, Avatar cache) set according to OpenShift requirements/conventions
 * Django site name and ALLOWED_HOSTS set to DNS name of your OpenShift application
 * Email sender addresses set to no-reply@<OPENSHIFT_CLOUD_DOMAIN>, where <OPENSHIFT_CLOUD_DOMAIN> is the domain OpenShift runs under. In case of OpenShift Online it's rhcloud.com.
 
@@ -897,14 +896,14 @@ case your database supports it.
 Migrating VCS repositories
 +++++++++++++++++++++++++++
 
-The VCS repositories stored under :setting:`GIT_ROOT` need to be migrated as
+The VCS repositories stored under :setting:`DATA_DIR` need to be migrated as
 well. You can simply copy them or use :command:`rsync` to do the migration
 more effectively.
 
 Migrating fulltext index
 ++++++++++++++++++++++++
 
-For the fulltext index (stored in :setting:`WHOOSH_INDEX`) it is better not to
+For the fulltext index (stored in :setting:`DATA_DIR`) it is better not to
 migrate it, but rather to generate fresh one using :djadmin:`rebuild_index`.
 
 Other notes

docs/admin/projects.rst

@@ -193,8 +193,9 @@ Clone Git repository in advance
 +++++++++++++++++++++++++++++++
 
 You can put in place Git repository which will be used by Weblate. The
-repositories are stored in path defined by :setting:`GIT_ROOT` in
-:file:`settings.py` in :file:`<project>/<component>` directories.
+repositories are stored in :file:`vcs` directory in path defined by
+:setting:`DATA_DIR` in :file:`settings.py` in :file:`<project>/<component>`
+directories.
 
 This can be especially useful if you already have local clone of this
 repository and you can use ``--reference`` option while cloning:

docs/admin/upgrade.rst

@@ -186,6 +186,20 @@ This upgrade also requires you to upgrade python-social-auth from 0.1.x to
 
 .. _Upgrading PSA with South: http://psa.matiasaguirre.net/docs/installing.html#django-with-south
 
+.. seealso::
+
+    If you are upgrading to Django 1.7 in same step, please consult
+    :ref:`django-17`.
+
+Upgrade from 2.0 to 2.1
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The filesystem paths configuration has changed, the :setting:`GIT_ROOT` and
+:setting:`WHOOSH_INDEX` are gone and now all data resides in
+:setting:`DATA_DIR`. The existing data should be automatically migrated by
+supplied migration, but in case of non standard setup, you might need to move
+these manually.
+
 .. seealso::
 
     If you are upgrading to Django 1.7 in same step, please consult

docs/changes.rst

@@ -11,6 +11,7 @@ weblate 2.1
 * Documentation improvements.
 * Various bugfixes.
 * Automatic hiding of columns in translation listing for small screens.
+* Changed configuration of filesystem paths.
 
 Released on ? 2014.
 

docs/faq.rst

@@ -82,7 +82,7 @@ How to export Git repository weblate uses?
 ------------------------------------------
 
 There is nothing special about the repository, it lives under
-:setting:`GIT_ROOT` directory and is named as `project/component/`. If you
+:setting:`DATA_DIR` directory and is named as :file:`vcs/<project>/<component>/`. If you
 have SSH access to this machine, you can use the repository directly.
 
 For anonymous access you might want to run git server and let it serve the
@@ -249,8 +249,8 @@ The easiest way to do this is to run:
 
 .. code-block:: sh
 
-    # Go to GIT_ROOT directory
-    cd weblate/repos
+    # Go to DATA_DIR directory
+    cd data/vcs
     # Compress all Git repositories
     for d in */* ; do
         pushd $d
@@ -260,7 +260,7 @@ The easiest way to do this is to run:
 
 .. seealso::
 
-    :setting:`GIT_ROOT`
+    :setting:`DATA_DIR`
 
 .. _faq-ft-slow:
 

weblate.spec

@@ -71,8 +71,7 @@ cd ../..
 make -C docs html
 cp weblate/settings_example.py weblate/settings.py
 sed -i 's@^BASE_DIR = .*@BASE_DIR = "%{WLDIR}/weblate"@g' weblate/settings.py
-sed -i 's@^WHOOSH_INDEX = .*@WHOOSH_INDEX = "%{WLDATADIR}/whoosh-index"@g' weblate/settings.py
-sed -i 's@^GIT_ROOT = .*@GIT_ROOT = "%{WLDATADIR}/repos"@g' weblate/settings.py
+sed -i 's@^DATA_DIR = .*@DATA_DIR = "%{WLDATADIR}"@g' weblate/settings.py
 sed -i "s@'ENGINE': 'django.db.backends.sqlite3'@'ENGINE': 'django.db.backends.mysql'@" weblate/settings.py
 sed -i "s@'NAME': 'weblate.db'@'NAME': 'weblate'@" weblate/settings.py
 sed -i 's@/usr/lib/python.*/site-packages@%{python_sitelib}@g' examples/apache.conf

weblate/__init__.py

@@ -24,6 +24,7 @@ from weblate.requirements import (
     check_requirements, get_versions, get_optional_versions
 )
 from weblate.trans.vcs import GitRepository, RepositoryException
+from weblate.trans.data import check_data_writable
 
 logger = logging.getLogger('weblate')
 
@@ -112,3 +113,4 @@ def get_versions_string():
 # Check for requirements
 
 check_requirements()
+check_data_writable()

weblate/appsettings.py

@@ -33,6 +33,9 @@ def getvalue(name, default):
 # Weblate installation root
 BASE_DIR = getvalue('BASE_DIR', os.path.dirname(os.path.abspath(__file__)))
 
+# Data directory
+DATA_DIR = getvalue('DATA_DIR', os.path.join(BASE_DIR, '..', 'data'))
+
 # Machine translation API keys
 
 # Apertium Web Service, register at http://api.apertium.org/register.jsp
@@ -57,9 +60,6 @@ MT_GOOGLE_KEY = getvalue('MT_GOOGLE_KEY', None)
 # tmserver URL
 MT_TMSERVER = getvalue('MT_TMSERVER', None)
 
-# Path where git repositories are stored, it needs to be writable
-GIT_ROOT = getvalue('GIT_ROOT', os.path.join(BASE_DIR, 'repos'))
-
 # Title of site to use
 SITE_TITLE = getvalue('SITE_TITLE', 'Weblate')
 
@@ -95,9 +95,6 @@ AUTO_LOCK = getvalue('AUTO_LOCK', True)
 AUTO_LOCK_TIME = getvalue('AUTO_LOCK_TIME', 60)
 LOCK_TIME = getvalue('LOCK_TIME', 15 * 60)
 
-# Where to put Whoosh index
-WHOOSH_INDEX = getvalue('WHOOSH_INDEX', os.path.join(BASE_DIR, 'whoosh-index'))
-
 # List of quality checks
 CHECK_LIST = getvalue('CHECK_LIST', (
     'weblate.trans.checks.same.SameCheck',
@@ -192,3 +189,7 @@ HIDE_REPO_CREDENTIALS = getvalue('HIDE_REPO_CREDENTIALS', True)
 
 # Whiteboard
 ENABLE_WHITEBOARD = getvalue('ENABLE_WHITEBOARD', False)
+
+# Obsolete configs, needed for data migration
+GIT_ROOT = getvalue('GIT_ROOT', os.path.join(BASE_DIR, 'repos'))
+WHOOSH_INDEX = getvalue('WHOOSH_INDEX', os.path.join(BASE_DIR, 'whoosh-index'))

weblate/settings_example.py

@@ -64,6 +64,9 @@ DATABASES = {
 
 BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 
+# Data directory
+DATA_DIR = os.path.join(BASE_DIR, '..', 'data')
+
 # Local time zone for this installation. Choices can be found here:
 # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
 # although not all choices may be available on all operating systems.
@@ -402,9 +405,6 @@ MT_GOOGLE_KEY = None
 # tmserver URL
 MT_TMSERVER = None
 
-# Path where git repositories are stored, it needs to be writable
-GIT_ROOT = os.path.join(BASE_DIR, 'repos')
-
 # Title of site to use
 SITE_TITLE = 'Weblate'
 
@@ -452,9 +452,6 @@ LOCK_TIME = 15 * 60
 # Render forms using bootstrap
 CRISPY_TEMPLATE_PACK = 'bootstrap3'
 
-# Where to put Whoosh index
-WHOOSH_INDEX = os.path.join(BASE_DIR, 'whoosh-index')
-
 # List of quality checks
 # CHECK_LIST = (
 #     'weblate.trans.checks.same.SameCheck',

weblate/settings_openshift.py

@@ -40,6 +40,8 @@ DATABASES = {
 
 BASE_DIR = os.path.dirname(os.path.abspath(__file__))
 
+DATA_DIR = os.environ['OPENSHIFT_DATA_DIR']
+
 TIME_ZONE = None
 
 STATIC_ROOT = os.path.join(BASE_DIR, '..', 'wsgi', 'static')

weblate/settings_test.py

@@ -46,7 +46,7 @@ if 'CI_DATABASE' in os.environ:
 ADMINS = (('Weblate test', 'noreply@weblate.org'), )
 
 # Different root for test repos
-GIT_ROOT = os.path.join(BASE_DIR, 'test-repos')
+DATA_DIR = os.path.join(BASE_DIR, '..', 'data-test')
 
 # Avoid migrating during testsuite
 SOUTH_TESTS_MIGRATE = False

weblate/trans/data.py

+# -*- coding: utf-8 -*-
+#
+# Copyright © 2012 - 2014 Michal Čihař <michal@cihar.com>
+#
+# This file is part of Weblate <http://weblate.org/>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+"""
+Data files helpers.
+"""
+import shutil
+import os
+from weblate.appsettings import DATA_DIR, GIT_ROOT, WHOOSH_INDEX
+
+
+def check_data_writable():
+    """
+    Check we can write to data dir.
+    """
+    if not os.path.exists(DATA_DIR):
+        os.makedirs(DATA_DIR)
+    else:
+        if not os.access(DATA_DIR, os.W_OK):
+            raise OSError('DATA_DIR {0} is not writable!'.format(DATA_DIR))
+
+
+def data_dir(component):
+    """
+    Returns path to data dir for given component.
+    """
+    return os.path.join(DATA_DIR, component)
+
+
+def migrate_data_dirs(*args, **kwargs):
+    """
+    Migrate data directory from old locations to new consolidated data directory.
+    """
+    check_data_writable()
+
+    vcs = data_dir('vcs')
+    if os.path.exists(GIT_ROOT) and not os.path.exists(vcs):
+        shutil.move(GIT_ROOT, vcs)
+
+    whoosh = data_dir('whoosh')
+    if os.path.exists(WHOOSH_INDEX) and not os.path.exists(whoosh):
+        shutil.move(WHOOSH_INDEX, whoosh)
+
+def unmigrate_data_dirs(*args, **kwargs):
+    vcs = data_dir('vcs')
+    if not os.path.exists(GIT_ROOT) and os.path.exists(vcs):
+        shutil.move(vcs, GIT_ROOT)
+
+    whoosh = data_dir('whoosh')
+    # This one gets autocreated
+    if os.path.exists(WHOOSH_INDEX):
+        os.rmdir(WHOOSH_INDEX)
+    if os.path.exists(whoosh):
+        shutil.move(whoosh, WHOOSH_INDEX)

weblate/trans/migrations/0014_auto_20141202_1101.py

+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations
+from weblate.trans.data import migrate_data_dirs, unmigrate_data_dirs
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('trans', '0013_auto_20141124_1036'),
+    ]
+
+    operations = [
+        migrations.RunPython(
+            migrate_data_dirs,
+            reverse_code=unmigrate_data_dirs,
+        ),
+    ]

weblate/trans/models/project.py

@@ -19,7 +19,6 @@
 #
 
 from django.db import models
-from weblate import appsettings
 from django.utils.translation import ugettext as _, ugettext_lazy
 from django.core.exceptions import ValidationError, PermissionDenied
 from django.contrib import messages
@@ -33,6 +32,7 @@ import os.path
 from weblate.lang.models import Language
 from weblate.trans.mixins import PercentMixin, URLMixin, PathMixin
 from weblate.trans.util import get_site_url
+from weblate.trans.data import data_dir
 
 
 class ProjectManager(models.Manager):
@@ -237,7 +237,7 @@ class Project(models.Model, PercentMixin, URLMixin, PathMixin):
         return max([subproject.locked for subproject in subprojects])
 
     def _get_path(self):
-        return os.path.join(appsettings.GIT_ROOT, self.slug)
+        return os.path.join(data_dir('vcs'), self.slug)
 
     def __unicode__(self):
         return self.name

weblate/trans/search.py

@@ -32,8 +32,9 @@ from weblate import appsettings
 from whoosh.writing import AsyncWriter, BufferedWriter
 from django.dispatch import receiver
 from weblate.lang.models import Language
+from weblate.trans.data import data_dir
 
-STORAGE = FileStorage(appsettings.WHOOSH_INDEX)
+STORAGE = FileStorage(data_dir('whoosh'))
 
 
 class TargetSchema(SchemaClass):

weblate/trans/tests/test_models.py

@@ -46,30 +46,25 @@ class RepoTestCase(TestCase):
     Generic class for tests working with repositories.
     """
     def setUp(self):
-        if 'test-repos' in settings.GIT_ROOT:
-            test_dir = os.path.join(settings.GIT_ROOT, 'test')
-            if os.path.exists(test_dir):
-                shutil.rmtree(test_dir)
-
         # Path where to clone remote repo for tests
         self.git_base_repo_path = os.path.join(
-            settings.GIT_ROOT,
+            settings.DATA_DIR,
             'test-base-repo.git'
         )
         # Repository on which tests will be performed
         self.git_repo_path = os.path.join(
-            settings.GIT_ROOT,
+            settings.DATA_DIR,
             'test-repo.git'
         )
 
         # Path where to clone remote repo for tests
         self.hg_base_repo_path = os.path.join(
-            settings.GIT_ROOT,
+            settings.DATA_DIR,
             'test-base-repo.hg'
         )
         # Repository on which tests will be performed
         self.hg_repo_path = os.path.join(
-            settings.GIT_ROOT,
+            settings.DATA_DIR,
             'test-repo.hg'
         )
 
@@ -104,10 +99,11 @@ class RepoTestCase(TestCase):
         shutil.copytree(self.hg_base_repo_path, self.hg_repo_path)
 
         # Remove possibly existing project directory
-        test_repo_path = os.path.join(settings.GIT_ROOT, 'test')
+        test_repo_path = os.path.join(settings.DATA_DIR, 'vcs', 'test')
         if os.path.exists(test_repo_path):
             shutil.rmtree(test_repo_path)
 
+
     def create_project(self):
         """
         Creates test project.
@@ -280,8 +276,8 @@ class ProjectTest(RepoTestCase):
     def test_wrong_path(self):
         project = self.create_project()
 
-        backup = appsettings.GIT_ROOT
-        appsettings.GIT_ROOT = '/weblate-nonexisting-path'
+        backup = appsettings.DATA_DIR
+        appsettings.DATA_DIR = '/weblate-nonexisting-path'
 
         # Invalidate cache, pylint: disable=W0212
         project._dir_path = None
@@ -292,7 +288,7 @@ class ProjectTest(RepoTestCase):
             project.full_clean
         )
 
-        appsettings.GIT_ROOT = backup
+        appsettings.DATA_DIR = backup
 
     def test_acl(self):
         """