Commit caf606b0 authored by Jim Fulton's avatar Jim Fulton

Previously, database connections were managed as a stack. This

tended to cause the same connection(s) to be used over and over.
For example, the most used conection would typically be the onlyt
connection used.  In some rare situations, extra connections could
be opened and end up on the top of the stack, causing extreme memory
wastage.  Now, when connections are placed on the stack, they sink
below existing connections that have more active objects.
parent a9c20004
...@@ -55,6 +55,14 @@ New Features ...@@ -55,6 +55,14 @@ New Features
starting point. This enhancement makes it practical to take starting point. This enhancement makes it practical to take
advantage of the new storage server invalidation-age option. advantage of the new storage server invalidation-age option.
- Previously, database connections were managed as a stack. This
tended to cause the same connection(s) to be used over and over.
For example, the most used conection would typically be the onlyt
connection used. In some rare situations, extra connections could
be opened and end up on the top of the stack, causing extreme memory
wastage. Now, when connections are placed on the stack, they sink
below existing connections that have more active objects.
3.9.0a8 (2008-12-15) 3.9.0a8 (2008-12-15)
==================== ====================
......
...@@ -122,6 +122,21 @@ class ConnectionPool(AbstractConnectionPool): ...@@ -122,6 +122,21 @@ class ConnectionPool(AbstractConnectionPool):
# in this stack. # in this stack.
self.available = [] self.available = []
def _append(self, c):
available = self.available
cactive = c._cache.cache_non_ghost_count
if (available and
(available[-1][1]._cache.cache_non_ghost_count > cactive)
):
i = len(available) - 1
while (i and
(available[i-1][1]._cache.cache_non_ghost_count > cactive)
):
i -= 1
available.insert(i, (time.time(), c))
else:
available.append((time.time(), c))
def push(self, c): def push(self, c):
"""Register a new available connection. """Register a new available connection.
...@@ -132,7 +147,7 @@ class ConnectionPool(AbstractConnectionPool): ...@@ -132,7 +147,7 @@ class ConnectionPool(AbstractConnectionPool):
assert c not in self.available assert c not in self.available
self._reduce_size(strictly_less=True) self._reduce_size(strictly_less=True)
self.all.add(c) self.all.add(c)
self.available.append((time.time(), c)) self._append(c)
n = len(self.all) n = len(self.all)
limit = self.size limit = self.size
if n > limit: if n > limit:
...@@ -151,7 +166,7 @@ class ConnectionPool(AbstractConnectionPool): ...@@ -151,7 +166,7 @@ class ConnectionPool(AbstractConnectionPool):
assert c in self.all assert c in self.all
assert c not in self.available assert c not in self.available
self._reduce_size(strictly_less=True) self._reduce_size(strictly_less=True)
self.available.append((time.time(), c)) self._append(c)
def _reduce_size(self, strictly_less=False): def _reduce_size(self, strictly_less=False):
"""Throw away the oldest available connections until we're under our """Throw away the oldest available connections until we're under our
...@@ -210,7 +225,7 @@ class ConnectionPool(AbstractConnectionPool): ...@@ -210,7 +225,7 @@ class ConnectionPool(AbstractConnectionPool):
def availableGC(self): def availableGC(self):
"""Perform garbage collection on available connections. """Perform garbage collection on available connections.
If a connection is no longer viable because it has timed out, it is If a connection is no longer viable because it has timed out, it is
garbage collected.""" garbage collected."""
threshhold = time.time() - self.timeout threshhold = time.time() - self.timeout
......
...@@ -173,6 +173,47 @@ be reused by the next open(): ...@@ -173,6 +173,47 @@ be reused by the next open():
>>> len(pool.available), len(pool.all) >>> len(pool.available), len(pool.all)
(0, 3) (0, 3)
It's a bit more complicated though. The connection pool tries to keep
connections with larger caches at the top of the stack. It does this
by having connections with larger caches "sink" below connections with
larger caches when they are closed.
To see this, we'll add some objects to the caches:
>>> for i in range(10):
... c1.root()[i] = c1.root().__class__()
>>> import transaction
>>> transaction.commit()
>>> c1._cache.cache_non_ghost_count
11
>>> for i in range(5):
... _ = len(c2.root()[i])
>>> c2._cache.cache_non_ghost_count
6
Now, we'll close the connections and get them back:
>>> c1.close()
>>> c2.close()
>>> c3.close()
We closed c3 last, but c1 is the biggest, so we get c1 on the next
open:
>>> db.open() is c1
True
Similarly, c2 is the next buggest, so we get that next:
>>> db.open() is c2
True
and finally c3:
>>> db.open() is c3
True
What about the 3 in pool.all? We've seen that closing connections doesn't What about the 3 in pool.all? We've seen that closing connections doesn't
reduce pool.all, and it would be bad if DB kept connections alive forever. reduce pool.all, and it would be bad if DB kept connections alive forever.
...@@ -295,8 +336,13 @@ resources (like RDB connections), for the duration. ...@@ -295,8 +336,13 @@ resources (like RDB connections), for the duration.
Now open more connections so that the total exceeds pool_size (2): Now open more connections so that the total exceeds pool_size (2):
>>> conn1 = db.open() >>> conn1 = db.open(); _ = conn1.root()['a']
>>> conn2 = db.open() >>> conn2 = db.open(); _ = conn2.root()['a']
Note that we accessed the objects in the new connections so they would
be of the same size, so that when they get closed, they don't sink
below conn0.
>>> pool = db.pool >>> pool = db.pool
>>> len(pool.all), len(pool.available) # all Connections are in use >>> len(pool.all), len(pool.available) # all Connections are in use
(3, 0) (3, 0)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment