Commit 023e33f7 by Yoshinori Okuji

Rewrite in Python

1 parent bad49bcd
Aur?lien Calonne <aurel@nexedi.com>
Yoshinori Okuji <yo@nexedi.com>
2005-07-01 Aurelien Calonne <aurel@nexedi.com>
* src/master_node.c (h_beginTrans):
Copyright (C) 1994, 1995, 1996, 1999, 2000, 2001, 2002 Free Software
Foundation, Inc.
This file is free documentation; the Free Software Foundation gives
unlimited permission to copy, distribute and modify it.
Basic Installation
==================
These are generic installation instructions.
The `configure' shell script attempts to guess correct values for
various system-dependent variables used during compilation. It uses
those values to create a `Makefile' in each directory of the package.
It may also create one or more `.h' files containing system-dependent
definitions. Finally, it creates a shell script `config.status' that
you can run in the future to recreate the current configuration, and a
file `config.log' containing compiler output (useful mainly for
debugging `configure').
It can also use an optional file (typically called `config.cache'
and enabled with `--cache-file=config.cache' or simply `-C') that saves
the results of its tests to speed up reconfiguring. (Caching is
disabled by default to prevent problems with accidental use of stale
cache files.)
If you need to do unusual things to compile the package, please try
to figure out how `configure' could check whether to do them, and mail
diffs or instructions to the address given in the `README' so they can
be considered for the next release. If you are using the cache, and at
some point `config.cache' contains results you don't want to keep, you
may remove or edit it.
The file `configure.ac' (or `configure.in') is used to create
`configure' by a program called `autoconf'. You only need
`configure.ac' if you want to change it or regenerate `configure' using
a newer version of `autoconf'.
The simplest way to compile this package is:
1. `cd' to the directory containing the package's source code and type
`./configure' to configure the package for your system. If you're
using `csh' on an old version of System V, you might need to type
`sh ./configure' instead to prevent `csh' from trying to execute
`configure' itself.
Running `configure' takes awhile. While running, it prints some
messages telling which features it is checking for.
2. Type `make' to compile the package.
3. Optionally, type `make check' to run any self-tests that come with
the package.
4. Type `make install' to install the programs and any data files and
documentation.
5. You can remove the program binaries and object files from the
source code directory by typing `make clean'. To also remove the
files that `configure' created (so you can compile the package for
a different kind of computer), type `make distclean'. There is
also a `make maintainer-clean' target, but that is intended mainly
for the package's developers. If you use it, you may have to get
all sorts of other programs in order to regenerate files that came
with the distribution.
Compilers and Options
=====================
Some systems require unusual options for compilation or linking that
the `configure' script does not know about. Run `./configure --help'
for details on some of the pertinent environment variables.
You can give `configure' initial values for configuration parameters
by setting variables in the command line or in the environment. Here
is an example:
./configure CC=c89 CFLAGS=-O2 LIBS=-lposix
*Note Defining Variables::, for more details.
Compiling For Multiple Architectures
====================================
You can compile the package for more than one kind of computer at the
same time, by placing the object files for each architecture in their
own directory. To do this, you must use a version of `make' that
supports the `VPATH' variable, such as GNU `make'. `cd' to the
directory where you want the object files and executables to go and run
the `configure' script. `configure' automatically checks for the
source code in the directory that `configure' is in and in `..'.
If you have to use a `make' that does not support the `VPATH'
variable, you have to compile the package for one architecture at a
time in the source code directory. After you have installed the
package for one architecture, use `make distclean' before reconfiguring
for another architecture.
Installation Names
==================
By default, `make install' will install the package's files in
`/usr/local/bin', `/usr/local/man', etc. You can specify an
installation prefix other than `/usr/local' by giving `configure' the
option `--prefix=PATH'.
You can specify separate installation prefixes for
architecture-specific files and architecture-independent files. If you
give `configure' the option `--exec-prefix=PATH', the package will use
PATH as the prefix for installing programs and libraries.
Documentation and other data files will still use the regular prefix.
In addition, if you use an unusual directory layout you can give
options like `--bindir=PATH' to specify different values for particular
kinds of files. Run `configure --help' for a list of the directories
you can set and what kinds of files go in them.
If the package supports it, you can cause programs to be installed
with an extra prefix or suffix on their names by giving `configure' the
option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
Optional Features
=================
Some packages pay attention to `--enable-FEATURE' options to
`configure', where FEATURE indicates an optional part of the package.
They may also pay attention to `--with-PACKAGE' options, where PACKAGE
is something like `gnu-as' or `x' (for the X Window System). The
`README' should mention any `--enable-' and `--with-' options that the
package recognizes.
For packages that use the X Window System, `configure' can usually
find the X include and library files automatically, but if it doesn't,
you can use the `configure' options `--x-includes=DIR' and
`--x-libraries=DIR' to specify their locations.
Specifying the System Type
==========================
There may be some features `configure' cannot figure out
automatically, but needs to determine by the type of machine the package
will run on. Usually, assuming the package is built to be run on the
_same_ architectures, `configure' can figure that out, but if it prints
a message saying it cannot guess the machine type, give it the
`--build=TYPE' option. TYPE can either be a short name for the system
type, such as `sun4', or a canonical name which has the form:
CPU-COMPANY-SYSTEM
where SYSTEM can have one of these forms:
OS KERNEL-OS
See the file `config.sub' for the possible values of each field. If
`config.sub' isn't included in this package, then this package doesn't
need to know the machine type.
If you are _building_ compiler tools for cross-compiling, you should
use the `--target=TYPE' option to select the type of system they will
produce code for.
If you want to _use_ a cross compiler, that generates code for a
platform different from the build platform, you should specify the
"host" platform (i.e., that on which the generated programs will
eventually be run) with `--host=TYPE'.
Sharing Defaults
================
If you want to set default values for `configure' scripts to share,
you can create a site shell script called `config.site' that gives
default values for variables like `CC', `cache_file', and `prefix'.
`configure' looks for `PREFIX/share/config.site' if it exists, then
`PREFIX/etc/config.site' if it exists. Or, you can set the
`CONFIG_SITE' environment variable to the location of the site script.
A warning: not all `configure' scripts look for a site script.
Defining Variables
==================
Variables not defined in a site shell script can be set in the
environment passed to `configure'. However, some packages may run
configure again during the build, and the customized values of these
variables may be lost. In order to avoid this problem, you should set
them in the `configure' command line, using `VAR=value'. For example:
./configure CC=/usr/local2/bin/gcc
will cause the specified gcc to be used as the C compiler (unless it is
overridden in the site shell script).
`configure' Invocation
======================
`configure' recognizes the following options to control how it
operates.
`--help'
`-h'
Print a summary of the options to `configure', and exit.
`--version'
`-V'
Print the version of Autoconf used to generate the `configure'
script, and exit.
`--cache-file=FILE'
Enable the cache: use and save the results of the tests in FILE,
traditionally `config.cache'. FILE defaults to `/dev/null' to
disable caching.
`--config-cache'
`-C'
Alias for `--cache-file=config.cache'.
`--quiet'
`--silent'
`-q'
Do not print messages saying which checks are being made. To
suppress all normal output, redirect it to `/dev/null' (any error
messages will still be shown).
`--srcdir=DIR'
Look for the package's source code in directory DIR. Usually
`configure' can determine that directory automatically.
`configure' also accepts some other, not widely useful, options. Run
`configure --help' for more details.
SUBDIRS = include lib master storage test doc client
File mode changed
Nexedi Enterprise Objects
This is the second prototype of NEO. The design is described in:
This implementation is based on the protocol version 2,
which is documented in doc/protocol.txt.
http://www.nexedi.org/workspaces/neo
NEO must be rewritten based on the protocol version 3.0,
which is being documented in doc/spec.stx. The specification
is being worked on, and requires more effort to finish.
TODO
- Handling connection timeouts.
- Handling write timeouts.
#! /bin/sh
set -x
aclocal
autoheader
automake -a -c
autoconf
PYTHON_PY_FILES = __init__.py config.py neostorage.py mq.py
PYTHON_C_FILES = neoclient.c neoclient_wrap.c
ZOPE_DATA_FILES = component.xml
# XXX this should be configurable.
NEO_DIR = /usr/lib/zope/lib/python/NEO
EXTRA_DIST = $(PYTHON_PY_FILES) $(PYTHON_C_FILES) $(ZOPE_DATA_FILES)
PYTHON_EXTENSIONS = neoclient$(PYTHON_SO)
neoclient_OBJ_FILES = neoclient_wrap.$(OBJEXT) neoclient.$(OBJEXT)
BUILT_SOURCES = $(PYTHON_EXTENSIONS)
CLEANFILES = $(PYTHON_EXTENSIONS) $(neoclient_OBJ_FILES)
PYTHON_CCOMPILE = $(PYTHON_CC) $(PYTHON_CCSHARED) $(PYTHON_BASECFLAGS) \
$(PYTHON_OPT) -Wall -W -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include
neoclient$(PYTHON_SO): $(neoclient_OBJ_FILES) ../../lib/libneo.a
$(PYTHON_LDSHARED) -o $@ $(neoclient_OBJ_FILES) -L../../lib -lneo
neoclient_wrap.$(OBJEXT): neoclient_wrap.c
$(PYTHON_CCOMPILE) -c $< -o $@
neoclient.$(OBJEXT): neoclient.c
$(PYTHON_CCOMPILE) -c $< -o $@
install-data-local: $(PYTHON_EXTENSIONS) $(PYTHON_PY_FILES) $(ZOPE_DATA_FILES)
$(mkinstalldirs) $(DESTDIR)$(NEO_DIR)
for file in $?; do $(INSTALL_DATA) $$file $(DESTDIR)$(NEO_DIR); done
uninstall-local:
for file in $(PYTHON_EXTENSIONS) $(PYTHON_PY_FILES) $(ZOPE_DATA_FILES); do \
rm -f $(DESTDIR)$(NEO_DIR)/$$file; \
done
# Makefile.in generated by automake 1.7.9 from Makefile.am.
# @configure_input@
# Copyright 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003
# Free Software Foundation, Inc.
# This Makefile.in is free software; the Free Software Foundation
# gives unlimited permission to copy and/or distribute it,
# with or without modifications, as long as this notice is preserved.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY, to the extent permitted by law; without
# even the implied warranty of MERCHANTABILITY or FITNESS FOR A
# PARTICULAR PURPOSE.
@SET_MAKE@
srcdir = @srcdir@
top_srcdir = @top_srcdir@
VPATH = @srcdir@
pkgdatadir = $(datadir)/@PACKAGE@
pkglibdir = $(libdir)/@PACKAGE@
pkgincludedir = $(includedir)/@PACKAGE@
top_builddir = ../..
am__cd = CDPATH="$${ZSH_VERSION+.}$(PATH_SEPARATOR)" && cd
INSTALL = @INSTALL@
install_sh_DATA = $(install_sh) -c -m 644
install_sh_PROGRAM = $(install_sh) -c
install_sh_SCRIPT = $(install_sh) -c
INSTALL_HEADER = $(INSTALL_DATA)
transform = $(program_transform_name)
NORMAL_INSTALL = :
PRE_INSTALL = :
POST_INSTALL = :
NORMAL_UNINSTALL = :
PRE_UNINSTALL = :
POST_UNINSTALL = :
ACLOCAL = @ACLOCAL@
AMDEP_FALSE = @AMDEP_FALSE@
AMDEP_TRUE = @AMDEP_TRUE@
AMTAR = @AMTAR@
AUTOCONF = @AUTOCONF@
AUTOHEADER = @AUTOHEADER@
AUTOMAKE = @AUTOMAKE@
AWK = @AWK@
CC = @CC@
CCDEPMODE = @CCDEPMODE@
CFLAGS = @CFLAGS@
CPP = @CPP@
CPPFLAGS = @CPPFLAGS@
CYGPATH_W = @CYGPATH_W@
DEFS = @DEFS@
DEPDIR = @DEPDIR@
ECHO_C = @ECHO_C@
ECHO_N = @ECHO_N@
ECHO_T = @ECHO_T@
EGREP = @EGREP@
EXEEXT = @EXEEXT@
INSTALL_DATA = @INSTALL_DATA@
INSTALL_PROGRAM = @INSTALL_PROGRAM@
INSTALL_SCRIPT = @INSTALL_SCRIPT@
INSTALL_STRIP_PROGRAM = @INSTALL_STRIP_PROGRAM@
LDFLAGS = @LDFLAGS@
LIBOBJS = @LIBOBJS@
LIBS = @LIBS@
LTLIBOBJS = @LTLIBOBJS@
MAINT = @MAINT@
MAINTAINER_MODE_FALSE = @MAINTAINER_MODE_FALSE@
MAINTAINER_MODE_TRUE = @MAINTAINER_MODE_TRUE@
MAKEINFO = @MAKEINFO@
MYSQL_CONFIG = @MYSQL_CONFIG@
MYSQL_INCLUDE = @MYSQL_INCLUDE@
MYSQL_LDFLAGS = @MYSQL_LDFLAGS@
MYSQL_LIBS = @MYSQL_LIBS@
OBJEXT = @OBJEXT@
PACKAGE = @PACKAGE@
PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_STRING = @PACKAGE_STRING@
PACKAGE_TARNAME = @PACKAGE_TARNAME@
PACKAGE_VERSION = @PACKAGE_VERSION@
PATH_SEPARATOR = @PATH_SEPARATOR@
PYTHON = @PYTHON@
PYTHON_BASECFLAGS = @PYTHON_BASECFLAGS@
PYTHON_CC = @PYTHON_CC@
PYTHON_CCSHARED = @PYTHON_CCSHARED@
PYTHON_INCLUDE = @PYTHON_INCLUDE@
PYTHON_LDSHARED = @PYTHON_LDSHARED@
PYTHON_OPT = @PYTHON_OPT@
PYTHON_SO = @PYTHON_SO@
RANLIB = @RANLIB@
SET_MAKE = @SET_MAKE@
SHELL = @SHELL@
STRIP = @STRIP@
VERSION = @VERSION@
ac_ct_CC = @ac_ct_CC@
ac_ct_RANLIB = @ac_ct_RANLIB@
ac_ct_STRIP = @ac_ct_STRIP@
am__fastdepCC_FALSE = @am__fastdepCC_FALSE@
am__fastdepCC_TRUE = @am__fastdepCC_TRUE@
am__include = @am__include@
am__leading_dot = @am__leading_dot@
am__quote = @am__quote@
bindir = @bindir@
build_alias = @build_alias@
datadir = @datadir@
exec_prefix = @exec_prefix@
host_alias = @host_alias@
includedir = @includedir@
infodir = @infodir@
install_sh = @install_sh@
libdir = @libdir@
libexecdir = @libexecdir@
localstatedir = @localstatedir@
mandir = @mandir@
oldincludedir = @oldincludedir@
prefix = @prefix@
program_transform_name = @program_transform_name@
sbindir = @sbindir@
sharedstatedir = @sharedstatedir@
sysconfdir = @sysconfdir@
target_alias = @target_alias@
PYTHON_PY_FILES = __init__.py config.py neostorage.py mq.py
PYTHON_C_FILES = neoclient.c neoclient_wrap.c
ZOPE_DATA_FILES = component.xml
# XXX this should be configurable.
NEO_DIR = /usr/lib/zope/lib/python/NEO
EXTRA_DIST = $(PYTHON_PY_FILES) $(PYTHON_C_FILES) $(ZOPE_DATA_FILES)
PYTHON_EXTENSIONS = neoclient$(PYTHON_SO)
neoclient_OBJ_FILES = neoclient_wrap.$(OBJEXT) neoclient.$(OBJEXT)
BUILT_SOURCES = $(PYTHON_EXTENSIONS)
CLEANFILES = $(PYTHON_EXTENSIONS) $(neoclient_OBJ_FILES)
PYTHON_CCOMPILE = $(PYTHON_CC) $(PYTHON_CCSHARED) $(PYTHON_BASECFLAGS) \
$(PYTHON_OPT) -Wall -W -I$(PYTHON_INCLUDE) -I$(top_srcdir)/include
subdir = client/NEO
ACLOCAL_M4 = $(top_srcdir)/aclocal.m4
mkinstalldirs = $(SHELL) $(top_srcdir)/mkinstalldirs
CONFIG_HEADER = $(top_builddir)/config.h
CONFIG_CLEAN_FILES =
DIST_SOURCES =
DIST_COMMON = $(srcdir)/Makefile.in Makefile.am
all: $(BUILT_SOURCES)
$(MAKE) $(AM_MAKEFLAGS) all-am
.SUFFIXES:
$(srcdir)/Makefile.in: @MAINTAINER_MODE_TRUE@ Makefile.am $(top_srcdir)/configure.ac $(ACLOCAL_M4)
cd $(top_srcdir) && \
$(AUTOMAKE) --gnu client/NEO/Makefile
Makefile: @MAINTAINER_MODE_TRUE@ $(srcdir)/Makefile.in $(top_builddir)/config.status
cd $(top_builddir) && $(SHELL) ./config.status $(subdir)/$@ $(am__depfiles_maybe)
uninstall-info-am:
tags: TAGS
TAGS:
ctags: CTAGS
CTAGS:
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
top_distdir = ../..
distdir = $(top_distdir)/$(PACKAGE)-$(VERSION)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`; \
topsrcdirstrip=`echo "$(top_srcdir)" | sed 's|.|.|g'`; \
list='$(DISTFILES)'; for file in $$list; do \
case $$file in \
$(srcdir)/*) file=`echo "$$file" | sed "s|^$$srcdirstrip/||"`;; \
$(top_srcdir)/*) file=`echo "$$file" | sed "s|^$$topsrcdirstrip/|$(top_builddir)/|"`;; \
esac; \
if test -f $$file || test -d $$file; then d=.; else d=$(srcdir); fi; \
dir=`echo "$$file" | sed -e 's,/[^/]*$$,,'`; \
if test "$$dir" != "$$file" && test "$$dir" != "."; then \
dir="/$$dir"; \
$(mkinstalldirs) "$(distdir)$$dir"; \
else \
dir=''; \
fi; \
if test -d $$d/$$file; then \
if test -d $(srcdir)/$$file && test $$d != $(srcdir); then \
cp -pR $(srcdir)/$$file $(distdir)$$dir || exit 1; \
fi; \
cp -pR $$d/$$file $(distdir)$$dir || exit 1; \
else \
test -f $(distdir)/$$file \
|| cp -p $$d/$$file $(distdir)/$$file \
|| exit 1; \
fi; \
done
check-am: all-am
check: $(BUILT_SOURCES)
$(MAKE) $(AM_MAKEFLAGS) check-am
all-am: Makefile
installdirs:
install: $(BUILT_SOURCES)
$(MAKE) $(AM_MAKEFLAGS) install-am
install-exec: install-exec-am
install-data: install-data-am
uninstall: uninstall-am
install-am: all-am
@$(MAKE) $(AM_MAKEFLAGS) install-exec-am install-data-am
installcheck: installcheck-am
install-strip:
$(MAKE) $(AM_MAKEFLAGS) INSTALL_PROGRAM="$(INSTALL_STRIP_PROGRAM)" \
install_sh_PROGRAM="$(INSTALL_STRIP_PROGRAM)" INSTALL_STRIP_FLAG=-s \
`test -z '$(STRIP)' || \
echo "INSTALL_PROGRAM_ENV=STRIPPROG='$(STRIP)'"` install
mostlyclean-generic:
clean-generic:
-test -z "$(CLEANFILES)" || rm -f $(CLEANFILES)
distclean-generic:
-rm -f $(CONFIG_CLEAN_FILES)
maintainer-clean-generic:
@echo "This command is intended for maintainers to use"
@echo "it deletes files that may require special tools to rebuild."
-test -z "$(BUILT_SOURCES)" || rm -f $(BUILT_SOURCES)
clean: clean-am
clean-am: clean-generic mostlyclean-am
distclean: distclean-am
-rm -f Makefile
distclean-am: clean-am distclean-generic
dvi: dvi-am
dvi-am:
info: info-am
info-am:
install-data-am: install-data-local
install-exec-am:
install-info: install-info-am
install-man:
installcheck-am:
maintainer-clean: maintainer-clean-am
-rm -f Makefile
maintainer-clean-am: distclean-am maintainer-clean-generic
mostlyclean: mostlyclean-am
mostlyclean-am: mostlyclean-generic
pdf: pdf-am
pdf-am:
ps: ps-am
ps-am:
uninstall-am: uninstall-info-am uninstall-local
.PHONY: all all-am check check-am clean clean-generic distclean \
distclean-generic distdir dvi dvi-am info info-am install \
install-am install-data install-data-am install-data-local \
install-exec install-exec-am install-info install-info-am \
install-man install-strip installcheck installcheck-am \
installdirs maintainer-clean maintainer-clean-generic \
mostlyclean mostlyclean-generic pdf pdf-am ps ps-am uninstall \
uninstall-am uninstall-info-am uninstall-local
neoclient$(PYTHON_SO): $(neoclient_OBJ_FILES) ../../lib/libneo.a
$(PYTHON_LDSHARED) -o $@ $(neoclient_OBJ_FILES) -L../../lib -lneo
neoclient_wrap.$(OBJEXT): neoclient_wrap.c
$(PYTHON_CCOMPILE) -c $< -o $@
neoclient.$(OBJEXT): neoclient.c
$(PYTHON_CCOMPILE) -c $< -o $@
install-data-local: $(PYTHON_EXTENSIONS) $(PYTHON_PY_FILES) $(ZOPE_DATA_FILES)
$(mkinstalldirs) $(DESTDIR)$(NEO_DIR)
for file in $?; do $(INSTALL_DATA) $$file $(DESTDIR)$(NEO_DIR); done
uninstall-local:
for file in $(PYTHON_EXTENSIONS) $(PYTHON_PY_FILES) $(ZOPE_DATA_FILES); do \
rm -f $(DESTDIR)$(NEO_DIR)/$$file; \
done
# Tell versions [3.59,3.63) of GNU make to not export all variables.
# Otherwise a system limit (for SysV at least) may be exceeded.
.NOEXPORT:
<component prefix="NEO.config">
<sectiontype name="NEO" datatype=".neostorage"
implements="ZODB.storage">
<description>
A scalable storage for Zope
</description>
<key name="master" required="yes">
<description>
Give the ip of the master node
</description>
</key>
</sectiontype>
</component>
from ZODB.config import BaseConfig
class neostorage(BaseConfig):
def open(self):
from neostorage import neostorage
return neostorage(master = self.config.master)
##############################################################################
#
# Copyright (c) 2005 Nexedi SARL and Contributors. All Rights Reserved.
# Yoshinori Okuji <yo@nexedi.com>
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsability of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# garantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
"""
Multi-Queue Cache Algorithm.
"""
from math import log
class Element:
"""
This class defines an element of a FIFO buffer.
"""
pass
class FIFO:
"""
This class implements a FIFO buffer.
"""
def __init__(self):
self._head = None
self._tail = None
self._len = 0
def __len__(self):
return self._len
def append(self):
element = Element()
element.next = None
element.prev = self._tail
if self._tail is not None:
self._tail.next = element
self._tail = element
if self._head is None:
self._head = element
self._len += 1
return element
def head(self):
return self._head
def tail(self):
return self._tail
def shift(self):
element = self._head
if element is None:
return None
del self[element]
return element
def __delitem__(self, element):
if element.next is None:
self._tail = element.prev
else:
element.next.prev = element.prev
if element.prev is None:
self._head = element.next
else:
element.prev.next = element.next
self._len -= 1
class Data:
"""
Data for each element in a FIFO buffer.
"""
pass
class MQ:
"""
This class manages cached data by a variant of Multi-Queue.
This class caches various sizes of objects. Here are some considerations:
- Expired objects are not really deleted immediately. But if GC is invoked too often,
it degrades the performance significantly.
- If large objects are cached, the number of cached objects decreases. This might affect
the cache hit ratio. It might be better to tweak a buffer level according to the size of
an object.
- Stored values must be strings.
- The size calculation is not accurate.
"""
def __init__(self, life_time=10000, buffer_levels=9, max_history_size=100000, max_size=20*1024*1024):
self._history_buffer = FIFO()
self._cache_buffers = []
for level in range(buffer_levels):
self._cache_buffers.append(FIFO())
self._data = {}
self._time = 0
self._life_time = life_time
self._buffer_levels = buffer_levels
self._max_history_size = max_history_size
self._max_size = max_size
self._size = 0
def has_key(self, id):
if id in self._data:
data = self._data[id]
if data.level >= 0:
return 1
return 0
__contains__ = has_key
def fetch(self, id):
"""
Fetch a value associated with the id.
"""
if id in self._data:
data = self._data[id]
if data.level >= 0:
del self._cache_buffers[data.level][data.element]
value = data.value
self._size -= len(value) # XXX inaccurate
self.store(id, value)
return value
raise KeyError, "%s was not found in the cache" % id
__getitem__ = fetch
def get(self, id, d=None):
try:
return self.fetch(id)
except KeyError:
return d
def _evict(self, id):
"""
Evict an element to the history buffer.
"""
data = self._data[id]
self._size -= len(data.value) # XXX inaccurate
del self._cache_buffers[data.level][data.element]
element = self._history_buffer.append()
data.level = -1
data.element = element
delattr(data, 'value')
delattr(data, 'expire_time')
element.data = data
if len(self._history_buffer) > self._max_history_size:
element = self._history_buffer.shift()
del self._data[element.data.id]
def store(self, id, value):
if id in self._data:
data = self._data[id]
level, element, counter = data.level, data.element, data.counter + 1
if level >= 0:
del self._cache_buffers[level][element]
else:
del self._history_buffer[element]
else:
counter = 1
# XXX It might be better to adjust the level according to the object size.
level = int(log(counter, 2))
if level >= self._buffer_levels:
level = self._buffer_levels - 1
element = self._cache_buffers[level].append()
data = Data()
data.id = id
data.expire_time = self._time + self._life_time
data.level = level
data.element = element
data.value = value
data.counter = counter
element.data = data
self._data[id] = data
self._size += len(value) # XXX inaccurate
self._time += 1
# Expire old elements.
for level in range(self._buffer_levels):
cache_buffer = self._cache_buffers[level]
head = cache_buffer.head()
if head is not None and head.data.expire_time < self._time:
del cache_buffer[head]
data = head.data
if level > 0:
new_level = level - 1
element = cache_buffer[new_level].append()
element.data = data
data.expire_time = self._time + self._life_time
data.level = new_level
data.element = element
else:
self._evict(data.id)
# Limit the size.
size = self._size
max_size = self._max_size
if size > max_size:
for cache_buffer in self._cache_buffers:
while size > max_size:
element = cache_buffer.shift()
if element is None:
break
data = element.data
del self._data[data.id]
size -= len(data.value) # XXX inaccurate
if size <= max_size:
break
self._size = size
__setitem__ = store
def invalidate(self, id):
if id in self._data:
data = self._data[id]
if data.level >= 0:
del self._cache_buffers[data.level][data.element]
self._evict(id)
return
raise KeyError, "%s was not found in the cache" % id
__delitem__ = invalidate
# Here is a test.
if __name__ == '__main__':
cache = MQ()
cache[1] = "1"
cache[2] = "2"
assert cache.get(1) == "1", 'cannot get 1'
assert cache.get(2) == "2", 'cannot get 2'
assert cache.get(3) == None, 'can get 3!'
del cache[1]
assert cache.get(1) == None, 'can get 1!'
/* config.h.in. Generated from configure.ac by autoheader. */