Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
M
mitogen
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Labels
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Commits
Open sidebar
nexedi
mitogen
Commits
83f8f186
Commit
83f8f186
authored
Sep 29, 2017
by
David Wilson
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
docs: fix pickler docs, begin relabelling master/slave->parent/child
parent
4327baab
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
39 additions
and
41 deletions
+39
-41
docs/howitworks.rst
docs/howitworks.rst
+39
-41
No files found.
docs/howitworks.rst
View file @
83f8f186
...
...
@@ -54,7 +54,7 @@ pipe, and the child half writes the string ``EC0\n``, then begins reading the
writing
the
decompressed
result
to
the
write
-
end
of
the
UNIX
pipe
.
To
allow
recovery
of
``
stdin
``
for
reuse
by
the
bootstrapped
process
for
master
<->
slave
communication
,
it
is
necessary
for
the
first
stage
to
avoid
parent
<->
child
communication
,
it
is
necessary
for
the
first
stage
to
avoid
closing
``
stdin
``
or
reading
from
it
until
until
EOF
.
Therefore
,
the
master
sends
the
:
py
:
mod
:`
zlib
`-
compressed
payload
prefixed
with
an
integer
size
,
allowing
reading
by
the
first
stage
of
exactly
the
required
bytes
.
...
...
@@ -100,10 +100,10 @@ Preserving The `mitogen.core` Source
####################################
One
final
trick
is
implemented
in
the
first
stage
:
after
bootstrapping
the
new
slave
,
it
writes
a
duplicate
copy
of
the
:
py
:
mod
:`
mitogen
.
core
`
source
it
just
used
to
bootstrap
it
back
into
another
pipe
connected
to
the
slave
.
The
slave
's
child
,
it
writes
a
duplicate
copy
of
the
:
py
:
mod
:`
mitogen
.
core
`
source
it
just
used
to
bootstrap
it
back
into
another
pipe
connected
to
the
child
.
The
child
's
module importer cache is initialized with a copy of the source, so that
subsequent bootstraps of
slave-of-slaves
do not require the source to be
subsequent bootstraps of
children-of-children
do not require the source to be
fetched from the master a second time.
...
...
@@ -129,7 +129,7 @@ Generating A Synthetic `mitogen` Package
Since
the
bootstrap
consists
of
the
:
py
:
mod
:`
mitogen
.
core
`
source
code
,
and
this
code
is
loaded
by
Python
by
way
of
its
main
script
(``
__main__
``
module
),
initially
the
module
layout
in
the
slave
will
be
incorrect
.
initially
the
module
layout
in
the
child
will
be
incorrect
.
The
first
step
taken
after
bootstrap
is
to
rearrange
:
py
:
data
:`
sys
.
modules
`
slightly
so
that
:
py
:
mod
:`
mitogen
.
core
`
appears
in
the
correct
location
,
and
all
...
...
@@ -158,12 +158,12 @@ it. Therefore the bootstrap must call :py:func:`os.wait` soon after startup.
Setup Logging
#############
The
slave
'
s
:
py
:
mod
:`
logging
`
package
root
logger
is
configured
to
have
the
The
child
'
s
:
py
:
mod
:`
logging
`
package
root
logger
is
configured
to
have
the
same
log
level
as
the
root
logger
in
the
master
,
and
:
py
:
class
:`
mitogen
.
core
.
LogHandler
`
is
installed
to
forward
logs
to
the
master
context
's :py:data:`FORWARD_LOG <mitogen.core.FORWARD_LOG>` handle.
The log level is copied into the
slave
to avoid generating a potentially large
The log level is copied into the
child
to avoid generating a potentially large
amount of network IO forwarding logs that will simply be filtered away once
they reach the master.
...
...
@@ -185,8 +185,8 @@ end is added to the IO multiplexer, and whose write end is used to overwrite
the
handles
inherited
during
process
creation
.
Even
without
IO
redirection
,
something
must
replace
``
stdin
``
and
``
stdout
``,
otherwise
it
is
possible
for
the
stream
used
for
communication
between
the
master
and
slave
to
be
accidentally
corrupted
by
subprocesses
run
by
user
code
.
otherwise
it
is
possible
for
the
stream
used
for
communication
between
parent
and
child
to
be
accidentally
corrupted
by
subprocesses
run
by
user
code
.
The
inherited
``
stdin
``
is
replaced
by
a
file
descriptor
pointing
to
``/
dev
/
null
``.
...
...
@@ -198,7 +198,7 @@ active, so that ``print`` statements and suchlike promptly appear in the logs.
Function Call Dispatch
######################
After all initialization is complete, the
slave
'
s
main
thread
sits
in
a
loop
After all initialization is complete, the
child
'
s
main
thread
sits
in
a
loop
reading
from
a
:
py
:
class
:`
Channel
<
mitogen
.
core
.
Channel
>`
connected
to
the
:
py
:
data
:`
CALL_FUNCTION
<
mitogen
.
core
.
CALL_FUNCTION
>`
handle
.
This
handle
is
written
to
by
...
...
@@ -211,14 +211,14 @@ Shutdown
When
the
master
signals
the
:
py
:
data
:`
CALL_FUNCTION
<
mitogen
.
core
.
CALL_FUNCTION
>`
:
py
:
class
:`
Channel
<
mitogen
.
core
.
Channel
>`
is
closed
,
the
slave
calls
:
py
:
meth
:`
shutdown
()
<
mitogen
.
core
.
Broker
.
shutdown
>`
closed
,
the
child
calls
:
py
:
meth
:`
shutdown
()
<
mitogen
.
core
.
Broker
.
shutdown
>`
followed
by
:
py
:
meth
:`
wait
()
<
mitogen
.
core
.
Broker
.
wait
>`
on
its
own
broker
,
triggering
graceful
shutdown
.
During
shutdown
,
the
master
will
wait
a
few
seconds
for
slaves
to
disconnect
gracefully
before
force
disconnecting
them
,
while
the
slaves
will
use
that
time
t
o
call
:
py
:
meth
:`
socket
.
shutdown
(
SHUT_WR
)
<
socket
.
socket
.
shutdown
>`
on
their
:
py
:
class
:`
IoLogger
<
mitogen
.
core
.
IoLogger
>`
socket
's write ends before
During
shutdown
,
the
master
will
wait
a
few
seconds
for
children
to
disconnect
gracefully
before
force
disconnecting
them
,
while
the
children
will
use
that
t
ime
to
call
:
py
:
meth
:`
socket
.
shutdown
(
SHUT_WR
)
<
socket
.
socket
.
shutdown
>`
on
their
:
py
:
class
:`
IoLogger
<
mitogen
.
core
.
IoLogger
>`
socket
's write ends before
draining any remaining data buffered on the read ends.
An alternative approach is to wait until the socket is completely closed, with
...
...
@@ -239,7 +239,7 @@ Stream Protocol
---------------
Once connected, a basic framing protocol is used to communicate between
master and slave
:
parent and child
:
+--------------------+------+------------------------------------------------------+
| Field | Size | Description |
...
...
@@ -273,13 +273,13 @@ Masters listen on the following handles:
..
data
::
mitogen
.
core
.
ALLOCATE_ID
Replies
to
any
message
sent
to
it
with
a
newly
allocated
unique
context
ID
,
to
allow
slaves
to
safely
start
their
own
contexts
.
In
future
this
is
to
allow
children
to
safely
start
their
own
contexts
.
In
future
this
is
likely
to
be
replaced
by
32
-
bit
context
IDs
and
random
allocation
,
with
an
improved
``
ADD_ROUTE
``
message
sent
upstream
rather
than
downstream
that
generates
NACKs
if
any
ancestor
already
knows
the
ID
.
Slaves
listen
on
the
following
handles
:
Children
listen
on
the
following
handles
:
..
data
::
mitogen
.
core
.
CALL_FUNCTION
...
...
@@ -290,9 +290,9 @@ Slaves listen on the following handles:
`
class_name
.
func_name
(\*
args
,
\**
kwargs
)`.
When
this
channel
is
closed
(
by
way
of
sending
``
_DEAD
``
to
it
),
the
slave
's main thread begins graceful shutdown of its own `Broker` and
`Router`. Each
slave
is responsible for sending ``_DEAD`` to each of its
directly connected
slaves
in response to the master sending ``_DEAD`` to
child
's main thread begins graceful shutdown of its own `Broker` and
`Router`. Each
child
is responsible for sending ``_DEAD`` to each of its
directly connected
children
in response to the master sending ``_DEAD`` to
it, and arranging for the connection to its parent context to be closed
shortly thereafter.
...
...
@@ -310,7 +310,7 @@ Slaves listen on the following handles:
Given a chain `master -> ssh1 -> sudo1`, no `ADD_ROUTE` message is
necessary, since :py:class:`mitogen.core.Router` in the `ssh` context can
arrange to update its routes while setting up the new
slave
during
arrange to update its routes while setting up the new
child
during
`proxy_connect()`.
However, given a chain like `master -> ssh1 -> sudo1 -> ssh2 -> sudo2`,
...
...
@@ -319,8 +319,8 @@ Slaves listen on the following handles:
establishment.
Slaves that have ever been used to create a descendent child context also
listen on
the following handles:
Children that have ever been used to create a descendent child also listen on
the following handles:
.. data:: mitogen.core.GET_MODULE
...
...
@@ -328,7 +328,7 @@ listen on the following handles:
(:
py
:
class
:`
mitogen
.
master
.
ModuleForwarder
`)
serves
responses
using
:
py
:
class
:`
mitogen
.
core
.
Importer
`
's cache before forwarding the request to
its parent context. The response is cached by each context in turn before
being forwarded on to the
slave
context that originally made the request.
being forwarded on to the
child
context that originally made the request.
In this way, the master need never re-send a module it has already sent to
a direct descendant.
...
...
@@ -352,17 +352,16 @@ still registered with :py:meth:`add_handler()
Use of Pickle
#############
The current implementation uses the Python :py:mod:`cPickle` module, with
mitigations to prevent untrusted slaves from triggering code excution in the
master. The primary reason for using :py:mod:`cPickle` is that it is
computationally efficient, and avoids including a potentially large body of
serialization code
in the bootstrap.
The current implementation uses the Python :py:mod:`cPickle` module, with
a
restrictive class whitelist to prevent triggering undesirable code execution.
The primary reason for using :py:mod:`cPickle` is that it is computationally
efficient, and avoids including a potentially large body of serialization code
in the bootstrap.
The pickler active in slave contexts will instantiate any class, however in the
master it is initially restricted to only permitting
:py:class:`CallError <mitogen.core.CallError>` and :py:data:`_DEAD
<mitogen.core._DEAD>`. While not recommended, it is possible to register more
using :py:meth:`mitogen.master.LocalStream.allow_class`.
The pickler will instantiate only built-in types and one of 3 constructor
functions, to support unpickling :py:class:`CallError
<mitogen.core.CallError>`, :py:data:`_DEAD <mitogen.core._DEAD>`, and
:py:class:`Context <mitogen.core.Context>`.
The choice of Pickle is one area to be revisited later. All accounts suggest it
cannot be used securely, however few of those accounts appear to be expert, and
...
...
@@ -440,12 +439,12 @@ a new child context, but that is okay for now, since child contexts cannot
currently allocate new context IDs anyway.
Differences Between Master And
Slave
Brokers
Differences Between Master And
Child
Brokers
############################################
The main difference between :py:class:`mitogen.core.Broker` and
:py:class:`mitogen.master.Broker` is that when the stream connection to the
parent is lost in a
slave
, the broker will trigger its own shutdown.
parent is lost in a
child
, the broker will trigger its own shutdown.
The Module Importer
...
...
@@ -512,7 +511,6 @@ mechanism is not portable to non-UNIX operating systems, and does not work in
every
case
,
for
example
when
Python
blocks
signals
during
a
variety
of
:
py
:
mod
:`
threading
`
package
operations
.
At
some
point
it
is
likely
Mitogen
will
be
extended
to
support
starting
slaves
running
on
Windows
.
When
that
happens
,
it
would
be
nice
if
the
process
model
on
Windows
and
UNIX
did
not
differ
,
and
in
fact
the
code
used
on
both
were
identical
.
At
some
point
it
is
likely
Mitogen
will
be
extended
to
support
children
running
on
Windows
.
When
that
happens
,
it
would
be
nice
if
the
process
model
on
Windows
and
UNIX
did
not
differ
,
and
in
fact
the
code
used
on
both
were
identical
.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment