CHARMING PYTHON #7
Dynamically reloading modules in long-running processes
David Mertz, Ph.D.
Mesmerist, Gnosis Software, Inc.
September 2000
One of the great advantages of Python over most other
programming languages is the extreme runtime dynamism it is
capable of. As a consequence of having the handy 'reload()'
function, we can write programs that run persistently, but
that load components that have been modified during the run
of the process. Pretty handy for services where continuous
uptime is critical. This column illustrates runtime program
modification by means of some enhancements to the Txt2Html
front-end discussed in an earlier column. Specifically,
our sample program will do a background check for new
versions of the Txt2Html conversion library on the Internet,
and download and reload any new version as needed without
manual user intervention.
WHAT IS PYTHON?
------------------------------------------------------------------------
Python is a freely available, very-high-level, interpreted
language developed by Guido van Rossum. It combines a clear
syntax with powerful (but optional) object-oriented semantics.
Python is available for almost every computer platform you
might find yourself working on, and has strong portability
between platforms.
INTRODUCTION
------------------------------------------------------------------------
Let us paint a scenario for this column: Suppose you want to
run a process on your local machine, but part of your program
logic lives somewhere else. Specifically, let us assume that
this program logic is updated from time-to-time, and when you
run your process, you would like to use the most current
program logic. There are a number of approaches to addressing
the requirement just described; this column will walk a reader
through several of them.
As this column has progressed, I have discussed ongoing
enhancements to my public-domain utility 'Txt2Html'. This
utility converts "smart ASCII" text files to HTML. Previous
columns discussed the web-proxy version of the utility and a
'curses' interface for the utility. It also happens that from
time-to-time I notice that some ASCII markup could be converted
in a more useful way, or I fix a bug in handling a particular
markup construct.
In fact, these columns are written in ASCII, and converted
during the editorial process to the HTML format you are
probably reading. Prior to sending a draft article off for
publication, I run something like the following process:
#-------- Command-line HTMLization of articles ----------#
txt2html charming_python_7.txt > charming_python_7.html
If I wanted, I could specify some flags to modify the
operation; but either way, I just rely on the fact that the
most current version of the coverter is on my local drive and
path. If I were working on a different machine, or for readers
who might want want to use the utility, the process is more
cumbersome: check my website, compare version numbers and
filedates (sometimes I make a change too small to number),
download the current version, copy current version to the right
directory, run the command-line converter.
There are several manual and moderately time consuming steps
involved in the above. It ought to be easier, and it can be.
COMMAND-LINE WEB ACCESS
------------------------------------------------------------------------
Most people think of the web as a way interactively to browse
pages in a GUI environment. Doing that is nice, of course.
But there is also a lot of power in a command-line. Systems
with the text-mode web-browser 'lynx' can largely treat the
entire WWW as just another set of files for command-line tools
to work with. For example, some commands I find useful are:
#--------- Command-line web browsing with lynx ----------#
lynx -dump http://gnosis.cx/publish/.
lynx -dump http://ibm.com/developer/. > ibm_developer.txt
lynx -dump http://gnosis.cx/publish | wc | sed "s/( *[0-9]* *\)\([0-9]*\)\(.*\)/\2/g"
The first of these says "Display David Mertz' homepage to the
console" (as ASCII text). The second says, "Save an ASCII
version of IBM's current developerWorks homepage to a file."
The third example says "Display the number of words in David's
homepage" (Don't worry about the specifics, it just shows
command-line tools being combined with pipes).
One thing about 'lynx' is that (with the '-dump' option) it
does almost exactly the opposite thing as 'Txt2Html': the
former converts HTML to text, and latter converts in the other
direction. But there is no reason not to be able to use
'Txt2Html' in the same fashion as 'lynx'. Doing so can be
accomplished with a short Python script:
#------ 'fetch_txt2html.py' command-line converter ------#
import sys
from urllib import urlopen, urlencode
if len(sys.argv) == 2:
cgi = 'http://gnosis.cx/cgi-bin/txt2html.cgi'
opts = urlencode({'source':sys.argv[1], 'proxy':'NONE'})
print urlopen(cgi, opts).read()
else:
print "Please specify URL for Txt2Html conversion"
To run this script, just do something like:
python fetch_txt2html.py http://gnosis.cx/publish/programming/charming_python_7.txt
This does not provide you with all the switches of local
'Txt2Html' process, but it would be simple to add those if
needed. You can pipe and redirect the output just as you would
with any command-line tool. However, in the above version,
you can only process data files that can be reached by URL, not
local files.
Actually, 'fetch_txt2html.py' does something 'lynx' does not
(and neither does 'Txt2Html' by itself): it not only fetches
the data source from a URL, it also gets the *program logic*
remotely. If you use 'fetch_txt2html.py' there is no need to
even -have- 'Txt2Html' on your local machine; the processing is
remotely invoked (with the latest version), and the results are
sent back to you exactly as if you had run a local process.
Neat, huh? The local version of 'Txt2Html' can access remote
URLs just as with local files, but it cannot make sure it keeps
itself up to date... yet!
DYNAMIC INITIALIZATION
------------------------------------------------------------------------
Using 'fetch_txt2html.py' assures that the latest program logic
is always used in conversions. Another thing this approach
does, however, is move the processor (and memory) requirements
onto the gnosis.cx webserver. The load imposed by this
particular process is not particularly high, but it is easy to
imagine other types of processes where processing on the client
is more efficient and desirable.
The way 'Txt2Html' is organized--like the way most programs are
organized--is with a couple core flow-control functions
assisted by a variety of utility functions. In particular, the
utility functions are the ones that I update fairly frequenty;
the core functions ('main()' and a few others) will only be
touched in the event of a major rewrite. In short, what could
helpfully be updated at each program run is the utility
functions. Most of the time, in fact, most of the functions
will be fine in the main 'Txt2Html' module [dmTxt2Html].
To reflect this organization, I have pulled out -copies- of all
the text-manipulation utility functions into a support module
called [d2h_textfuncs]. The same functions still exist in the
primary module, but the support module might contain more
up-to-date versions of functions. The [d2h_textfuncs] module
is mostly a skeleton right now, but it might be filled in over
time. Most functions are merely commented headers, real
updated program logic can be filled in as needed. The support
module looks like the below (abridged):
#----- 'd2h_textfuncs.py' dyanamic Txt2Html updates -----#
"""Hot-pluggable replacement functions for Txt2Html"""
#-- Functions to massage blocks by type
#def Titleify(block):
#def Authorify(block):
# ... [more block massaging functions] ...
#-- Utility functions for text transformation
#def AdjustCaps(txt):
#def capwords(txt):
#def URLify(txt):
def Typographify(txt):
# [module] names
r = re.compile(r"""([\(\s'/">]|^)\[(.*?)\]([<\s\.\),:;'"?!/-])""", re.M | re.S)
txt = r.sub('\\1\\2
\\3',txt)
# *strongly emphasize* words
r = re.compile(r"""([\(\s'/"]|^)\*(.*?)\*([\s\.\),:;'"?!/-])""", re.M | re.S)
txt = r.sub('\\1\\2\\3', txt)
# ... [more text massaging] ...
return txt
# ... [more text transformation functions] ...
In order to utilize the support module in its latest-and-
greatest incarnation, a few steps of preparation are necessary.
First, download the main 'Txt2Html' module to your local system
(this is a one-time step). Second, create a Python script on
your local system that reads:
#------- 'dyn_txt2html.py' command-line converter -------#
from dmTxt2Html import * # Import the body of 'Txt2Html' code
from urllib import urlopen
import sys
# Check for updated functions (fail gracefully if not fetchable)
try:
updates = urlopen('http://gnosis.cx/download/t2h_textfuncs.py').read()
fh = open('t2h_textfuncs.py', 'w')
fh.write(updates)
fh.close()
except:
sys.stderr.write('Cannot currently download Txt2Html updates')
# Import the updated functions (if available)
try:
from t2h_textfuncs import *
except:
sys.stderr.write('Cannot import the updated Txt2Html functions')
if len(sys.argv) == 2:
cfg_dict = ParseArgs(sys.argv[1:])
main(cfg_dict)
else:
print "Please specify URL (and options) for Txt2Html conversion"
A few comments are warranted concerning the 'dyn_txt2html.py'
script. Notice that when the 'from t2h_textfuncs import *'
statement is executed, all the functions (like
'Typographify()') that were defined previously in [dmTxt2Html]
are replaced by the [t2h_textfuncs] version of the same-named
functions. Of coure, where functions in [t2h_textfuncs] are
only commented-out placeholders, no replacement occurs.
One minor matter is that different systems handle
writes to STDERR differently. Under Unix-like systems, you can
redirect STDERR when you run the script; however, under my
current OS/2 shell, and under Windows/DOS, the STDERR messages
will be appended to the console output. You might want to
either write the above errors/warning to a log file, or simply
get in the habit of directing the STDOUT to a file (where it is
probably more useful anyway. For example:
#-------- Command-line session of 'dyn_txt2html' --------#
G:\txt2html> python dyn_txt2html.py test.txt > test.html
Cannot currently download Txt2Html updates
The error goes to console, the converted output to a file.
A more interesting matter is why 'dyn_txt2html.py' does not
just download the whole [dmTxt2Html] module instead of the
support module only. There are a few things going on here.
The [t2h_textfuncs] support module is significantly smaller
than the main [dmTxt2Html] module, especially since most of the
functions are stubbed/commented out. On a modem connection
this could be significantly faster. But download size is not
the main thing.
For 'Txt2Html' it probably does not matter if users
auto-download the whole latest module. But what about a system
where the program logic is *distributed* -- and especially
where the responsibility for maintenance is distributed? You
might have Alice, Bob and Charlie be responsible for modules
[Funcs_A], [Funcs_B] and [Funcs_C], respectively. Each of them
make periodic (and independent) changes to the functions under
their control, and upload the latest-and-greatest to their own
website (such as http://alice.com/Funcs_A.py). In this
scenario it is not feasable to have all three programmers make
changes to the same main module. But a script similar to
'dyn_txt2html.py' can straightforwardly be extended to try
importing [Funcs_A], [Funcs_B] and [Funcs_C] all at startup
(and fallback to [MainProg] version if these resources cannot
be obtained).
A LONG-RUNNING DYNAMIC PROCESS
------------------------------------------------------------------------
The tools we have looked at so far get their dynamic program
logic by downloading updated resources at initialization. This
makes a lot of sense for command-line or batch processes, but
what about long-running applications. Such long-running
applications are most likely to be server processes that
respond to client requests continuously. In our case, however,
we will use the 'curses_txt2html.py' script developed for a
previous column to illustrate Python's 'reload()' function.
The program 'curses_txt2html' is a wrapper for a local copy
of [dmTxt2Html]. Without trying to address 'curses' programming
a second time herein, it is enough to mention that
'curses_txt2html' provides a set of interactive menus to
configure and run multiple, sequential 'Txt2Html' conversions.
'curses_txt2html' could potentially be left running in the
background all the time, and we would like it to be able to
utilize up-to-date program logic when we switch to its session,
and run a conversion. For this specific simple example, it
admittedly would not be difficult to close and relaunch the
application, and no particular disadvantages would be incurred.
But it is easy to imagine other processes that genuinely do
depend on being left running all the time, perhaps ones that
are stateful as to action performed in a session.
For this column, a new File/Update submenu was added. When
activated, it simply calls a new function called
'update_txt2html()'. Aside from the [curses] calls that relate
to providing some confirmation of what occurred, we have mostly
already seen these steps in other examples in this column:
#----- 'curses_txt2html.py' dynamic update function -----#
def update_txt2html():
# Check for updated functions (fail gracefully if not fetchable)
s = curses.newwin(6, 60, 4, 5)
s.box()
s.addstr(1, 2, "* PRESS ANY KEY TO CONTINUE *", curses.A_BOLD)
s.addstr(3, 2, "...downloading...")
s.refresh()
try:
from urllib import urlopen
updates = urlopen('http://gnosis.cx/download/dmTxt2Html.py').read()
fh = open('dmTxt2Html.py', 'w')
fh.write(updates)
fh.close()
s.addstr(3, 2, "Module [dmTxt2Html] downloaded to current directory")
except:
s.addstr(3, 2, "Download of updated [dmTxt2Html] module failed!")
reload(dmTxt2Html)
s.addstr(4, 2, "Module [dmTxt2Html] reloaded from current directory ")
s.refresh()
c = s.getch()
s.erase()
There are two significant differences between 'dyn_txthtml.py'
and our 'update_txt2html()' function. One is that we go ahead
and import the main [dmTxt2Html] module rather than just the
support functions. This is largely just to simplify the
import. The issue here is that we use an 'import dmTxt2Html'
to access the module instead of a 'from dmTxt2Html import *'.
In many ways this is a safer procedure, but a consequence is
that it is more difficult to (accidentally or deliberately)
overwrite functions in [dmTxt2Html]. If we wanted to attach
functions from [d2h_textfuncs] we would have to do a 'dir()' on
the imported support module, and attach members to the
"dmTxt2Html" namespace in attribute fashion. Doing this style
of overwriting is left as an exercise for readers.
The most significant difference introduced by the
'update_txt2html()' function is the use of Python's built-in
'reload()' function. Just performing a brand new
'import dmTxt2Html' will *not* overwrite the functions
previously imported. Watch out for this! A lot of beginners
assume that reimporting a module will update the version in
memory. It won't. Instead, the way to update the in-memory
image of the functions in a module is to 'reload()' the module.
There is another small trick performed above. The download
location of an updated [dmTxt2Html] module is the local working
directory, which may or may not be the directory where
[dmTxt2Html] was originally loaded from. In fact, if it is in
the Python library directory, you probably will not be working
there (and probably don't have permissions to it as a user).
But the 'reload()' call tries loading from the current
directory first, then from the rest of Python's path. So
whether or not the download succeeds, the 'reload()' should be
a safe operation (it may or may not load anything new though).
RESOURCES
------------------------------------------------------------------------
The web-proxy CGI version of 'Txt2Html':
http://gnosis.cx/cgi-bin/txt2html.cgi
Support module [t2h_textfuncs]:
http://gnosis.cx/download/t2h_textfuncs.py
Main module [dmTxt2Html]:
http://gnosis.cx/download/dmTxt2Html.py
Files used and mentioned in this article:
http://gnosis.cx/download/charming_python_7.zip
ABOUT THE AUTHOR
------------------------------------------------------------------------
{Picture of Author: http://gnosis.cx/cgi-bin/img_dqm.cgi}
As David Mertz went to sleep one night after troubled
wakefulness, he dreamed himself transformed into a technical
journalist. He needs the rest. David may be reached at
mertz@gnosis.cx; his life pored over at
http://gnosis.cx/publish/. Suggestions and recommendations on
this, past, or future, columns are welcomed.