CHARMING PYTHON #18: [PyDoc] and [distutils] Improving the Social Infrastructure of Python David Mertz, Ph.D. Auteur Provocateur, Gnosis Software, Inc. June, 2001 Several modules and tools introduced in recent versions of Python have improved Python not so much as a language, but as a -tool-. The modules discussed in this column make the job of Python developers substantially easier by improving the documentation and distribution of Python modules and packages. KEYWORDS: documentation installation distribution pydoc doctest distutils Python modules packages WHAT IS PYTHON? ------------------------------------------------------------------------ Python is a freely available, very-high-level, interpreted language developed by Guido van Rossum. It combines a clear syntax with powerful (but optional) object-oriented semantics. Python is available for almost every computer platform you might find yourself working on, and has strong portability between platforms. INTRODUCTION ------------------------------------------------------------------------ One year ago, if you were to ask an honest Python evangelist if Python was missing anything important that Perl, for example, had, she would have to fess up an affirmative answer. It wasn't that Python lacked a breadth of module and package support--both Python native and extension modules. It certainly wasn't the clarity of expression or clean object orientation in which Python positively excels. What Python was missing was what Perl developers describe as "social factors." But even here, the missing social factors were not an absence of an active, intelligent and supportive Python community--Python has abounded in that. What the Python of one year ago sorely lacked was a sufficient programmatic infrastructure for the sharing of Python code. Code sharing was ad hoc, decentralized, and just plain too much work in the general case. The first step in improving the social infrastructure of Python was probably Tim Middleton's creation of the Vaults of Parnassus (see Resources). For the first time, Python developers had a single place to turn for (nearly) all contributed third-party modules, packages and tools. The Vaults still have their quirks that make them possibly less advanced (but nicer looking) than the Comprehensive Perl Archive Network. The Vaults merely point to actual resources rather than mirroring them; the Vaults are manually maintained by Middleton, and updates are sometimes slow to happen; and vex.net who generously hosts the Vaults has had intermittent outages. But overall, the service provided by the Vaults of Parnassus has been an invaluable resource in building the architectural prerequisites of a strong Python community. With such a common site available, all the Python community needed was consistent and solid ways of installing all these available modules, packages and tools; and equally clear ways of figuring out just what they did. To the rescue came several new standard modules in the standard Python distribution.. [pydoc] ------------------------------------------------------------------------ Ka-Ping Yee has created a quite remarkable module by the name [pydoc] (for the "competition": [pydoc] does everything that 'perldoc' does, but better and prettier :-)). As of Python 2.1, [pydoc] (and its supporting [inspect]) is part of the standard library. However, for users of Python 1.5.2, 1.6 or 2.0, downloading and installing [pydoc] is extremely easy--do so right away (see Resources). By way of background for any Python beginners reading this, Python has long had some semi-formal documentation standards. These standards have not attempted to contrain developers unduly, but rather simply to seem like the "one obvious way to do it." Fortunately, Python developers, as a rule, have always been far better documenters than typical developers in other languages. The main element of good Python documentation is the use of so-called "docstrings." While a docstring is really just a variable called '__doc__', there is a ubiquitous shortcut for creating them: just put a bare (triple-)quoted string at the very beginning of a module, function 'def', class definition, or method 'def'. As well, there are several more-or-less standard module-level "magic" variable names that are often used. Despite the informality of the documentation rules, almost all 3rd party and standard modules use the same patterns. Let's look at a simplistic example that has most of the elements used: #------ Module mymod.py with typical documentation ------# #!/usr/bin/python """Show off features of [pydoc] module This is a silly module to demonstrate docstrings """ __author__ = 'David Mertz' __version__= '1.0' __nonsense__ = 'jabberwocky' class MyClass: """Demonstrate class docstrings""" def __init__(self, spam=1, eggs=2): """Set default attribute values only Keyword arguments: spam -- a processed meat product eggs -- a fine breakfast for lumberjacks """ self.spam = spam self.eggs = eggs The [pydoc] module takes advantage of Python documentation conventions, as well as having some savvy about Python imports, inheritance, and the like. Moreover, [pydoc] has the absolute genius of allowing itself to be used in multiple modes of operation. More on this shortly; for a few moments I'll look at the 'manpage' style usage at an OS command-line. Let's say you had the above module [mymod] installed on your system, but weren't sure what it was for (not much in the example). You might read the source, but even easier might be: #--------- Getting 'manpage' style documentation --------# % pydoc.py mymod Python Library Documentation: module mymod NAME mymod - Show off features of [pydoc] module FILE /articles/scratch/cp18/mymod.py DESCRIPTION This is a silly module to demonstrate docstrings CLASSES MyClass class MyClass | Demonstrate class docstrings | | __init__(self, spam=1, eggs=2) | Set default attribute values only | | Keyword arguments: | spam -- a processed meat product | eggs -- a fine breakfast for lumberjacks DATA __author__ = 'David Mertz' __file__ = './mymod.pyc' __name__ = 'mymod' __nonsense__ = 'jabberwocky' __version__ = '1.0' VERSION 1.0 AUTHOR David Mertz Depending on your specific platform and setup, the above sample will probably be presented in a text viewer that allows scrolling, searching, and so on, and with some highlighting of key words. For something this simple, what you get is only slightly better than just reading the source, but consider something as as simple as: #----- Examining the inheritance structure of a class -----# % cat mymod2.py from mymod import MyClass class MyClass2(MyClass): """Child class""" def foo(self): pass % pydoc.py mymod2.MyClass2 Python Library Documentation: class MyClass2 in mymod2 class MyClass2(mymod.MyClass) | Child class | | __init__(self, spam=1, eggs=2) from mymod.MyClass | | foo(self) In this quick report we can tell not only that 'MyClass2' has the methods '__init__()' and 'foo()' (and the arguments thereto), but also which methods are implemented locally, and which other methods come from ancestors (and where those ancestors live). Another nice 'manpage' like feature is the '-k' option for searching modules for keywords. For example: #--------- Locating the right module for a task ---------# % pydoc.py -k uuencode uu - Implementation of the UUencode and UUdecode functions. % pydoc.py uu Python Library Documentation: module uu NAME uu - Implementation of the UUencode and UUdecode functions. [...] Besides its command-line usage, [pydoc] has four other "modes" that can present the same generated documentation. *Shell mode* -- Inside the Python interactive shell, you may import [pydoc]'s 'help()' function, and get assistance on any object without leaving the interactive session. You can also just type 'help' by itself to get into an interactive "help interpreter." For example: #------- Interactive shell with help enhancements -------# >>> from pydoc import help >>> import uu >>> help(uu.test) Help on function test in module uu: test() uuencode/uudecode main program >>> help Welcome to Python 2.0! This is the online help utility. [...introductory message about help shell...] help> *Webserver mode* -- Just use the '-p' option, and [pydoc] will launch itself as a simple webserver on LOCALHOST. You can use any web browser to browse all the modules installed on the current system. The homepage for this server is a list of modules, grouped by directory (and with attractive color blocks for browsers supporting that). Moreover, every module whose documentation you view becomes generously littered with links to any modules, functions and methods it imports. *HTML generator mode* -- The '-w' option can generate an HTML documentation page for anything [pydoc] can document. Pretty much, these pages are the same thing you might browse in webserver mode, but the pages are static and available for archiving, transmission, etc. *TK browser mode* -- The '-g' option will create a "graphical help browser," much along the lines of 'xman' or [tkman]. [distutils] ------------------------------------------------------------------------ As of Python 1.6, a package called [distutils] has become part of the standard Python library. There are two aspects to what [distutils] does. On the one side, [distutils] hopes to make installation of new modules, packages, and tools uniform and easy for end-users. On the other side, [distutils] also hopes to make the creation of these easy-to-install distributions easy on their developers. Let's look at both aspects briefly. In the very simplest case, a developer will have chosen to create an installer for your specific platform. If this is the case, you really don't need to know that anything called [distutils] exists at all. Currently, [distutils] is capable of creating RPMs for those Linux systems that support that format, and Windows EXE self-installers for Win32 systems. While these are big players, other platforms exist also and/or the developer might have had access to your platform (or the time or interest in making an installer). Fortunately, short of the simplest case, the next best case is hardly any more difficult. Assuming you get a [distutils] aware source distribution, you can count on a number of things (if nothing goes wrong, of course). The distribution archive should be in a standard archive format--usually either '.zip' or '.tgz' / '.tar.gz' (in odd cases you might find '.tbz' or 'tar.Z', and hopefully '.sit' support will be added for MacOS soon). Most of the time, Windows users use zip's and Linux/Unix users use tarballs. But it is not hard to unpack most formats on most platforms. Once you have unpacked the archive, you'll get a collection of files in a directory named in the same fashion as the archive was. For example: #--------- Unpacking a [distutils] archive --------------# E:\archive\devel>unzip -q Distutils-1_0_2.zip E:\archive\devel>cd Distutils-1.0.2 E:\archive\devel\Distutils-1.0.2>ls The volume label in drive E is ARCHIVE. The Volume Serial Number is E825:C814. Directory of E:\archive\devel\Distutils-1.0.2 6-14-01 0:38a