[from 2000-02-25] Second attempt at describing an installation scheme; this is

the simplified scheme that Guido proposed.  Also already-obsolete,
and saved only for posterity.
This commit is contained in:
Greg Ward 2000-03-10 01:57:51 +00:00
parent 7c1e5f65e3
commit 169f91b808

View file

@ -27,8 +27,9 @@
% Should these be added to the standard Python doc tools? (They'll be
% needed for my "Distributing Python Modules" guide, too.)
\newcommand{\command}[1]{\code{#1}}
\newcommand{\option}[1]{\code{#1}}
\newcommand{\option}[1]{\textsf{\small{#1}}}
\newcommand{\filevar}[1]{{\textsl{\filenq{#1}}}}
\newcommand{\homefile}[1]{\file{\tilde/#1}}
\newcommand{\comingsoon}{\emph{Coming soon$\ \ldots$}}
% And how about these? Very handy for writing pathnames (tilde for
@ -120,402 +121,177 @@ In any event, you can easily install to non-standard locations with a
couple of options to the \command{install} command:
\begin{tableii}{ll}{option}{Option}{Description}
\lineii{prefix}{base dir for pure Python distributions
(overrides \code{sys.prefix})}
\lineii{exec-prefix}{base dir for distributions with extensions
(overrides \code{sys.exec_prefix})}
\lineii{install-lib}{install dir for top-level modules from pure
Python distributions}
\lineii{install-platlib}{install dir for top-level modules from
distributions with extensions}
\lineii{install-path}{extra path under \option{install-lib} or
\option{install-platlib} to install to}
\lineii {install-lib}
{install directory for modules from pure Python distributions}
\lineii {install-platlib}
{install directory for modules from distributions with extensions}
\lineii {prefix}
{override \code{sys.prefix}:
point to an alternate Python installation}
\lineii {exec-prefix}
{override \code{sys.exec_prefix}:
point to an alternate Python installation}
\lineii {install-path}
{extra sub-path to append to \option{install-lib} (for
non-package-ized distributions)}
\end{tableii}
\subsection{Prefix options}
\label{sec:prefix-options}
There are a lot of picky little rules that govern the interactions of
these five options. As usual, it's easier to explain things with
examples, so we'll save all the picky rules for later, after you've seen
a bunch of examples. However, we really have to establish some ground
rules before we can dive into the examples:
\begin{itemize}
\item in a normal \UNIX{} installation, \code{sys.prefix} and
\code{sys.exec\_prefix} are both \file{/usr/local}.
\item in a multi-platform \UNIX{} installation, \code{sys.prefix} and
\code{sys.exec\_prefix} are different, and are selected when you
configure and build Python itself. Our canonical example of a
multi-platform installation will have a \code{sys.prefix} of
\file{/usr/local} and a \code{sys.exec\_prefix} of
\file{/usr/local.\filevar{plat}} (for whatever value of \filevar{plat}
is appropriate).
\item the canonical place to install third-party modules is
either \file{\filevar{prefix}/lib/python1.\filevar{X}/site-packages}
or \file{\filevar{exec\_prefix}/lib/python1.\filevar{X}/site-packages}.
These will be referred to as ``the site-packages directories.''
\end{itemize}
Of these, the most commonly used will probably be \option{install-lib}
and \option{install-platlib}: you use them to point module installation
right at a particular directory. (You'll only need
\option{install-platlib} if you maintain a multi-platform installation,
which is often done on \UNIX{} networks with different architectures and
operating systems.) The two prefix options are intended for the
somewhat arcane purpose of installing modules into a different Python
installation than the Python binary used to perform the installation.
The last, \option{install-path}, is mainly used for module developers to
ensure that their module will go into a directory of their own, but it
may occasionally be useful to you as a module installer.
\subsubsection{Pure Python module distribution}
To demonstrate, consider a hypothetical module distribution that
contains one top-level module and a package with two modules:
\begin{tableii}{ll}{module}{Module}{Filename}
\lineii{mymod}{\filenq{mymod.py}}
\lineii{mypkg.mod1}{\filenq{mypkg/mod1.py}}
\lineii{mypkg.mod2}{\filenq{mypkg/mod2.py}}
\end{tableii}
where the filenames are relative to \file{build/lib} after building, or
to some directory in \code{sys.path} after installation.
The goal of installation is to copy these files into a directory in
\code{sys.path} without interfering with the standard Python library.
The canonical, preferred, and most obvious thing to do is to put them in
the ``site-packages'' directory, which is exactly what the
\command{install} comand does by default: under a normal \UNIX{} Python
installation,
\begin{verbatim}
python setup.py install
\end{verbatim}
installs \file{/usr/local/lib/python1.\filevar{X}/lib/site-packages/mymod.py},
with the \module{mypkg} package in a \file{mypkg} directory under
\file{site-packages}.
However, if you were interested in a standard installation, you wouldn't
be reading this section. The next-most-standard thing to do is to
specify a custom prefix to override \code{sys.prefix}. For example:
\begin{verbatim}
python setup.py install --prefix=/home/greg
\end{verbatim}
is a sensible way to install Python modules to your home directory: this
results in the installation of \file{/home/greg/lib/python/mymod.py},
with the \module{mypkg} modules in \file{/home/greg/lib/python/mypkg/}.
An important point here is that in both this example and the ``plain
vanilla'' example above, the actual installation directory is derived
from the \option{prefix} option. However, when \option{prefix} differs
from \code{sys.prefix}, the installation directory is derived
differently: the Python version and \file{site-packages} are omitted.
(The version number is part of the standard library directory name to
describe the version of the standard library, so it doesn't make sense
to include it in the name of a non-standard-library directory; likewise,
\file{site-packages} is meant to denote non-standard modules living in
the same area as the standard library, so it doesn't make sense to
include it when installing to a non-standard library. [XXX check with
Guido that this reasoning is valid and correct; Fred disagrees!])
\subsubsection{Module distribution with extensions}
Now let's consider a different hypothetical module distribution, which
consists of a single package, \module{foo}, containing one pure Python
module and one extension module:
\begin{tableii}{ll}{module}{Module}{Filename}
\lineii{foo.pure}{\filenq{foo/pure.py}}
\lineii{foo.ext}{\filenq{foo/ext.so} (or \file{foo/extmodule.so})}
\end{tableii}
In this case, the two modules will be in different locations in the
build tree: \file{build/lib/foo/pure.py} and
\file{build/platlib/foo/ext.so}. (The \file{.so} (``shared object'')
extension isn't universal, but it's the norm on \UNIX-like systems;
under Windows, the extension module will be in \file{foo/ext.pyd} or
\file{foo/extmodule.pyd}.)
Consider again a standard, plain-vanilla installation:
\begin{verbatim}
python setup.py install
\end{verbatim}
In this case, \emph{both} modules will be installed to the site-packages
directory under \code{sys.exec\_prefix}, e.g. to
\file{/usr/local.\filevar{plat}/lib/python1.\filevar{X}/site-packages}
on a \UNIX{} system where Python was configured with
\samp{--exec-prefix=/usr/local.plat}. (On Windows, again, there is no
site-packages directory and \code{sys.prefix} and
\code{sys.exec\_prefix} are the same---so both modules will just be
installed to \code{sys.prefix}.)
Of course, we've already established that you're not interested in
standard installations. If you just want to install these modules to
your home directory, and you don't maintain a multi-platform home
directory, no problem---just set the prefix as before:
\begin{verbatim}
python setup.py install --prefix=/home/greg
\end{verbatim}
and both modules will be installed to \file{/home/greg/lib/python}.
Now let's say your Python installation is in \file{/usr}---as is the
case in many Linux distributions---but your local policy is to install
third-party software to a network-wide \file{/usr/local} and
\file{/usr/local.\filevar{plat}}. That is, \code{sys.prefix} and
\code{sys.exec\_prefix} are both \file{/usr}, and you want Python
modules to be installed to either \file{/usr/local/lib/python} or
\file{/usr/local.\filevar{plat}/lib/python}. This is one case where you
want to specify both \option{prefix} and \option{exec-prefix}:
\begin{verbatim}
python setup.py install --prefix=/usr/local \
--exec-prefix=/usr/local.plat
\end{verbatim}
An oddity of this situation is that for any given module distribution,
you only have to supply \emph{one} of \option{prefix} and
\option{exec-prefix}, because pure Python distributions are always
installed under \option{prefix}, and extension-containing distributions
are always installed under \option{exec-prefix}. For consistency's
sake, though, it's best always to supply both---and the best way to do
that is by using a system-wide configuration file (see
Section~\ref{sec:config-files}).
You could use a similar scheme to maintain a multi-platform personal
Python library. For example, if you install lots of stuff to your home
directory (not just Python modules), you might have a complete
\file{\tilde/usr} with \file{include}, \file{man}, \file{lib}, and so
forth. (The advantage of this scheme is that it keeps those mock system
directories out of your home directory and makes it easier to support a
multi-platform personal \file{usr} tree.) If you don't care about a
multi-platform installation, you can just install with
\begin{verbatim}
python setup.py install --prefix=$HOME/usr
\end{verbatim}
But if you want to keep separate \file{usr} trees for each architecture
that you use, you could say
\begin{verbatim}
python setup.py install --prefix=$HOME/usr \
--exec-prefix=$HOME/usr.plat
\end{verbatim}
for various values of \file{plat}.
% this paragraph is for Michel Sanner ;-)
(Perceptive readers will note that on a multi-platform Python
installation, multiple identical copies of \file{foo/pure.py} will be
installed, one for each platform. This is deliberate. First, it makes
Python's module search algorithm simpler (XXX check this): when you say
\samp{import foo.pure}, Python searches \code{sys.path} until it finds a
directory containing \file{foo/__init__.py}. When it finds one, that
directory is deemed to be the directory containing the \module{foo}
package for this import. Even if the search algorithm were changed
(necessitating a trip back in time to ``fix'' Python 1.5), the only way
to make multiple candidate \module{foo} directories (one for pure
Python, one for extension modules) would be to make copies of
\file{__init__.py}---in which case, why not make copies of all the pure
Python modules? Second, if you kept pure Python modules related to
extension modules in a platform-shared directory, what happens while you
are upgrading your favourite extension from version 1.0 to 1.1 on
platforms X and Y? After you install 1.1 for platform X, the 1.1
\file{.py} files will be in the platform-shared directory---but the 1.0
extensions will still be in the platform Y directory. If the interval
between installing 1.1 for platform X and for platform Y is long---e.g.,
there are portability problems with platform Y---then there's a good
probability of a version mismatch between the 1.1 Python modules and the
1.0 extensions on platform Y. The solution to both problems is to
install separate copies of the pure Python modules for every platform.
In this day and age, unnecessary disk use is no argument.)
Other ways to support a multi-platform personal Python library are
discussed below, when we cover the \option{install-lib} and
\option{install-platlib} options.
% Gory details on the prefix options (still need to work these into the
% surrounding text):
XXX need to finish these rules and give them some context!
\begin{itemize}
\item \code{sys.exec\_prefix} (and the \option{exec-prefix} option)
only matters on a multi-platform installation. If you don't have a
multi-platform installation (or even know what that is), then you
don't care about \option{exec-prefix}.
\item in a normal Windows installation, \code{sys.prefix} and
\code{sys.exec\_prefix} are both \file{C:\bslash Program Files\bslash
Python}; they are never different under Windows (XXX check!).
\item you may supply \emph{both} of \option{prefix} and
\option{exec-prefix}, or \emph{neither} of them, or \emph{just}
\option{prefix}---but you may not supply just \option{exec-prefix}.
\end{itemize}
\subsection{Installation directory options}
\subsection{Directly specifying installation directories}
\label{sec:install-dirs}
Most of the time, it's enough to specify just \option{prefix} (and
possibly \option{exec-prefix})---your modules are installed to
\file{lib/python} under one or the other, you add the appropriate
directory to \code{sys.path}, and that's it.
The most common type of custom module installation is where you maintain
a personal stash of Python modules under your home directory, say in
\homefile{lib/python}. If you only care about a single platform
there, then you only need to specify the \option{install-lib} option and
can forget about \option{install-platlib}:
\begin{verbatim}
python setup.py install --install-lib=~/lib/python
\end{verbatim}
You can, of course, supply whatever directory you like in place of
\homefile{lib/python}. More importantly, you can specify this
directory permanently in your personal configuration file (XXX
filename?):
\begin{verbatim}
[install]
install-lib=~/lib/python
\end{verbatim}
Note that use of shell-style tilde and environment variable expansion is
supported both on the command line and in configuration files. (See
section~\ref{sec:config-files} for more information on configuration
files.)
However, there will inevitably be times when you want finer control over
the installation directories, and that is when the \option{install-lib},
\option{install-platlib}, and \option{install-path} options are
essential. Normally, \option{install-lib} and \option{install-platlib}
are simply the directories where pure Python modules and extension
modules, respectively, are installed. That is, top-level modules
(modules not in a package) are installed straight to
\option{install-lib} (or \option{install-platlib} if there are any
extensions in the module distribution). (If \option{install-path} is
supplied, then things are a bit more complex; we'll deal with that
Of course, in order for this personal Python library scheme to work, you
have to ensure that \homefile{lib/python} is in \code{sys.path} when you
run Python. The easiest way to do this under \UNIX{} is to add it to
your \code{PYTHONPATH} environment variable when you login. For
example, if you use a Bourne shell derivative such as bash, zsh, or ksh,
add the following to your \homefile{.profile} (or \homefile{.bashrc}, or
\homefile{.zshenv}, depending on your shell and personal preferences):
\begin{verbatim}
export PYTHONPATH=$HOME/lib/python
\end{verbatim}
If you use a csh-derivative such as tcsh, add the following to your
\homefile{.cshrc}:
\begin{verbatim}
setenv PYTHONPATH $HOME/lib/python
\end{verbatim}
If you use multiple platforms (architectures and/or operating systems)
from the same home directory, then you probably want to maintain a
multi-platform personal Python library. One possible scheme is to put
platform-neutral (pure Python) distributions in \homefile{lib/python}
and platform-specific distributions (any that containe extension
modules) in \homefile{lib/python.\filevar{plat}}:
\begin{verbatim}
python setup.py install --install-lib=~/lib/python \
--install-lib-plat=~/lib/python.plat \
\end{verbatim}
On the command line, of course, you can just type in the current
platform in place of \filevar{plat}: \file{linux-x86},
\file{solaris-sparc}, \file{linux-alpha}, whatever. That's not an
option in a configuration file, though---the same file has to cover all
platforms for which you maintain a personal Python library. So the
Distutils provide a \code{PLAT} environment variable which will expand
to the current platform name:
\begin{verbatim}
[install]
install-lib=~/lib/python
install-platlib=~/lib/python.$PLAT
\end{verbatim}
(If \code{PLAT} is already defined in your environment, the Distutils
won't override it: that way you can maintain consistency with other
applications that look for a \code{PLAT} variable; this is especially
useful when you refer to \code{PLAT} in your login scripts, as explained
below.)
Normally, \option{install-lib} and \option{install-platlib} are derived
from \option{prefix} and/or \option{exec-prefix}. For example, if you
don't supply anything, then \option{prefix} defaults to
\code{sys.prefix}, and \option{install-lib} defaults to
\file{\filevar{prefix}/lib/python1.\filevar{X}/site-packages}. If you
supply \option{prefix} but not \option{install-lib}, then
\option{install-lib} defaults to \file{\filevar{prefix}/lib/python}
(unless you just happen to supply a prefix which equals
\code{sys.prefix}, which is treated the same as if you don't supply
\option{prefix} at all). (The rules for \option{exec-prefix} and
\option{install-platlib} are a bit more complex; the following examples
should clarify. Consult the Distutils source for the gory details.)
(XXX danger danger! this environment-variable-in-config-file thing is
frighteningly make-like: is there any way to avoid it?)
To illustrate, let's go back to our hypothetical pure-Python module
distribution containing \module{mymod}, \module{mypkg.mod1}, and
\module{mypkg.mod2}. If you maintain a personal stash of Python modules
in your home directory, but don't like the \file{\tilde/lib/python}
convention, no problem---you can put the modules right in a
\file{\tilde/python} directory with
Again, you have to make sure that your personal Python library appears
in \code{sys.path}, and again the easiest way to do this is to set
\code{PYTHONPATH} in your login scripts. This time, though, you have to
be sure to set \emph{both} directories (platform-neutral and the current
platform-specific directory). For Bourne-shell derivatives:
\begin{verbatim}
python setup.py install --install-lib=$HOME/python
export PYTHONPATH=$HOME/lib/python:$HOME/lib/python.$PLAT
\end{verbatim}
which will install \file{\$HOME/python/mymod.py},
\file{\$HOME/python/mypkg/mod1.py}, and
\file{\$HOME/python/mypkg/mod2.py}.
If you happen to install a module distribution that contains extensions,
again that's no problem---in the absence of \option{exec-prefix},
\option{install-platlib} defaults to \option{install-lib}, so the above
example will also put extension modules in \file{\$HOME/python}.
(XXX is this correct? is this the best way to describe it? should it be
implemented this way or some other way? how should it be described?)
This may not be what you want, though, if you maintain a multi-platform
stash of Python modules in your home directory. In that case, you need
to specify \option{install-platlib}---this is the directory where module
distributions with extensions will be installed. For example, if you
keep pure Python module distributions in \file{\tilde/python} and
extension distributions in \file{\tilde/python.plat}:
and for csh-derivatives:
\begin{verbatim}
python setup.py install --install-lib=$HOME/python \
--install-platlib=$HOME/python.plat
setenv PYTHONPATH $HOME/lib/python:$HOME/lib/python.$PLAT
\end{verbatim}
(Just as with \option{prefix} and \option{exec-prefix}, it's only
necessary to supply one of \option{install-lib} and
\option{install-platlib} for any given module distribution, but to
ensure consistency you should always supply them both using a
configuration file (section~\ref{sec:config-files}).)
An alternate way to maintain a multi-platform personal Python library is
in \file{\tilde/lib/python} and \file{\tilde/lib/python.plat}. In that
case, you can get away with supplying \option{prefix} and
\option{install-platlib}:
Note that it is your responsibility to set the \code{PATH} environment
variable (unless your system administrator has kindly taken care of it
in the system-wide login scripts, which is a wise thing to do on
multi-platform networks). One way to do this is with the \code{uname}
command:
\begin{verbatim}
python setup.py install --prefix=$HOME \
--install-platlib=$HOME/lib/python.plat
export PLAT=`uname -sm | tr 'A-Z ' 'a-z-'`
\end{verbatim}
(XXX check that this works well on other Unices: on Linux, \code{-m}
becomes eg. \code{i586}, which is not the \emph{machine} but the
\emph{processor}. Arggh!)
Of course, there are more reasons to do custom installation than
maintaining a personal Python library. Even if you have write access to
the system-wide directories for third-party modules
(\file{\filevar{prefix}/lib/python1.\filevar{x}/site-packages} and
\file{\filevar{exec-prefix}/lib/python1.\filevar{x}/site-packages}), you
might want to install new module distributions---especially upgrades of
modules that are crucial to your local infrastructure---to a temporary
location, in order to test them before installing them ``for real.''
This is fundamentally no different from installing to your home
directory, except that you probably won't bother to set
\code{PYTHONPATH} permanently. For example, to install a module
distribution to \file{/tmp/pylib}:
\begin{verbatim}
python setup.py install --install-lib=/tmp/pylib
\end{verbatim}
Then, of course, you'll want to run some script that depends on these
modules to make sure that they still work with your installed base of
code:
\begin{verbatim}
env PYTHONPATH=/tmp/pylib python /usr/local/bin/crucial_script ...
\end{verbatim}
Finally, the \option{install-path} option, which exists mainly to gum up
the whole works---but in a productive (and important) way.
Specifically, \option{install-path} exists to give a directory of their
own to module distributions that wouldn't otherwise have one, i.e.\ that
are not distributed as a (Python) package.
Consider a module distribution, Foo, that consists of (pure Python)
modules \module{foobar}, \module{foobaz}, and \module{fooqux}.
Obviously these are related, and if the project had started in the
Python 1.5 era (and doesn't worry about backwards compatibility with
Python 1.4), they probably would be packaged up and called
\module{foo.bar}, \module{foo.baz}, and \module{foo.qux}.
Unfortunately, they aren't, but we still want the Foo modules to go into
a directory of their own.
Normally, this will be taken care of by the module developer: he adds a
line \samp{install_path = 'Foo'} to his setup script, which has the
following consequences:
\begin{enumerate}
\item instead of \option{install-lib} the modules would be installed in
\file{\filevar{install-lib}/Foo}
\item if \option{install-lib} is the same as the default
\option{install-lib}---e.g., you supplied neither \option{prefix} or
\option{install-lib}---then a \file{Foo.pth} will be created in
\option{install-lib}, so that Python adds
\file{\filevar{install-lib}/Foo} to \code{sys.path}
\item if \option{install-lib} is not the default, then a warning will be
printed, reminding you to add \file{\filevar{install-lib}/Foo} to
\code{sys.path} yourself, such as with the \code{PYTHONPATH}
environment variable
\end{enumerate}
Thus, you as a module installer have to be aware of the
\option{install-path} option---especially if you maintain a personal
stash of Python modules and don't have write permission to the standard
library, so Distutils can't create \file{.pth} files for you---but you
don't often have to supply it yourself. There are situations in which
you might want to supply it, though:
\begin{itemize}
\item a module developer forgot to include it (the distribution really
should go in a directory of its own, but it won't unless you make it)
\item you want to override the \option{install-path} supplied by the
developer (e.g., you'd rather have a huge jumble of files in
\file{site-packages} than make Python wade through a bunch of
\file{.pth} files at startup)
\end{itemize}
The first case is easy: say we're dealing with the Foo distribution
again, but the developer forgot to include \option{install-path}. No
problem, you can supply it on the command line:
Of course, you can do this temporary installation with separate
\option{install-lib} and \option{install-platlib} options. If you're
doing this to a network-wide directory, not \file{/tmp}, this might be
essential. As you might have guessed, it's not too hard:
\begin{verbatim}
python setup.py install --install-path=Foo
python setup.py install --install-lib=/scratch/pylib \
--install-platlib=/scratch/pylib.plat
\end{verbatim}
Note that this will work just fine if you supply \option{prefix} or
\option{install-lib}---but of course, you'll probably have to ensure
that the \file{Foo} directory is in \code{sys.path} yourself.
If you're really fanatical about keeping track of what you have
installed, you might want to supply your own \option{install-path} that
records the version as well as the name of the module distribution; this
overrides any \option{install-path} included by the module developer in
the setup script:
and then, testing your crucial scripts on multiple platforms:
\begin{verbatim}
python setup.py install --install-path=Foo-1.3
env PYTHONPATH=/scratch/pylib:/scratch/pylib.plat \
python /usr/local/bin/crucial_script ...
\end{verbatim}
Finally, you can disable \option{install-path} entirely:
However you do the testing, once you're satisfied that the new version
doesn't break anything, you can install it to the system-wide
third-party module directory as usual:
\begin{verbatim}
python setup.py install --install-path=''
python setup.py install
\end{verbatim}
...but the mess that will result (modules from many different
distributions in the same \option{install-lib} and
\option{install-platlib} directories) is your own problem.
% Points to make
% * only one of prefix or exec_prefix matters
% * don't have to specify exec_prefix unless != prefix
% * thus, usually enough to supply prefix
% * only have to supply install_lib if you don't like
% "prefix/lib/python"
% * likewise for install_platlib and exec_prefix
% * don't have to supply install_platlib unless != install_lib (??)
% * in the absence of install_path, top-level modules wind up in
% install_lib or install_platlib
In case you're interested, here are the exact rules for how
\option{install-lib} and \option{install-platlib} are initialized, and
how they and \option{install-path} affect where modules (pure Python and
extensions) are installed to:
\begin{itemize}
\item If you don't supply \option{prefix} (and possibly
\option{exec-prefix}), then \option{install-lib} and
\option{install-platlib} will be, respectively,
\file{\filevar{\$prefix}/lib/python1.\filevar{X}/site-packages} and
\file{\filevar{\$exec\_prefix}/lib/python1.\filevar{X}/site-packages}. In a
normal \UNIX{} installation, both of these resolve to
\file{/usr/local/lib/python1.\filevar{X}/site-packages}.
\item in the absence of an \option{install-path} option, top-level
modules and packages from a pure Python distribution are installed to
\option{install-lib}
\item in the absence of an \option{install-path} option, top-level
modules and packages from a distribution that contains \emph{any}
extension modules are installed to \option{install-platlib}.
\item \emph{there're more, but I don't remember everything offhand}
%\item \option{install-lib} is initialized from \option{prefix} (which
% in turn is initialized from \code{sys.prefix})---so you should
\end{itemize}
\subsection{Indirect specification: prefix directories}
\label{sec:prefix-dirs}
Occasionally, you may want to install a module distribution
\section{Custom Installation (Windows)}