cpython/Tools/msi

Quick Build Info
================

For testing, the installer should be built with the Tools/msi/build.bat
script:

    build.bat [-x86] [-x64] [--doc]

For an official release, the installer should be built with the
Tools/msi/buildrelease.bat script and environment variables:

    set PYTHON=<path to Python 2.7 or 3.4>
    set SPHINXBUILD=<path to sphinx-build.exe>
    set PATH=<path to Mercurial (hg.exe)>;
             <path to HTML Help Compiler (hhc.exe)>;%PATH%

    buildrelease.bat [-x86] [-x64] [-D] [-B]
        [-o <output directory>] [-c <certificate name>]

See the Building the Installer section for more information.

Overview
========

Python is distributed on Windows as an installer that will configure the
user's system. This allows users to have a functioning copy of Python
without having to build it themselves.

The main tasks of the installer are:

* copy required files into the expected layout
* configure system settings so the installation can be located by
  other programs
* add entry points for modifying, repairing and uninstalling Python
* make it easy to launch Python, its documentation, and IDLE

Each of these is discussed in a later section of this document.

Structure of the Installer
==========================

The installer is structured as a 'layout', which consists of a number of
CAB and MSI files and a single EXE.

The EXE is the main entry point into the installer. It contains the UI
and command-line logic, as well as the ability to locate and optionally
download other parts of the layout.

Each MSI contains the logic required to install a component or feature
of Python. These MSIs should not be launched directly by users. MSIs can
be embedded into the EXE or automatically downloaded as needed.

Each CAB contains the files making up a Python installation. CABs are
embedded into their associated MSI and are never seen by users.

MSIs are only required when the related feature or component is being
installed. When components are not selected for installation, the
associated MSI is not downloaded. This allows the installer to offer
options to install debugging symbols and binaries without increasing
the initial download size by separating them into their own MSIs.

Building the Installer
======================

Before building the installer, download extra build dependencies using
Tools\msi\get_externals.bat. (Note that this is in addition to the
similarly named file in PCbuild.)

One of the dependencies used in builds is WiX, a toolset that lets developers
create installers for Windows Installer, the Windows installation engine. WiX
has a dependency on the Microsoft .NET Framework Version 3.5 (which may not be
configured on recent versions of Windows, such as Windows 10). If you are
building on a recent Windows version, use the Control Panel (Programs | Programs
and Features | Turn Windows Features on or off) and ensure that the entry
".NET Framework 3.5 (includes .NET 2.0 and 3.0)" is enabled.

For testing, the installer should be built with the Tools/msi/build.bat
script:

    build.bat [-x86] [-x64] [--doc] [--test-marker] [--pack]

This script will build the required configurations of Python and
generate an installer layout in PCbuild/(win32|amd64)/en-us.

Specify -x86 and/or -x64 to build for each platform. If neither is
specified, both platforms will be built. Currently, both the debug and
release versions of Python are required for the installer.

Specify --doc to build the documentation (.chm) file. If the file is not
available, it will simply be excluded from the installer. Ensure
%PYTHON% and %SPHINXBUILD% are set when passing this option. You may
also set %HTMLHELP% to the Html Help Compiler (hhc.exe), or put HHC on
your PATH or in externals/.

Specify --test-marker to build an installer that works side-by-side with
an official Python release. All registry keys and install locations will
include an extra marker to avoid overwriting files. This marker is
currently an 'x' prefix, but may change at any time.

Specify --pack to build an installer that does not require all MSIs to
be available alongside. This takes longer, but is easier to share.


For an official release, the installer should be built with the
Tools/msi/buildrelease.bat script:

    set PYTHON=<path to Python 2.7 or 3.4>
    set SPHINXBUILD=<path to sphinx-build.exe>
    set PATH=<path to Mercurial (hg.exe)>;
             <path to HTML Help Compiler (hhc.exe)>;%PATH%

    buildrelease.bat [-x86] [-x64] [-D] [-B]
        [-o <output directory>] [-c <certificate name>]

Specify -x86 and/or -x64 to build for each platform. If neither is
specified, both platforms will be built. Currently, both the debug and
release versions of Python are required for the installer.

Specify -D to skip rebuilding the documentation. The documentation is
required for a release and the build will fail if it is not available.

Specify -B to skip rebuilding Python. This is useful to only rebuild the
installer layout after a previous call to buildrelease.bat.

Specify -o to set an output directory. The installer layouts will be
copied to platform-specific subdirectories of this path.

Specify -c to choose a code-signing certificate to be used for all the
signable binaries in Python as well as each file making up the
installer. Official releases of Python must be signed.

Ensure %PYTHON% and %SPHINXBUILD% are set when passing this option. You
may also set %HTMLHELP% to the Html Help Compiler (hhc.exe), or put HHC
on your PATH or in externals/. You will also need Mercurial (hg.exe) on
your PATH.

If WiX is not found on your system, it will be automatically downloaded
and extracted to the externals/ directory.

To manually build layouts of the installer, build one of the projects in
the bundle folder.

    msbuild bundle\snapshot.wixproj
    msbuild bundle\releaseweb.wixproj
    msbuild bundle\releaselocal.wixproj
    msbuild bundle\full.wixproj

snapshot.wixproj produces a test installer versioned based on the date.

releaseweb.wixproj produces a release installer that does not embed any
of the layout.

releaselocal.wixproj produces a release installer that embeds the files
required for a default installation.

full.wixproj produces a test installer that embeds the entire layout.

The following properties may be passed when building these projects.

  /p:BuildForRelease=(true|false)
    When true, adds extra verification to ensure a complete installer is
    produced. For example, binutils is required when building for a release
    to generate MinGW-compatible libraries, and the build will be aborted if
    this fails. Defaults to false.

  /p:ReleaseUri=(any URI)
    Used to generate unique IDs for the installers to allow side-by-side
    installation. Forks of Python can use the same installer infrastructure
    by providing a unique URI for this property. It does not need to be an
    active internet address. Defaults to $(ComputerName).

    Official releases use http://www.python.org/(architecture name)

  /p:DownloadUrlBase=(any URI)
    Specifies the base of a URL where missing parts of the installer layout
    can be downloaded from. The build version and architecture will be
    appended to create the full address. If omitted, missing components will
    not be automatically downloaded.

  /p:DownloadUrl=(any URI)
    Specifies the full URL where missing parts of the installer layout can
    be downloaded from. Should normally include '{2}', which will be
    substituted for the filename. If omitted, missing components will not be
    automatically downloaded. If specified, this value overrides
    DownloadUrlBase.

  /p:SigningCertificate=(certificate name)
    Specifies the certificate to sign the installer layout with. If omitted,
    the layout will not be signed.

  /p:RebuildAll=(true|false)
    When true, rebuilds all of the MSIs making up the layout. Defaults to
    true.

Uploading the Installer
=======================

For official releases, the uploadrelease.bat script should be used.

You will require PuTTY so that plink.exe and pscp.exe can be used, and your
SSH key can be activated in pageant.exe. PuTTY should be either on your path
or in %ProgramFiles(x86)%\PuTTY.

To include signatures for each uploaded file, you will need gpg2.exe on your
path or have run get_externals.bat. You may also need to "gpg2.exe --import"
your key before running the upload script.

    uploadrelease.bat --host <host> --user <username> [--dry-run] [--no-gpg]

The host is the URL to the server. This can be provided by the Release
Manager. You should be able to SSH to this address.

The username is your own username, which you have permission to SSH into
the server containing downloads.

Use --dry-run to display the generated upload commands without executing
them. Signatures for each file will be generated but not uploaded unless
--no-gpg is also passed.

Use --no-gpg to suppress signature generation and upload.

The default target directory (which appears in uploadrelease.proj) is
correct for official Python releases, but may be overridden with
--target <path> for other purposes. This path should generally not include
any version specifier, as that will be added automatically.

Modifying the Installer
=======================

The code for the installer is divided into three main groups: packages,
the bundle and the bootstrap application.

Packages
--------

Packages appear as subdirectories of Tools/msi (other than the bundle/
directory). The project file is a .wixproj and the build output is a
single MSI. Packages are built with the WiX Toolset. Some project files
share source files and use preprocessor directives to enable particular
features. These are typically used to keep the sources close when the
files are related, but produce multiple independent packages.

A package is the smallest element that may be independently installed or
uninstalled (as used in this installer). For example, the test suite has
its own package, as users can choose to add or remove it after the
initial installation.

All the files installed by a single package should be related, though
some packages may not install any files. For example, the pip package
executes the ensurepip package, but does not add or remove any of its
own files. (It is represented as a package because of its
installed/uninstalled nature, as opposed to the "precompile standard
library" option, for example.) Dependencies between packages are handled
by the bundle, but packages should detect when dependencies are missing
and raise an error.

Packages that include a lot of files may use an InstallFiles element in
the .wixproj file to generate sources. See lib/lib.wixproj for an
example, and msi.targets and csv_to_wxs.py for the implementation. This
element is also responsible for generating the code for cleaning up and
removing __pycache__ folders in any directory containing .py files.

All packages are built with the Tools/msi/common.wxs file, and so any
directory or property in this file may be referenced. Of particular
interest:

  REGISTRYKEY (property)
    The registry key for the current installation.

  InstallDirectory (directory)
    The root install directory for the current installation. Subdirectories
    are also specified in this file (DLLs, Lib, etc.)

  MenuDir (directory)
    The Start Menu folder for the current installation.

  UpgradeTable (property)
    Every package should reference this property to include upgrade
    information.

  OptionalFeature (Component)
    Packages that may be enabled or disabled should reference this component
    and have an OPTIONAL_FEATURES entry in the bootstrap application to
    properly handle Modify and Upgrade.

The .wxl_template file is specially handled by the build system for this
project to perform {{substitutions}} as defined in msi.targets. They
should be included in projects as <WxlTemplate> items, where .wxl files
are normally included as <EmbeddedResource> items.

Bundle
------

The bundle is compiled to the main EXE entry point that for most users
will represent the Python installer. It is built from Tools/msi/bundle
with packages references in Tools/msi/bundle/packagegroups.

Build logic for the bundle is in bundle.targets, but should be invoked
through one of the .wixproj files as described in Building the
Installer.

The UI is separated between Default.thm (UI layout), Default.wxl
(strings), bundle.wxs (properties) and the bootstrap application.
Bundle.wxs also contains the chain, which is the list of packages to
install and the order they should be installed in. These refer to named
package groups in bundle/packagegroups.

Each package group specifies one or more packages to install. Most
packages require two separate entries to support both per-user and
all-users installations. Because these reuse the same package, it does
not increase the overall size of the package.

Package groups refer to payload groups, which allow better control over
embedding and downloading files than the default settings. Whether files
are embedded and where they are downloaded from depends on settings
created by the project files.

Package references can include install conditions that determine when to
install the package. When a package is a dependency for others, the
condition should be crafted to ensure it is installed.

MSI packages are installed or uninstalled based on their current state
and the install condition. This makes them most suitable for features
that are clearly present or absent from the user's machine.

EXE packages are executed based on a customisable condition that can be
omitted. This makes them suitable for pre- or post-install tasks that
need to run regardless of whether features have been added or removed.

Bootstrap Application
---------------------

The bootstrap application is a C++ application that controls the UI and
installation. While it does not directly compile into the main EXE of
the installer, it forms the main active component. Most of the
installation functionality is provided by WiX, and so the bootstrap
application is predominantly responsible for the code behind the UI that
is defined in the Default.thm file. The bootstrap application code is in
bundle/bootstrap and is built automatically when building the bundle.

Installation Layout
===================

There are two installation layouts for Python on Windows, with the only
differences being supporting files. A layout is selected implicitly
based on whether the install is for all users of the machine or just for
the user performing the installation.

The default installation location when installing for all users is
"%ProgramFiles%\Python3X" for the 64-bit interpreter and
"%ProgramFiles(x86)%\Python3X-32" for the 32-bit interpreter. (Note that
the latter path is equivalent to "%ProgramFiles%\Python3X-32" when
running a 32-bit version of Windows.) This location requires
administrative privileges to install or later modify the installation.

The default installation location when installing for the current user
is "%LocalAppData%\Programs\Python\Python3X" for the 64-bit interpreter
and "%LocalAppData%\Programs\Python\Python3X-32" for the 32-bit
interpreter. Only the current user can access this location. This
provides a suitable level of protection against malicious modification
of Python's files.

(Default installation locations are set in Tools\msi\bundle\bundle.wxs.)

Within this install directory is the following approximate layout:

.\python[w].exe The core executable files
.\DLLs          Stdlib extensions (*.pyd) and dependencies
.\Doc           Documentation (*.chm)
.\include       Development headers (*.h)
.\Lib           Standard library
.\Lib\test      Test suite
.\libs          Development libraries (*.lib)
.\Scripts       Launcher scripts (*.exe, *.py)
.\tcl           Tcl dependencies (*.dll, *.tcl and others)
.\Tools         Tool scripts (*.py)

When installed for all users, the following files are installed to
either "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64" as
appropriate. For the current user, they are installed in the Python
install directory.

.\python3x.dll      The core interpreter
.\python3.dll       The stable ABI reference

When installed for all users, the following files are installed to
"%SystemRoot%" (typically "C:\Windows") to ensure they are always
available on PATH. (See Launching Python below.) For the current user,
they are installed in "%LocalAppData%\Programs\Python\PyLauncher".

.\py[w].exe         PEP 397 launcher

System Settings
===============

On installation, registry keys are created so that other applications
can locate and identify installations of Python. The locations of these
keys vary based on the install type.

For 64-bit interpreters installed for all users, the root key is:
    HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.X

For 32-bit interpreters installed for all users on a 64-bit operating
system, the root key is:
    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\3.X-32

For 32-bit interpreters installed for all users on a 32-bit operating
system, the root key is:
    HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.X-32

For 64-bit interpreters installed for the current user:
    HKEY_CURRENT_USER\Software\Python\PythonCore\3.X

For 32-bit interpreters installed for the current user:
    HKEY_CURRENT_USER\Software\Python\PythonCore\3.X-32

When the core Python executables are installed, a key "InstallPath" is
created within the root key with its default value set to the
executable's install directory. A value named "ExecutablePath" is added
with the full path to the main Python interpreter, and a key
"InstallGroup" is created with its default value set to the product
name "Python 3.X".

When the Python standard library is installed, a key "PythonPath" is
created within the root key with its default value set to the full path
to the Lib folder followed by the path to the DLLs folder, separated by
a semicolon.

When the documentation is installed, a key "Help" is created within the
root key, with a subkey "Main Python Documentation" with its default
value set to the full path to the installed CHM file.


The py.exe launcher is installed as part of a regular Python install,
but using a separate mechanism that allows it to more easily span
versions of Python. As a result, it has different root keys for its
registry entries:

When installed for all users on a 64-bit operating system, the
launcher's root key is:
    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\Launcher

When installed for all users on a 32-bit operating system, the
launcher's root key is:
    HKEY_LOCAL_MACHINE\Software\Python\Launcher

When installed for the current user:
    HKEY_CURRENT_USER\Software\Python\Launcher

When the launcher is installed, a key "InstallPath" is created within
its root key with its default value set to the launcher's install
directory. File associations are also created for .py, .pyw, .pyc and
.pyo files.

Launching Python
================

When a feature offering user entry points in the Start Menu is
installed, a folder "Python 3.X" is created. Every shortcut should be
created within this folder, and each shortcut should include the version
and platform to allow users to identify the shortcut in a search results
page.

The core Python executables creates a shortcut "Python 3.X (32-bit)" or
"Python 3.X (64-bit)" depending on the interpreter.

The documentation creates a shortcut "Python 3.X 32-bit Manuals" or
"Python 3.X 64-bit Manuals". The documentation is identical for all
platforms, but the shortcuts need to be separate to avoid uninstallation
conflicts.

Installing IDLE creates a shortcut "IDLE (Python 3.X 32-bit)" or "IDLE
(Python 3.X 64-bit)" depending on the interpreter.


For users who often launch Python from a Command Prompt, an option is
provided to add the directory containing python.exe to the user or
system PATH variable. If the option is selected, the install directory
and the Scripts directory will be added at the start of the system PATH
for an all users install and the user PATH for a per-user install.

When the user only has one version of Python installed, this will behave
as expected. However, because Windows searches the system PATH before
the user PATH, users cannot override a system-wide installation of
Python on their PATH. Further, because the installer can only prepend to
the path, later installations of Python will take precedence over
earlier installations, regardless of interpreter version.

Because it is not possible to automatically create a sensible PATH
configuration, users are recommended to use the py.exe launcher and
manually modify their PATH variable to add Scripts directories in their
preferred order. System-wide installations of Python should consider not
modifying PATH, or using an alternative technology to modify their
users' PATH variables.


The py.exe launcher is recommended because it uses a consistent and
sensible search order for Python installations. User installations are
preferred over system-wide installs, and later versions are preferred
regardless of installation order (with the exception that py.exe
currently prefers 2.x versions over 3.x versions without the -3 command
line argument).

For both 32-bit and 64-bit interpreters, the 32-bit version of the
launcher is installed. This ensures that the search order is always
consistent (as the 64-bit launcher is subtly different from the 32-bit
launcher) and also avoids the need to install it multiple times. Future
versions of Python will upgrade the launcher in-place, using Windows
Installer's upgrade functionality to avoid conflicts with earlier
installed versions.

When installed, file associations are created for .py, .pyc and .pyo
files to launch with py.exe and .pyw files to launch with pyw.exe. This
makes Python files respect shebang lines by default and also avoids
conflicts between multiple Python installations.


Repair, Modify and Uninstall
============================

After installation, Python may be modified, repaired or uninstalled by
running the original EXE again or via the Programs and Features applet
(formerly known as Add or Remove Programs).

Modifications allow features to be added or removed. The install
directory and kind (all users/single user) cannot be modified. Because
Windows Installer caches installation packages, removing features will
not require internet access unless the package cache has been corrupted
or deleted. Adding features that were not previously installed and are
not embedded or otherwise available will require internet access.

Repairing will rerun the installation for all currently installed
features, restoring files and registry keys that have been modified or
removed. This operation generally will not redownload any files unless
the cached packages have been corrupted or deleted.

Removing Python will clean up all the files and registry keys that were
created by the installer, as well as __pycache__ folders that are
explicitly handled by the installer. Python packages installed later
using a tool like pip will not be removed. Some components may be
installed by other installers and these will not be removed if another
product has a dependency on them.