============================== pkgutil -- Package Utilities ============================== .. module:: pkgutil :synopsis: Package utilities :Purpose: Add to the module search path for a specific package and work with resources included in a package. :Available In: 2.3 and later The :mod:`pkgutil` module includes functions for working with Python packages. :func:`extend_path` changes the import path for sub-modules of the package, and :func:`get_data` provides access to file resources distributed with the package. Package Import Paths ==================== The :func:`extend_path` function is used to modify the search path for modules in a given package to include other directories in :ref:`sys.path `. This can be used to override installed versions of packages with development versions, or to combine platform-specific and shared modules into a single package namespace. The most common way to call :func:`extend_path` is by adding these two lines to the ``__init__.py`` inside the packag: e:: import pkgutil __path__ = pkgutil.extend_path(__path__, __name__) :func:`extend_path` scans ``sys.path`` for directories that include a subdirectory named for the package given as the second argument. The list of directories is combined with the path value passed as the first argument and returned as a single list, suitable for use as the package import path. An example package called :mod:`demopkg` includes these files: :: $ find demopkg1 -name '*.py' demopkg1/__init__.py demopkg1/shared.py ``demopkg1/__init__.py`` contains: .. include:: demopkg1/__init__.py :literal: :start-after: #end_pymotw_header The :command:`print` statements shows the search path before and after it is modified, to highlight the difference. And an ``extension`` directory, with add-on features for :mod:`demopkg`, contains :: $ find extension -name '*.py' extension/__init__.py extension/demopkg1/__init__.py extension/demopkg1/not_shared.py A simple test program imports the :mod:`demopkg1` package: .. include:: pkgutil_extend_path.py :literal: :start-after: #end_pymotw_header When this test program is run directly from the command line, the :mod:`not_shared` module is not found. .. note:: The full filesystem paths in these examples have been shortened to emphasize the parts that change. :: $ python pkgutil_extend_path.py demopkg1.__path__ before: ['.../PyMOTW/pkgutil/demopkg1'] demopkg1.__path__ after: ['.../PyMOTW/pkgutil/demopkg1'] demopkg1 : .../PyMOTW/pkgutil/demopkg1/__init__.py demopkg1.shared : .../PyMOTW/pkgutil/demopkg1/shared.py demopkg1.not_shared: Not found (No module named not_shared) However, if the ``extension`` directory is added to the :data:`PYTHONPATH` and the program is run again, different results are produced. :: $ export PYTHONPATH=extension $ python pkgutil_extend_path.py demopkg1.__path__ before: ['.../PyMOTW/pkgutil/demopkg1'] demopkg1.__path__ after: ['.../PyMOTW/pkgutil/demopkg1', '.../PyMOTW/pkgutil/extension/demopkg1'] demopkg1 : .../PyMOTW/pkgutil/demopkg1/__init__.pyc demopkg1.shared : .../PyMOTW/pkgutil/demopkg1/shared.pyc demopkg1.not_shared: .../PyMOTW/pkgutil/extension/demopkg1/not_shared.py The version of :mod:`demopkg1` inside the ``extension`` directory has been added to the search path, so the :mod:`not_shared` module is found there. Extending the path in this manner is useful for combining platform-specific versions of packages with common packages, especially if the platform-specific versions include C extension modules. Development Versions of Packages -------------------------------- While develop enhancements to a project, it is common to need to test changes to an installed package. Replacing the installed copy with a development version may be a bad idea, since it is not necessarily correct and other tools on the system are likely to depend on the installed package. A completely separate copy of the package could be configured in a development environment using `virtualenv`_, but for small modifications the overhead of setting up a virtual environment with all of the dependencies may be excessive. Another option is to use :mod:`pkgutil` to modify the module search path for modules that belong to the package under development. In this case, however, the path must be reversed so development version overrides the installed version. Given a package :mod:`demopkg2` like this: :: $ find demopkg2 -name '*.py' demopkg2/__init__.py demopkg2/overloaded.py With the function under development located in ``demopkg2/overloaded.py``. The installed version contains .. include:: demopkg2/overloaded.py :literal: :start-after: #end_pymotw_header and ``demopkg2/__init__.py`` contains .. include:: demopkg2/__init__.py :literal: :start-after: #end_pymotw_header :func:`reverse` is used to ensure that any directories added to the search path by :mod:`pkgutil` are scanned for imports *before* the default location. This program imports :mod:`demopkg2.overloaded` and calls :func:`func`: .. include:: pkgutil_devel.py :literal: :start-after: #end_pymotw_header Running it without any special path treatment produces output from the installed version of :func:`func`. :: $ python pkgutil_devel.py demopkg2 : .../PyMOTW/pkgutil/demopkg2/__init__.py demopkg2.overloaded: .../PyMOTW/pkgutil/demopkg2/overloaded.py A development directory containing :: $ find develop -name '*.py' develop/demopkg2/__init__.py develop/demopkg2/overloaded.py and a modified version of :mod:`overloaded` .. include:: develop/demopkg2/overloaded.py :literal: :start-after: #end_pymotw_header will be loaded when the test program is run with the ``develop`` directory in the search path. :: $ export PYTHONPATH=develop $ python pkgutil_devel.py demopkg2 : .../PyMOTW/pkgutil/demopkg2/__init__.pyc demopkg2.overloaded: .../PyMOTW/pkgutil/develop/demopkg2/overloaded.pyc Managing Paths with PKG Files ----------------------------- The first example above illustrated how to extend the search path using extra directories included in the :data:`PYTHONPATH`. It is also possible to add to the search path using ``*.pkg`` files containing directory names. PKG files are similar to the PTH files used by the :mod:`site` module. They can contain directory names, one per line, to be added to the search path for the package. Another way to structure the platform-specific portions of the application from the first example is to use a separate directory for each operating system, and include a ``.pkg`` file to extend the search path. This example uses the same :mod:`demopkg1` files, and also includes the following files: :: $ find os_* -type f os_one/demopkg1/__init__.py os_one/demopkg1/not_shared.py os_one/demopkg1.pkg os_two/demopkg1/__init__.py os_two/demopkg1/not_shared.py os_two/demopkg1.pkg The PKG files are named ``demopkg1.pkg`` to match the package being extended. They both contain:: demopkg This demo program shows the version of the module being imported: .. include:: pkgutil_os_specific.py :literal: :start-after: #end_pymotw_header A simple run script can be used to switch between the two packages: .. include:: with_os.sh :literal: :start-after: #end_pymotw_header And when run with ``"one"`` or ``"two"`` as the arguments, the path is adjusted appropriately: :: $ ./with_os.sh one PYTHONPATH=os_one demopkg1.__path__ before: ['.../PyMOTW/pkgutil/demopkg1'] demopkg1.__path__ after: ['.../PyMOTW/pkgutil/demopkg1', '.../PyMOTW/pkgutil/os_one/demopkg1', 'demopkg'] demopkg1 : .../PyMOTW/pkgutil/demopkg1/__init__.pyc demopkg1.shared : .../PyMOTW/pkgutil/demopkg1/shared.pyc demopkg1.not_shared: .../PyMOTW/pkgutil/os_one/demopkg1/not_shared.pyc :: $ ./with_os.sh two PYTHONPATH=os_two demopkg1.__path__ before: ['.../PyMOTW/pkgutil/demopkg1'] demopkg1.__path__ after: ['.../PyMOTW/pkgutil/demopkg1', '.../PyMOTW/pkgutil/os_two/demopkg1', 'demopkg'] demopkg1 : .../PyMOTW/pkgutil/demopkg1/__init__.pyc demopkg1.shared : .../PyMOTW/pkgutil/demopkg1/shared.pyc demopkg1.not_shared: .../PyMOTW/pkgutil/os_two/demopkg1/not_shared.pyc PKG files can appear anywhere in the normal search path, so a single PKG file in the current working directory could also be used to include a development tree. Nested Packages --------------- For nested packages, it is only necessary to modify the path of the top-level package. For example, with this directory structure :: $ find nested -name '*.py' nested/__init__.py nested/second/__init__.py nested/second/deep.py nested/shallow.py Where ``nested/__init__.py`` contains .. include:: nested/__init__.py :literal: :start-after: #end_pymotw_header and a development tree like :: $ find develop/nested -name '*.py' develop/nested/__init__.py develop/nested/second/__init__.py develop/nested/second/deep.py develop/nested/shallow.py Both the :mod:`shallow` and :mod:`deep` modules contain a simple function to print out a message indicating whether or not they come from the installed or development version. This test program exercises the new packages. .. include:: pkgutil_nested.py :literal: :start-after: #end_pymotw_header When ``pkgutil_nested.py`` is run without any path manipulation, the installed version of both modules are used. :: $ python pkgutil_nested.py nested.shallow: .../PyMOTW/pkgutil/nested/shallow.pyc This func() comes from the installed version of nested.shallow nested.second.deep: .../PyMOTW/pkgutil/nested/second/deep.pyc This func() comes from the installed version of nested.second.deep When the ``develop`` directory is added to the path, the development version of both functions override the installed versions. :: $ PYTHONPATH=develop python pkgutil_nested.py nested.shallow: .../PyMOTW/pkgutil/develop/nested/shallow.pyc This func() comes from the development version of nested.shallow nested.second.deep: .../PyMOTW/pkgutil/develop/nested/second/deep.pyc This func() comes from the development version of nested.second.deep Package Data ============ In addition to code, Python packages can contain data files such as templates, default configuration files, images, and other supporting files used by the code in the package. The :func:`get_data` function gives access to the data in the files in a format-agnostic way, so it does not matter if the package is distributed as an EGG, part of a frozen binary, or regular files on the filesystem. With a package :mod:`pkgwithdata` containing a ``templates`` directory :: $ find pkgwithdata -type f pkgwithdata/__init__.py pkgwithdata/templates/base.html and ``pkgwithdata/templates/base.html`` containing .. literalinclude:: pkgwithdata/templates/base.html This program uses :func:`get_data` to retrieve the template contents and print them out. .. include:: pkgutil_get_data.py :literal: :start-after: #end_pymotw_header The arguments to :func:`get_data` are the dotted name of the package, and a filename relative to the top of the package. The return value is a byte sequence, so it is encoded as UTF-8 before being printed. .. {{{cog .. cog.out(run_script(cog.inFile, 'pkgutil_get_data.py')) .. }}} :: $ python pkgutil_get_data.py PyMOTW Template

Example Template

This is a sample data file.

.. {{{end}}} :func:`get_data` is distribution format-agnostic because it uses the import hooks defined in :pep:`302` to access the package contents. Any loader that provides the hooks can be used, including the ZIP archive importer in :mod:`zipfile`. .. include:: pkgutil_get_data_zip.py :literal: :start-after: #end_pymotw_header This example creates a ZIP archive with a copy of the :mod:`pkgwithdata` package, including a renamed version of the template file. It then adds the ZIP archive to the import path before using :mod:`pkgutil` to load the template and print it. .. {{{cog .. cog.out(run_script(cog.inFile, 'pkgutil_get_data_zip.py')) .. }}} :: $ python pkgutil_get_data_zip.py Loading pkgwithdata from pkgwithdatainzip.zip/pkgwithdata/__init__.pyc Template: PyMOTW Template

Example Template

This is a sample data file.

.. {{{end}}} .. seealso:: `pkgutil `_ Standard library documentation for this module. `virtualenv`_ Ian Bicking's virtual environment script. :mod:`distutils` Packaging tools from Python standard library. `Distribute`_ Next-generation packaging tools. :pep:`302` Import Hooks :mod:`zipfile` Create importable ZIP archives. :mod:`zipimport` Importer for packages in ZIP archives. .. _virtualenv: http://pypi.python.org/pypi/virtualenv .. _Distribute: http://packages.python.org/distribute/