Modules and Imports¶

Most Python programs end up as a combination of several modules with a main application importing them. Whether using the features of the standard library, or organizing custom code in separate files to make it easier to maintain, understanding and managing the dependencies for a program is an important aspect of development. sys includes information about the modules available to an application, either as built-ins or after being imported. It also defines hooks for overriding the standard import behavior for special cases.

Imported Modules¶

sys.modules is a dictionary mapping the names of imported modules to the module object holding the code.

import sys
import textwrap

names = sorted(sys.modules.keys())
name_text = ', '.join(names)

print textwrap.fill(name_text)

The contents of sys.modules change as new modules are imported.

$ python sys_modules.py

UserDict, __builtin__, __main__, _abcoll, _codecs, _sre, _warnings,
_weakref, _weakrefset, abc, codecs, copy_reg, encodings,
encodings.__builtin__, encodings.aliases, encodings.codecs,
encodings.encodings, encodings.utf_8, errno, exceptions, genericpath,
linecache, os, os.path, posix, posixpath, re, signal, site,
sphinxcontrib, sre_compile, sre_constants, sre_parse, stat, string,
strop, sys, textwrap, types, warnings, zipimport

Built-in Modules¶

The Python interpreter can be compiled with some C modules built right in, so they do not need to be distributed as separate shared libraries. These modules don’t appear in the list of imported modules managed in sys.modules because they were not technically imported. The only way to find the available built-in modules is through sys.builtin_module_names.

import sys

for name in sys.builtin_module_names:
    print name

The output of this script will vary, especially if run with a custom-built version of the interpreter. This output was created using a copy of the interpreter installed from the standard python.org installer for OS X.

$ python sys_builtins.py

__builtin__
__main__
_ast
_codecs
_sre
_symtable
_warnings
_weakref
errno
exceptions
gc
imp
marshal
posix
pwd
signal
sys
thread
xxsubtype
zipimport

Import Path¶

The search path for modules is managed as a Python list saved in sys.path. The default contents of the path include the directory of the script used to start the application and the current working directory.

import sys

for d in sys.path:
    print d

The first directory in the search path is the home for the sample script itself. That is followed by a series of platform-specific paths where compiled extension modules (written in C) might be installed, and then the global site-packages directory is listed last.

$ python sys_path_show.py
/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

The import search path list can be modified before starting the interpreter by setting the shell variable PYTHONPATH to a colon-separated list of directories.

$ PYTHONPATH=/my/private/site-packages:/my/shared/site-packages python sys_path_show.py
/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
/my/private/site-packages
/my/shared/site-packages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages

A program can also modify its path by adding elements to sys.path directly.

import sys
import os

base_dir = os.path.dirname(__file__) or '.'
print 'Base directory:', base_dir

# Insert the package_dir_a directory at the front of the path.
package_dir_a = os.path.join(base_dir, 'package_dir_a')
sys.path.insert(0, package_dir_a)

# Import the example module
import example
print 'Imported example from:', example.__file__
print '\t', example.DATA

# Make package_dir_b the first directory in the search path
package_dir_b = os.path.join(base_dir, 'package_dir_b')
sys.path.insert(0, package_dir_b)

# Reload the module to get the other version
reload(example)
print 'Reloaded example from:', example.__file__
print '\t', example.DATA

Reloading an imported module re-imports the file, and uses the same module object to hold the results. Changing the path between the initial import and the call to reload() means a different module may be loaded the second time.

$ python sys_path_modify.py

Base directory: .
Imported example from: ./package_dir_a/example.pyc
        This is example A
Reloaded example from: ./package_dir_b/example.pyc
        This is example B

Custom Importers¶

Modifying the search path lets a programmer control how standard Python modules are found, but what if a program needs to import code from somewhere other than the usual .py or .pyc files on the filesystem? PEP 302 solves this problem by introducing the idea of import hooks that can trap an attempt to find a module on the search path and take alternative measures to load the code from somewhere else or apply pre-processing to it.

Finders¶

Custom importers are implemented in two separate phases. The finder is responsible for locating a module and providing a loader to manage the actual import. Adding a custom module finder is as simple as appending a factory to the sys.path_hooks list. On import, each part of the path is given to a finder until one claims support (by not raising ImportError). That finder is then responsible for searching data storage represented by its path entry for named modules.

import sys

class NoisyImportFinder(object):
    
    PATH_TRIGGER = 'NoisyImportFinder_PATH_TRIGGER'
    
    def __init__(self, path_entry):
        print 'Checking NoisyImportFinder support for %s' % path_entry
        if path_entry != self.PATH_TRIGGER:
            print 'NoisyImportFinder does not work for %s' % path_entry
            raise ImportError()
        return
    
    def find_module(self, fullname, path=None):
        print 'NoisyImportFinder looking for "%s"' % fullname
        return None

sys.path_hooks.append(NoisyImportFinder)

sys.path.insert(0, NoisyImportFinder.PATH_TRIGGER)

try:
    import target_module
except Exception, e:
    print 'Import failed:', e

This example illustrates how the finders are instantiated and queried. The NoisyImportFinder raises ImportError when instantiated with a path entry that does not match its special trigger value, which is obviously not a real path on the filesystem. This test prevents the NoisyImportFinder from breaking imports of real modules.

$ python sys_path_hooks_noisy.py

Checking NoisyImportFinder support for NoisyImportFinder_PATH_TRIGGER
NoisyImportFinder looking for "target_module"
Checking NoisyImportFinder support for /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
NoisyImportFinder does not work for /Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys
Import failed: No module named target_module

Importing from a Shelve¶

When the finder locates a module, it is responsible for returning a loader capable of importing that module. This example illustrates a custom importer that saves its module contents in a database created by shelve.

The first step is to create a script to populate the shelf with a package containing a sub-module and sub-package.

import sys
import shelve
import os

filename = '/tmp/pymotw_import_example.shelve'
if os.path.exists(filename):
    os.unlink(filename)
db = shelve.open(filename)
try:
    db['data:README'] = """
==============
package README
==============

This is the README for ``package``.
"""
    db['package.__init__'] = """
print 'package imported'
message = 'This message is in package.__init__'
"""
    db['package.module1'] = """
print 'package.module1 imported'
message = 'This message is in package.module1'
"""
    db['package.subpackage.__init__'] = """
print 'package.subpackage imported'
message = 'This message is in package.subpackage.__init__'
"""
    db['package.subpackage.module2'] = """
print 'package.subpackage.module2 imported'
message = 'This message is in package.subpackage.module2'
"""
    db['package.with_error'] = """
print 'package.with_error being imported'
raise ValueError('raising exception to break import')
"""
    print 'Created %s with:' % filename
    for key in sorted(db.keys()):
        print '\t', key
finally:
    db.close()

A real packaging script would read the contents from the filesystem, but using hard-coded values is sufficient for a simple example like this.

$ python sys_shelve_importer_create.py

Created /tmp/pymotw_import_example.shelve with:
        data:README
        package.__init__
        package.module1
        package.subpackage.__init__
        package.subpackage.module2
        package.with_error

Next, it needs to provide finder and loader classes that know how to look in a shelf for the source of a module or package.

import contextlib
import imp
import os
import shelve
import sys


@contextlib.contextmanager
def shelve_context(filename, flag='r'):
    """Context manager to make shelves work with 'with' statement."""
    db = shelve.open(filename, flag)
    try:
        yield db
    finally:
        db.close()

    
def _mk_init_name(fullname):
    """Return the name of the __init__ module for a given package name."""
    if fullname.endswith('.__init__'):
        return fullname
    return fullname + '.__init__'


def _get_key_name(fullname, db):
    """Look in an open shelf for fullname or fullname.__init__, return the name found."""
    if fullname in db:
        return fullname
    init_name = _mk_init_name(fullname)
    if init_name in db:
        return init_name
    return None
    

class ShelveFinder(object):
    """Find modules collected in a shelve archive."""
    
    def __init__(self, path_entry):
        if not os.path.isfile(path_entry):
            raise ImportError
        try:
            # Test the path_entry to see if it is a valid shelf
            with shelve_context(path_entry):
                pass
        except Exception, e:
            raise ImportError(str(e))
        else:
            print 'new shelf added to import path:', path_entry
            self.path_entry = path_entry
        return
        
    def __str__(self):
        return '<%s for "%s">' % (self.__class__.__name__, self.path_entry)
        
    def find_module(self, fullname, path=None):
        path = path or self.path_entry
        print 'looking for "%s" in %s ...' % (fullname, path),
        with shelve_context(path) as db:
            key_name = _get_key_name(fullname, db)
            if key_name:
                print 'found it as %s' % key_name
                return ShelveLoader(path)
        print 'not found'
        return None


class ShelveLoader(object):
    """Load source for modules from shelve databases."""
    
    def __init__(self, path_entry):
        self.path_entry = path_entry
        return
        
    def _get_filename(self, fullname):
        # Make up a fake filename that starts with the path entry
        # so pkgutil.get_data() works correctly.
        return os.path.join(self.path_entry, fullname)
        
    def get_source(self, fullname):
        print 'loading source for "%s" from shelf' % fullname
        try:
            with shelve_context(self.path_entry) as db:
                key_name = _get_key_name(fullname, db)
                if key_name:
                    return db[key_name]
                raise ImportError('could not find source for %s' % fullname)
        except Exception, e:
            print 'could not load source:', e
            raise ImportError(str(e))
            
    def get_code(self, fullname):
        source = self.get_source(fullname)
        print 'compiling code for "%s"' % fullname
        return compile(source, self._get_filename(fullname), 'exec', dont_inherit=True)
    
    def get_data(self, path):
        print 'looking for data in %s for "%s"' % (self.path_entry, path)
        if not path.startswith(self.path_entry):
            raise IOError
        path = path[len(self.path_entry)+1:]
        key_name = 'data:' + path
        try:
            with shelve_context(self.path_entry) as db:
                return db[key_name]
        except Exception, e:
            # Convert all errors to IOError
            raise IOError
        
    def is_package(self, fullname):
        init_name = _mk_init_name(fullname)
        with shelve_context(self.path_entry) as db:
            return init_name in db

    def load_module(self, fullname):
        source = self.get_source(fullname)

        if fullname in sys.modules:
            print 'reusing existing module from previous import of "%s"' % fullname
            mod = sys.modules[fullname]
        else:
            print 'creating a new module object for "%s"' % fullname
            mod = sys.modules.setdefault(fullname, imp.new_module(fullname))

        # Set a few properties required by PEP 302
        mod.__file__ = self._get_filename(fullname)
        mod.__name__ = fullname
        mod.__path__ = self.path_entry
        mod.__loader__ = self
        mod.__package__ = '.'.join(fullname.split('.')[:-1])
        
        if self.is_package(fullname):
            print 'adding path for package'
            # Set __path__ for packages
            # so we can find the sub-modules.
            mod.__path__ = [ self.path_entry ]
        else:
            print 'imported as regular module'
        
        print 'execing source...'
        exec source in mod.__dict__
        print 'done'
        return mod
        

Now ShelveFinder and ShelveLoader can be used to import code from a shelf. For example, importing the package created above:

import sys
import sys_shelve_importer

def show_module_details(module):
    print '  message    :', module.message
    print '  __name__   :', module.__name__
    print '  __package__:', module.__package__
    print '  __file__   :', module.__file__
    print '  __path__   :', module.__path__
    print '  __loader__ :', module.__loader__

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

print 'Import of "package":'
import package

print
print 'Examine package details:'
show_module_details(package)

print
print 'Global settings:'
print 'sys.modules entry:', sys.modules['package']

The shelf is added to the import path the first time an import occurs after the path is modified. The finder recognizes the shelf and returns a loader, which is used for all imports from that shelf. The initial package-level import creates a new module object and then execs the source loaded from the shelf, using the new module as the namespace so that names defined in the source are preserved as module-level attributes.

$ python sys_shelve_importer_package.py

Import of "package":
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found i
t as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done

Examine package details:
  message    : This message is in package.__init__
  __name__   : package
  __package__:
  __file__   : /tmp/pymotw_import_example.shelve/package
  __path__   : ['/tmp/pymotw_import_example.shelve']
  __loader__ : <sys_shelve_importer.ShelveLoader object at 0x100473950
>

Global settings:
sys.modules entry: <module 'package' from '/tmp/pymotw_import_example.
shelve/package'>

Packages¶

The loading of other modules and sub-packages proceeds in the same way.

import sys
import sys_shelve_importer

def show_module_details(module):
    print '  message    :', module.message
    print '  __name__   :', module.__name__
    print '  __package__:', module.__package__
    print '  __file__   :', module.__file__
    print '  __path__   :', module.__path__
    print '  __loader__ :', module.__loader__

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

print
print 'Import of "package.module1":'
import package.module1

print
print 'Examine package.module1 details:'
show_module_details(package.module1)

print
print 'Import of "package.subpackage.module2":'
import package.subpackage.module2

print
print 'Examine package.subpackage.module2 details:'
show_module_details(package.subpackage.module2)

$ python sys_shelve_importer_module.py


Import of "package.module1":
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found i
t as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done
looking for "package.module1" in /tmp/pymotw_import_example.shelve ...
 found it as package.module1
loading source for "package.module1" from shelf
creating a new module object for "package.module1"
imported as regular module
execing source...
package.module1 imported
done

Examine package.module1 details:
  message    : This message is in package.module1
  __name__   : package.module1
  __package__: package
  __file__   : /tmp/pymotw_import_example.shelve/package.module1
  __path__   : /tmp/pymotw_import_example.shelve
  __loader__ : <sys_shelve_importer.ShelveLoader object at 0x100473a90
>

Import of "package.subpackage.module2":
looking for "package.subpackage" in /tmp/pymotw_import_example.shelve
... found it as package.subpackage.__init__
loading source for "package.subpackage" from shelf
creating a new module object for "package.subpackage"
adding path for package
execing source...
package.subpackage imported
done
looking for "package.subpackage.module2" in /tmp/pymotw_import_example
.shelve ... found it as package.subpackage.module2
loading source for "package.subpackage.module2" from shelf
creating a new module object for "package.subpackage.module2"
imported as regular module
execing source...
package.subpackage.module2 imported
done

Examine package.subpackage.module2 details:
  message    : This message is in package.subpackage.module2
  __name__   : package.subpackage.module2
  __package__: package.subpackage
  __file__   : /tmp/pymotw_import_example.shelve/package.subpackage.mo
dule2
  __path__   : /tmp/pymotw_import_example.shelve
  __loader__ : <sys_shelve_importer.ShelveLoader object at 0x1006db990
>

Reloading¶

Reloading a module is handled slightly differently. Instead of creating a new module object, the existing module is re-used.

import sys
import sys_shelve_importer

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

print 'First import of "package":'
import package

print
print 'Reloading "package":'
reload(package)

By re-using the same object, existing references to the module are preserved even if class or function definitions are modified by the reload.

$ python sys_shelve_importer_reload.py

First import of "package":
new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found i
t as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done

Reloading "package":
looking for "package" in /tmp/pymotw_import_example.shelve ... found i
t as package.__init__
loading source for "package" from shelf
reusing existing module from previous import of "package"
adding path for package
execing source...
package imported
done

Import Errors¶

When a module cannot be located by any finder, ImportError is raised by the main import code.

import sys
import sys_shelve_importer

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

try:
    import package.module3
except ImportError, e:
    print 'Failed to import:', e

Other errors during the import are propagated.

$ python sys_shelve_importer_missing.py

new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found i
t as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done
looking for "package.module3" in /tmp/pymotw_import_example.shelve ...
 not found
Failed to import: No module named module3

Package Data¶

In addition to defining the API loading executable Python code, PEP 302 defines an optional API for retrieving package data intended for distributing data files, documentation, and other non-code resources used by a package. By implementing get_data(), a loader can allow calling applications to support retrieval of data associated with the package without considering how the package is actually installed (especially without assuming that the package is stored as files on a filesystem).

import sys
import sys_shelve_importer
import os
import pkgutil

filename = '/tmp/pymotw_import_example.shelve'
sys.path_hooks.append(sys_shelve_importer.ShelveFinder)
sys.path.insert(0, filename)

import package

readme_path = os.path.join(package.__path__[0], 'README')

#readme = package.__loader__.get_data(readme_path)
readme = pkgutil.get_data('package', 'README')
print readme

foo_path = os.path.join(package.__path__[0], 'foo')
#foo = package.__loader__.get_data(foo_path)
foo = pkgutil.get_data('package', 'foo')
print foo

get_data() takes a path based on the module or package that owns the data, and returns the contents of the resource “file” as a string, or raises IOError if the resource does not exist.

$ python sys_shelve_importer_get_data.py

new shelf added to import path: /tmp/pymotw_import_example.shelve
looking for "package" in /tmp/pymotw_import_example.shelve ... found i
t as package.__init__
loading source for "package" from shelf
creating a new module object for "package"
adding path for package
execing source...
package imported
done
looking for data in /tmp/pymotw_import_example.shelve for "/tmp/pymotw
_import_example.shelve/README"

==============
package README
==============

This is the README for ``package``.

looking for data in /tmp/pymotw_import_example.shelve for "/tmp/pymotw
_import_example.shelve/foo"
Traceback (most recent call last):
  File "sys_shelve_importer_get_data.py", line 29, in <module>
    foo = pkgutil.get_data('package', 'foo')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.
7/pkgutil.py", line 583, in get_data
    return loader.get_data(resource_name)
  File "/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys/sys_shelve_im
porter.py", line 116, in get_data
    raise IOError
IOError

Importer Cache¶

Searching through all of the hooks each time a module is imported can become expensive. To save time, sys.path_importer_cache is maintained as a mapping between a path entry and the loader that can use the value to find modules.

import sys
import pprint

print 'PATH:',
pprint.pprint(sys.path)
print
print 'IMPORTERS:'
for name, cache_value in sys.path_importer_cache.items():
    name = name.replace(sys.prefix, '...')
    print '%s: %r' % (name, cache_value)

A cache value of None means to use the default filesystem loader. Each missing directory is associated with an imp.NullImporter instance, since modules cannot be imported from directories that do not exist. In the example output below, several zipimport.zipimporter instances are used to manage EGG files found on the path.

$ python sys_path_importer_cache.py

PATH:['/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys',
 '/Users/dhellmann/Documents/PyMOTW/sphinx-graphviz-paragraphs',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/site-packages/distribute-
0.6.14-py2.7.egg',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/site-packages/pip-0.8.1-p
y2.7.egg',
 '/Users/dhellmann/Envs/pymotw/lib/python27.zip',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site
-packages/distribute-0.6.10-py2.7.egg',
 '/Users/dhellmann/Devel/virtualenvwrapper/virtualenvwrapper',
 '/Users/dhellmann/Devel/virtualenvwrapper/bitbucket',
 '/Users/dhellmann/Devel/virtualenvwrapper/emacs-desktop',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/plat-darwin',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/plat-mac',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/plat-mac/lib-scriptpackag
es',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/lib-tk',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/lib-old',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat
-darwin',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-
tk',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat
-mac',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat
-mac/lib-scriptpackages',
 '/Users/dhellmann/Envs/pymotw/lib/python2.7/site-packages',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site
-packages']

IMPORTERS:
/Users/dhellmann/Documents/PyMOTW/src/PyMOTW/sys: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/lib-old: <imp.NullImporter
object at 0x1002ae0d0>
/Users/dhellmann/Devel/virtualenvwrapper/virtualenvwrapper: None
sys_path_importer_cache.py: <imp.NullImporter object at 0x1002ae0e0>
/Users/dhellmann/Devel/virtualenvwrapper/bitbucket: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/site-packages: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/site-packages/distribute-0.
6.14-py2.7.egg: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/encodings: None
/Users/dhellmann/Envs/pymotw/lib/python2.7: None
/Users/dhellmann/Envs/pymotw/lib/python27.zip: <imp.NullImporter objec
t at 0x1002ae080>
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/plat-darwin: <imp.NullImpor
ter object at 0x1002ae090>
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/site-packages/pip-0.8.1-py2
.7.egg: None
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-p
ackages: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/lib-dynload: None
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-d
arwin: None
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-m
ac/lib-scriptpackages: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/plat-mac: <imp.NullImporter
 object at 0x1002ae0a0>
/Users/dhellmann/Devel/virtualenvwrapper/emacs-desktop: None
/Users/dhellmann/Documents/PyMOTW/sphinx-graphviz-paragraphs: None
.../lib/python2.7/: None
/Users/dhellmann/Envs/pymotw/lib/python2.7/plat-mac/lib-scriptpackages
: <imp.NullImporter object at 0x1002ae0b0>
.../lib/python27.zip: <imp.NullImporter object at 0x1002ae030>
/Users/dhellmann/Envs/pymotw/lib/python2.7/lib-tk: <imp.NullImporter o
bject at 0x1002ae0c0>
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-p
ackages/distribute-0.6.10-py2.7.egg: None
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-m
ac: None

Meta Path¶

The sys.meta_path further extends the sources of potential imports by allowing a finder to be searched before the regular sys.path is scanned. The API for a finder on the meta-path is the same as for a regular path. The difference is that the meta-finder is not limited to a single entry in sys.path, it can search anywhere at all.

import sys
import sys_shelve_importer
import imp


class NoisyMetaImportFinder(object):
    
    def __init__(self, prefix):
        print 'Creating NoisyMetaImportFinder for %s' % prefix
        self.prefix = prefix
        return
    
    def find_module(self, fullname, path=None):
        print 'NoisyMetaImportFinder looking for "%s" with path "%s"' % (fullname, path)
        name_parts = fullname.split('.')
        if name_parts and name_parts[0] == self.prefix:
            print ' ... found prefix, returning loader'
            return NoisyMetaImportLoader(path)
        else:
            print ' ... not the right prefix, cannot load'
        return None


class NoisyMetaImportLoader(object):
    
    def __init__(self, path_entry):
        self.path_entry = path_entry
        return
        
    def load_module(self, fullname):
        print 'loading %s' % fullname
        if fullname in sys.modules:
            mod = sys.modules[fullname]
        else:
            mod = sys.modules.setdefault(fullname, imp.new_module(fullname))

        # Set a few properties required by PEP 302
        mod.__file__ = fullname
        mod.__name__ = fullname
        # always looks like a package
        mod.__path__ = [ 'path-entry-goes-here' ]
        mod.__loader__ = self
        mod.__package__ = '.'.join(fullname.split('.')[:-1])
        
        return mod


# Install the meta-path finder
sys.meta_path.append(NoisyMetaImportFinder('foo'))

# Import some modules that are "found" by the meta-path finder
print
import foo

print
import foo.bar

# Import a module that is not found
print
try:
    import bar
except ImportError, e:
    pass

Each finder on the meta-path is interrogated before sys.path is searched, so there is always an opportunity to have a central importer load modules without explicitly modifying sys.path. Once the module is “found”, the loader API works in the same way as for regular loaders (although this example is truncated for simplicity).

$ python sys_meta_path.py

Creating NoisyMetaImportFinder for foo

NoisyMetaImportFinder looking for "foo" with path "None"
 ... found prefix, returning loader
loading foo

NoisyMetaImportFinder looking for "foo.bar" with path "['path-entry-goes-here']"
 ... found prefix, returning loader
loading foo.bar

NoisyMetaImportFinder looking for "bar" with path "None"
 ... not the right prefix, cannot load

PyMOTW

Page Contents

Navigation

This Page

Examples

Modules and Imports¶

Imported Modules¶

Built-in Modules¶

Import Path¶

Custom Importers¶

Finders¶

Importing from a Shelve¶

Packages¶

Reloading¶

Import Errors¶

Package Data¶

Importer Cache¶

Meta Path¶