Working to make Python lazy

Python 3.15a7, which is now just a uv python install 3.15 away on all major platforms, has lazy imports! This exciting feature, proposed in PEP 810, promises to make CLI applications faster (especially when using flags like --help), and could make a lot of large code with lots of imports that don’t always get used faster too. Unlike the earlier, failed attempt, this requires libraries to put in some work. I’ve developed a helper tool to make it easy; I’d like to cover what lazy imports are and how to use my tool. Since this is the first library that I used AI heavily in developing, the second half of the post will cover how my experience with AI for a task like this went.TL;DR: run uvx flake8-lazy --apply=list to make your code magically faster on Python 3.15!What is a lazy import?Imagine you have a file like this, with a standard Python argparse CLI:import argparse import numpy def main(): parser = argparse.ArgumentParser() parser.add_argument("--foo", action="store_true") args = parser.parse_args() if args.foo: print(numpy.array([1, 2, 3])) What happens if you run this with --help? The numpy library will be imported, even though it is never used. If you are using modern uv tooling, this can be even worse, since uv doesn’t pre-compile bytecode unless you ask it to; that makes the install faster, but imports are slower the first time.The above is just one example; this can also happen when you have this common pattern:# __init__.py from . import a from . import b __all__ = ["a", "b"] The idea behind this is that a user can just use lib.a.stuff with just import lib, rather than import lib.a, but you pay the cost of import even if they never use all the imports. Some libraries, like rich, are careful to avoid this and ask users to import explicitly, but many older libraries did this.And there are also libraries that can do multiple things (like CLI libraries with subcommands), but you don’t need the dependencies for every subcommand.How to use Python 3.15’s lazy importsTake the first example. In Python 3.15, you can now write:lazy import argparse lazy import numpy def main(): parser = argparse.ArgumentParser() parser.add_argument("--foo", action="store_true") args = parser.parse_args() if args.foo: print(numpy.array([1, 2, 3])) Now, both imports are “lazy”, meaning nothing happens at all when you import them. They might not even be installed. The first time you try to use the object, though, it becomes a real, imported object. So if you do --help, numpy is never accessed and never imported.There is also a backward-compatible syntax:__lazy_modules__ = ["argparse", "numpy"] import argparse import numpy This works on older Pythons (it’s just not lazy), and you can also dynamically generate or manipulate that list if you want. Linters like Ruff have already updated to allow this to be placed above your imports without triggering a lint violation.I should mention there’s a flag and a variable to make Python treat all imports as lazy, -X lazy_imports=all and PYTHON_LAZY_IMPORTS=all (also normal and none). That’s mostly for testing.Why not lazy?Shouldn’t you just mark everything as lazy? You don’t have to. There are some modules that have side effects when you import them; if those side effects need to happen at the import site, then those can’t be lazy. This pattern, for example, can’t be lazy:try: import numpy except ModuleNotFoundError: ... The error here will move to the first usage of something from numpy. There is a semi-lazy alternative:import importlib.util if importlib.util.find_spec("numpy") is None: ... # whatever you wanted to do if numpy is missing lazy import numpy This is slightly more expensive than doing nothing at all (which is why lazy importing doesn’t do it), will import packages to get to subpackages (a.b imports a), and some types of import errors won’t trigger when just finding the spec (for the above example, numpy._core could be missing/broken if someone didn’t compile numpy correctly - this is rare, though). Regardless, this is a pretty good way to check to see if a package is installed.The other case you don’t need lazy is if you use something at top level. For example:lazy import re REGEX = re.compile(...) # not lazy here Here, the lazy import is not needed, since you can’t process the file without importing this anyway. You can work around this by caching:import functools lazy import re @functools.cache def regex() -> re.Pattern: return re.compile(...) from __future__ import annotations __lazy_modules__ = ["re"] import functools import re @functools.cache def regex() -> re.Pattern: return re.compile(...) Notice I don’t need from __future__ import annotations to make this work; the annotation doesn’t cause the re module to be loaded because in Python 3.14 annotations became lazy by default in that version.You can make these sorts of imports lazy, but you are just moving the import errors for no good reason, so it’s a bit better not to.If you want to make everything in a file lazy, you can do it like this:class AllLazy: @staticmethod def __contains__(_: str) -> bool: return True __lazy_modules__ = AllLazy() This simply is used by testing with in on full module names, and you can put your own object in here. (The static tool below doesn’t look for this yet.)A tool to helpSo libraries ideally should start adding these __lazy_modules__, but it’s a little more complex than just putting all modules into it. So I wrote a tool, flake8-lazy, to help with figuring out exactly what to add, and with keeping it tidy. This is the first library I’ve used AI tools heavily in developing (I’ve started using them to help maintain plumbum, but that’s not from scratch), so I’ll end with a section about how that went (very well). I’ve developed flake8-errmsg in the past, so it’s not my first flake8 plugin. Like that project, there’s also a built-in standalone runner; early in the 3.15 lifecycle, I rather expect that to be the main way to use it.To use it:# Show flake8-style errors uvx flake8-lazy <filenames> # Show the lines you need to add uvx flake8-lazy --format=lazy-modules # Just add it! uvx flake8-lazy --apply=list <filenames> # Show flake8-style errors pipx run flake8-lazy <filenames> # Show the lines you need to add pipx run flake8-lazy --format=lazy-modules # Just add it! pipx run flake8-lazy --apply=list <filenames> This will report the errors (noqa doesn’t work with the simple runner).Here are the errors currently implemented (0.6.0):Code1xx: Missing lazy declarationsLZY101stdlib module should be listed in __lazy_modules__LZY102third-party or local module should be listed in __lazy_modules__These try to find things that are not used at top level, and suggest they be added to your __lazy_modules__ (the lazy syntax works too). Currently, they assume annotations do not trigger an import (since flake8, unlike Ruff, doesn’t know the minimum Python version you are targeting, it can’t tell if it’s 3.14+ or not).Code2xx: __lazy_modules__ validationLZY201__lazy_modules__ is not sortedLZY202module listed in __lazy_modules__ is never importedLZY203module listed in __lazy_modules__ is duplicatedLZY204__lazy_modules__ is assigned after importing modules it namesLZY205module listed in __lazy_modules__ must be an absolute nameThese look for general problems specifically with __lazy_modules__.Code3xx: Native lazy keyword (Python 3.15+)LZY301lazy import inside suppress(ImportError) is misleadingLZY302module declared lazy by both lazy keyword and __lazy_modules__LZY303module imported both eagerly and lazilyThese look for issues specific to Python 3.15+’s new syntax. These only work on 3.15+ as the host Python, as well. You can tell uv to use it already with --python=3.15.Code4xx: Lazy import safety and semanticsLZY401module is declared lazy but accessed at the top levelLZY402module is an enclosing package for this file and should not be lazyLZY401 is the opposite of the LZY101/LZY102 checks, basically; if you access something at top level, you might as well not make it lazy. This might get moved to a 9xx check, as it’s not problematic to do it, and the check system could be wrong.TipsDon’t apply this to test suites.Look for opportunities to make things lazy if they are not listed here. The re example above is an example of this. But also check the actual imported libraries, too - one library may import another anyway (quite a few libraries import re, including typing, making that one really hard to avoid! re is pretty slow, too, sadly). You can do this with -X importtime. Anything that is lazy and never gets imported will not show up here anymore. You can force lazy imports off to see the difference. You can also force lazy imports on to see how much time you might save before starting.Type checkers always treat TYPE_CHECKING as True, so you can avoid importing typing with this trick:TYPE_CHECKING = False if TYPE_CHECKING: ... With Ruff, you can even enforce this with the TID251 check:[tool.ruff.lint.flake8-tidy-imports.banned-api] "typing.TYPE_CHECKING".msg = "Use TYPE_CHECKING=False instead" The __lazy_modules__ system is completely dynamic (just needs a __contains__ method for absolute module names); the checks don’t handle anything dynamic here. The most common use case, relative imports, can be left static:__lazy_modules__ = [f"{__spec__.parent}.thing"] from . import thing Note that __package__ is the older form of __spec__.parent. Don’t use this on __main__.py, use absolute imports on that one (mypy will notice that __spec__ can be None on this file).ResultsI tried running this tool on its own source code, and managed to get the --help flag 2x faster on Python 3.15. On cibuildwheel, this managed a 3-4x speedup for things like --help and --print-build-identifiers. Hugo, the release manager for Python 3.14, was able to get a bit more speed in a PR to my PR. Here are some of the results I’ve gotten so far; for each case, I’m checking --help,