`_colorize` module is slow to import, affecting `traceback` and `logging` modules

[danielhollas]

“It is a truth universally acknowledged, that a module in possession of great colors, suffers import time slowdown.

_colorize module is slow to import due to its use of dataclasses

python -Ximporttime -c "import _colorize"

Visualization in tuna

Notice that significant time is also spent executing the _colorize, due to the creation of its dataclasses

❯ uvx python@3.15 -m cProfile -m _colorize | head -15
         5609 function calls (5489 primitive calls) in 0.006 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      9/1    0.003    0.000    0.006    0.006 {built-in method builtins.exec}
        1    0.000    0.000    0.006    0.006 <string>:1(<module>)
        1    0.000    0.000    0.006    0.006 <frozen runpy>:199(run_module)
        1    0.000    0.000    0.006    0.006 <frozen runpy>:65(_run_code)
        1    0.000    0.000    0.006    0.006 _colorize.py:1(<module>)
        7    0.000    0.000    0.005    0.001 dataclasses.py:1432(wrap)
        7    0.000    0.000    0.005    0.001 dataclasses.py:986(_process_class)
        7    0.000    0.000    0.003    0.000 dataclasses.py:478(add_fns_to_class)
        4    0.000    0.000    0.001    0.000 inspect.py:3342(signature)
        4    0.000    0.000    0.001    0.000 inspect.py:3055(from_callable)

The slow import time directly affects (among others) the traceback module which in turn affects logging, which is 40% slower in 3.14 and 3.15 compared to 3.13. That is quite unfortunate since in applications that care about import time, logging module is typically hard to avoid (I originally discovered this when looking at pip's startup time)

 hyperfine -w 10 "python3.13 -c 'import logging'"  "python3.15 -c 'import logging'" 
Benchmark 1: python3.13 -c 'import logging'
  Time (mean ± σ):      21.0 ms ±   3.4 ms    [User: 14.8 ms, System: 5.9 ms]
  Range (min … max):    16.2 ms …  28.2 ms    155 runs
 
Benchmark 2: python3.15 -c 'import logging'
  Time (mean ± σ):      29.6 ms ±   2.1 ms    [User: 22.2 ms, System: 7.0 ms]
  Range (min … max):    26.4 ms …  35.1 ms    100 runs
 
Summary
  python3.13 -c 'import logging' ran
    1.40 ± 0.25 times faster than /home/hollas/.local/share/uv/python/cpython-3.15.0a5-linux-x86_64-gnu/bin/python3.15 -c 'import logging'

It's not very clear how to make this better (besides not using dataclasses). We tried to make traceback lazy in logging, but failed. #112995

[picnixz]

cc @hugovk

[picnixz]

So the problem is colorize being imported by traceback which is imported by logging right? why not making colorize lazy in traceback?

[picnixz]

OTOH, what if instead of a dataclass we use a namedtuple? (not from typing but from collections) or a custom class with slots? While we would lose kw-only support with namedtuples, we can backport the change and avoid changing other modules.

[picnixz]

@StanFromIreland The fact that logging is affected makes it a bug for me as logging may be quite frequent in applications and those applications may also want to retain small startup time.

[danielhollas]

why not making colorize lazy in traceback?

I looked at that and it didn't seem easy as it is used in a bunch of places. But once the lazy import PR lands, it seems doable. But I am not sure if lazy importing in the traceback module is a great idea in general (e.g. you probably don't want to be importing modules in the middle of handling exceptions?).

[picnixz]

Unless I am mistaken, traceback is only meant for rendering tracebacks, so it should be fine. Lazy imports will not be available for 3.14 so it may be a good solution for the backport. How slower is the logging import?

[ZeroIntensity]

How difficult would it be to speed up dataclass creation in general? I have two ideas:

1. Add a C accelerator for dataclasses. This might be maintenance-heavy.
2. Add a way to lazily construct a dataclass, so parsing of the class and whatnot isn't done until __new__ is called for the first time.

Dataclasses are becoming increasingly common, and I think it would be nice if they weren't such a significant performance hit at import time.

[picnixz]

I do not think the problem is @dataclass; the problem is that we import a lot of modules in the dataclasses module:

import re
import sys
import copy
import types
import inspect
import keyword
import itertools
import annotationlib
import abc

We import inspect and enum and those are heavy modules.