pytest-xdist in 70 Seconds: -n auto and the Fixture Trap
Added pytest -n auto and got 219 errors? Session-scoped fixtures run once per xdist worker, not once per run. Here's how to run pytest in parallel with pytest-xdist and fix the fixture trap. Install pytest-xdist, run -n auto, hit the "database already exists" collision, fix it with worker_id, run one-time setup with FileLock, then split test files across machines with HyperExecute. Chapters 0:00 Install and run -n auto 0:20 The session fixture trap 0:30 Fix: worker_id and FileLock 0:48 Scale out with HyperExecute Commands pip install pytest-xdist psutil pip install filelock sqlalchemy-utils psycopg2-binary # used by the snippets below pytest -n auto # physical cores (with psutil) pytest -n logical # logical cores (needs psutil) pytest -n auto --dist loadfile # same file, same worker PYTEST_XDIST_AUTO_NUM_WORKERS=4 pytest -n auto # cap workers in CI One database per worker (conftest.py) import pytest from sqlalchemy_utils import create_database, drop_database BASE = "postgresql://localhost" @pytest.fixture(scope="session") def db_url(worker_id): url = f"{BASE}/shop_test_{worker_id}" create_database(url) yield url drop_database(url) Run setup once per run (conftest.py) import pytest from filelock import FileLock @pytest.fixture(scope="session") def token(tmp_path_factory, worker_id): if worker_id == "master": return login() root = tmp_path_factory.getbasetemp().parent f = root / "token.txt" with FileLock(f"{f}.lock"): if not f.is_file(): f.write_text(login()) return f.read_text() Good to know - worker_id is "gw0", "gw1"… and "master" when xdist is off. PYTEST_XDIST_WORKER holds the same name. - On our 240-test sample: 5.2 s serial, 2.5 s with 8 workers. Parallel runs pay off when tests are slow (I/O, DB, browser). Tiny unit tests can run slower because workers take time to start. - --dist loadgroup with @pytest.mark.xdist_group("name") keeps related tests on one worker. When one machine isn't enough pytest-xdist stops at the cores on a single machine. HyperExecute by TestMu AI spreads your test files across isolated cloud VMs. Each VM runs the test runner, browser, and dependencies together, so tests don't make network hops to a remote grid. You set concurrency in one YAML file, and HyperExecute auto-splits the suite. Each VM is a clean environment that gets wiped after the run, so per-machine setup like databases and caches doesn't collide between machines. Logs and video for every task land in one dashboard. TestMu AI reports runs up to 70% faster than a traditional cloud grid. Start free: https://www.testmuai.com/register/?utm_source=youtube&utm_medium=organic&utm_term=v4FfULhvkPA&utm_campaign=pytestxdist HyperExecute: https://www.testmuai.com/hyperexecute/ pytest on HyperExecute docs: https://www.testmuai.com/support/docs/pytest-on-hyperexecute/ pytest-xdist docs: https://pytest-xdist.readthedocs.io/ #pytest #Python #SDET




Join the discussion
Have something to add?
Sign in to reply, follow this thread, and earn XP when your answer helps someone.