\n",
" Copyable Markdown
\n",
" | Package | Version |\n",
"| ------- | -------- |\n",
"| dask | 2025.1.0 |\n",
"| scanpy | 1.11.3 |\n",
"| anndata | 0.12.1 |\n",
"| h5py | 3.14.0 |\n",
"\n",
"| Dependency | Version |\n",
"| ------------------ | ----------- |\n",
"| kiwisolver | 1.4.8 |\n",
"| defusedxml | 0.7.1 |\n",
"| pure_eval | 0.2.3 |\n",
"| ipykernel | 6.30.0 |\n",
"| asttokens | 3.0.0 |\n",
"| wcwidth | 0.2.13 |\n",
"| pytz | 2025.2 |\n",
"| distributed | 2025.1.0 |\n",
"| ipython | 9.4.0 |\n",
"| igraph | 0.11.9 |\n",
"| pandas | 2.3.1 |\n",
"| numpy | 2.2.6 |\n",
"| prompt_toolkit | 3.0.51 |\n",
"| fsspec | 2025.7.0 |\n",
"| debugpy | 1.8.15 |\n",
"| cloudpickle | 3.1.1 |\n",
"| setuptools | 80.9.0 |\n",
"| numba | 0.61.2 |\n",
"| typing_extensions | 4.14.1 |\n",
"| legacy-api-wrap | 1.4.1 |\n",
"| jedi | 0.19.2 |\n",
"| scikit-learn | 1.7.1 |\n",
"| six | 1.17.0 |\n",
"| jupyter_core | 5.8.1 |\n",
"| comm | 0.2.3 |\n",
"| zict | 3.0.0 |\n",
"| pyzmq | 27.0.0 |\n",
"| cycler | 0.12.1 |\n",
"| platformdirs | 4.3.8 |\n",
"| python-dateutil | 2.9.0.post0 |\n",
"| tornado | 6.5.1 |\n",
"| jupyter_client | 8.6.3 |\n",
"| zarr | 3.1.0 |\n",
"| llvmlite | 0.44.0 |\n",
"| numcodecs | 0.16.1 |\n",
"| pillow | 11.3.0 |\n",
"| executing | 2.2.0 |\n",
"| stack-data | 0.6.3 |\n",
"| leidenalg | 0.10.2 |\n",
"| crc32c | 2.7.1 |\n",
"| texttable | 1.7.0 |\n",
"| donfig | 0.8.1.post1 |\n",
"| session-info2 | 0.2 |\n",
"| natsort | 8.4.0 |\n",
"| psutil | 7.0.0 |\n",
"| parso | 0.8.4 |\n",
"| matplotlib | 3.10.3 |\n",
"| pyparsing | 3.2.3 |\n",
"| Jinja2 | 3.1.6 |\n",
"| Pygments | 2.19.2 |\n",
"| locket | 1.0.0 |\n",
"| toolz | 1.0.0 |\n",
"| charset-normalizer | 3.4.2 |\n",
"| sortedcontainers | 2.4.0 |\n",
"| decorator | 5.2.1 |\n",
"| click | 8.2.1 |\n",
"| threadpoolctl | 3.6.0 |\n",
"| scipy | 1.15.3 |\n",
"| packaging | 25.0 |\n",
"| joblib | 1.5.1 |\n",
"| ipywidgets | 8.1.7 |\n",
"| traitlets | 5.14.3 |\n",
"| msgpack | 1.1.1 |\n",
"| PyYAML | 6.0.2 |\n",
"| MarkupSafe | 3.0.2 |\n",
"| tblib | 3.1.0 |\n",
"\n",
"| Component | Info |\n",
"| --------- | ------------------------------------------------------ |\n",
"| Python | 3.12.6 (main, Sep 9 2024, 22:11:19) [Clang 18.1.8 ] |\n",
"| OS | Linux-5.15.0-134-generic-x86_64-with-glibc2.35 |\n",
"| CPU | 16 logical CPU cores, x86_64 |\n",
"| GPU | ID: 0, Tesla T4, Driver: 550.144.03, Memory: 15360 MiB |\n",
"| Updated | 2025-07-29 13:28 |\n",
" "
],
"text/markdown": [
"| Package | Version |\n",
"| ------- | -------- |\n",
"| dask | 2025.1.0 |\n",
"| scanpy | 1.11.3 |\n",
"| anndata | 0.12.1 |\n",
"| h5py | 3.14.0 |\n",
"\n",
"| Dependency | Version |\n",
"| ------------------ | ----------- |\n",
"| kiwisolver | 1.4.8 |\n",
"| defusedxml | 0.7.1 |\n",
"| pure_eval | 0.2.3 |\n",
"| ipykernel | 6.30.0 |\n",
"| asttokens | 3.0.0 |\n",
"| wcwidth | 0.2.13 |\n",
"| pytz | 2025.2 |\n",
"| distributed | 2025.1.0 |\n",
"| ipython | 9.4.0 |\n",
"| igraph | 0.11.9 |\n",
"| pandas | 2.3.1 |\n",
"| numpy | 2.2.6 |\n",
"| prompt_toolkit | 3.0.51 |\n",
"| fsspec | 2025.7.0 |\n",
"| debugpy | 1.8.15 |\n",
"| cloudpickle | 3.1.1 |\n",
"| setuptools | 80.9.0 |\n",
"| numba | 0.61.2 |\n",
"| typing_extensions | 4.14.1 |\n",
"| legacy-api-wrap | 1.4.1 |\n",
"| jedi | 0.19.2 |\n",
"| scikit-learn | 1.7.1 |\n",
"| six | 1.17.0 |\n",
"| jupyter_core | 5.8.1 |\n",
"| comm | 0.2.3 |\n",
"| zict | 3.0.0 |\n",
"| pyzmq | 27.0.0 |\n",
"| cycler | 0.12.1 |\n",
"| platformdirs | 4.3.8 |\n",
"| python-dateutil | 2.9.0.post0 |\n",
"| tornado | 6.5.1 |\n",
"| jupyter_client | 8.6.3 |\n",
"| zarr | 3.1.0 |\n",
"| llvmlite | 0.44.0 |\n",
"| numcodecs | 0.16.1 |\n",
"| pillow | 11.3.0 |\n",
"| executing | 2.2.0 |\n",
"| stack-data | 0.6.3 |\n",
"| leidenalg | 0.10.2 |\n",
"| crc32c | 2.7.1 |\n",
"| texttable | 1.7.0 |\n",
"| donfig | 0.8.1.post1 |\n",
"| session-info2 | 0.2 |\n",
"| natsort | 8.4.0 |\n",
"| psutil | 7.0.0 |\n",
"| parso | 0.8.4 |\n",
"| matplotlib | 3.10.3 |\n",
"| pyparsing | 3.2.3 |\n",
"| Jinja2 | 3.1.6 |\n",
"| Pygments | 2.19.2 |\n",
"| locket | 1.0.0 |\n",
"| toolz | 1.0.0 |\n",
"| charset-normalizer | 3.4.2 |\n",
"| sortedcontainers | 2.4.0 |\n",
"| decorator | 5.2.1 |\n",
"| click | 8.2.1 |\n",
"| threadpoolctl | 3.6.0 |\n",
"| scipy | 1.15.3 |\n",
"| packaging | 25.0 |\n",
"| joblib | 1.5.1 |\n",
"| ipywidgets | 8.1.7 |\n",
"| traitlets | 5.14.3 |\n",
"| msgpack | 1.1.1 |\n",
"| PyYAML | 6.0.2 |\n",
"| MarkupSafe | 3.0.2 |\n",
"| tblib | 3.1.0 |\n",
"\n",
"| Component | Info |\n",
"| --------- | ------------------------------------------------------ |\n",
"| Python | 3.12.6 (main, Sep 9 2024, 22:11:19) [Clang 18.1.8 ] |\n",
"| OS | Linux-5.15.0-134-generic-x86_64-with-glibc2.35 |\n",
"| CPU | 16 logical CPU cores, x86_64 |\n",
"| GPU | ID: 0, Tesla T4, Driver: 550.144.03, Memory: 15360 MiB |\n",
"| Updated | 2025-07-29 13:28 |"
],
"text/plain": [
"dask\t2025.1.0\n",
"scanpy\t1.11.3\n",
"anndata\t0.12.1\n",
"h5py\t3.14.0\n",
"----\t----\n",
"kiwisolver\t1.4.8\n",
"defusedxml\t0.7.1\n",
"pure_eval\t0.2.3\n",
"ipykernel\t6.30.0\n",
"asttokens\t3.0.0\n",
"wcwidth\t0.2.13\n",
"pytz\t2025.2\n",
"distributed\t2025.1.0\n",
"ipython\t9.4.0\n",
"igraph\t0.11.9\n",
"pandas\t2.3.1\n",
"numpy\t2.2.6\n",
"prompt_toolkit\t3.0.51\n",
"fsspec\t2025.7.0\n",
"debugpy\t1.8.15\n",
"cloudpickle\t3.1.1\n",
"setuptools\t80.9.0\n",
"numba\t0.61.2\n",
"typing_extensions\t4.14.1\n",
"legacy-api-wrap\t1.4.1\n",
"jedi\t0.19.2\n",
"scikit-learn\t1.7.1\n",
"six\t1.17.0\n",
"jupyter_core\t5.8.1\n",
"comm\t0.2.3\n",
"zict\t3.0.0\n",
"pyzmq\t27.0.0\n",
"cycler\t0.12.1\n",
"platformdirs\t4.3.8\n",
"python-dateutil\t2.9.0.post0\n",
"tornado\t6.5.1\n",
"jupyter_client\t8.6.3\n",
"zarr\t3.1.0\n",
"llvmlite\t0.44.0\n",
"numcodecs\t0.16.1\n",
"pillow\t11.3.0\n",
"executing\t2.2.0\n",
"stack-data\t0.6.3\n",
"leidenalg\t0.10.2\n",
"crc32c\t2.7.1\n",
"texttable\t1.7.0\n",
"donfig\t0.8.1.post1\n",
"session-info2\t0.2\n",
"natsort\t8.4.0\n",
"psutil\t7.0.0\n",
"parso\t0.8.4\n",
"matplotlib\t3.10.3\n",
"pyparsing\t3.2.3\n",
"Jinja2\t3.1.6\n",
"Pygments\t2.19.2\n",
"locket\t1.0.0\n",
"toolz\t1.0.0\n",
"charset-normalizer\t3.4.2\n",
"sortedcontainers\t2.4.0\n",
"decorator\t5.2.1\n",
"click\t8.2.1\n",
"threadpoolctl\t3.6.0\n",
"scipy\t1.15.3\n",
"packaging\t25.0\n",
"joblib\t1.5.1\n",
"ipywidgets\t8.1.7\n",
"traitlets\t5.14.3\n",
"msgpack\t1.1.1\n",
"PyYAML\t6.0.2\n",
"MarkupSafe\t3.0.2\n",
"tblib\t3.1.0\n",
"----\t----\n",
"Python\t3.12.6 (main, Sep 9 2024, 22:11:19) [Clang 18.1.8 ]\n",
"OS\tLinux-5.15.0-134-generic-x86_64-with-glibc2.35\n",
"CPU\t16 logical CPU cores, x86_64\n",
"GPU\tID: 0, Tesla T4, Driver: 550.144.03, Memory: 15360 MiB\n",
"Updated\t2025-07-29 13:28"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from __future__ import annotations\n",
"\n",
"from pathlib import Path\n",
"\n",
"import anndata as ad\n",
"import dask.distributed as dd\n",
"import h5py\n",
"import pooch\n",
"\n",
"import scanpy as sc\n",
"\n",
"sc.logging.print_header()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, we'll be working with a moderately large dataset of 1.4 million cells taken from: [COVID-19 immune features revealed by a large-scale single-cell transcriptome atlas](https://cellxgene.cziscience.com/collections/0a839c4b-10d0-4d64-9272-684c49a2c8ba)\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"cell_atlas_path = Path(\n",
" pooch.retrieve(\n",
" \"https://datasets.cellxgene.cziscience.com/82eac9c1-485f-4e21-ab21-8510823d4f6e.h5ad\",\n",
" known_hash=\"sha256:0b24babfb34b4af87a76806039afa3513c2c04c9045e2a9fb31a6e9350b1fabe\",\n",
" fname=\"cell_atlas.h5ad\",\n",
" path=\"data\",\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more information on using distributed computing via `dask`, please see their [documentation](https://docs.dask.org/en/stable/deploying-python.html). In short, one needs to define both a cluster and a client to have some degree of control over the compute resources dask will use. It's very likely you will have to tune the number of workers and amount of memory per worker along with your chunk sizes.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/ubuntu/home_drive/volume/scanpy-tutorials/venv/lib/python3.12/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use.\n",
"Perhaps you already have a cluster running?\n",
"Hosting the HTTP server on port 43531 instead\n",
" warnings.warn(\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2025-07-29 14:03:53,481 - distributed.scheduler - WARNING - Worker failed to heartbeat for 421s; attempting restart: