Analyzing Experiment Results
This tutorial demonstrates how to use HydraFlow's powerful analysis capabilities to work with your experiment results.
Prerequisites
Before you begin this tutorial, you should:
- Understand the basic structure of a HydraFlow application (from the Basic Application tutorial)
- Be familiar with the concept of job definitions (from the Automated Workflows tutorial)
Project Setup
We'll start by running several experiments that we can analyze. We'll execute the three jobs defined in the Automated Workflows tutorial:
$ hydraflow run job_sequential
$ hydraflow run job_parallel
$ hydraflow run job_submit
2025/08/24 01:04:00 INFO mlflow.tracking.fluent: Experiment with name 'job_sequential' does not exist. Creating a new experiment.
[2025-08-24 01:04:02,200][HYDRA] Launching 3 jobs locally
[2025-08-24 01:04:02,200][HYDRA] #0 : width=100 height=100
[2025-08-24 01:04:02,325][__main__][INFO] - e195530f001e461888f7eb97bbfc0a17
[2025-08-24 01:04:02,325][__main__][INFO] - {'width': 100, 'height': 100}
[2025-08-24 01:04:02,328][HYDRA] #1 : width=100 height=200
[2025-08-24 01:04:02,413][__main__][INFO] - b8317aa9608a4b0a86d320fcd4d42da7
[2025-08-24 01:04:02,413][__main__][INFO] - {'width': 100, 'height': 200}
[2025-08-24 01:04:02,415][HYDRA] #2 : width=100 height=300
[2025-08-24 01:04:02,496][__main__][INFO] - b9e961a5585340efa20807dad1695d9b
[2025-08-24 01:04:02,496][__main__][INFO] - {'width': 100, 'height': 300}
[2025-08-24 01:04:04,185][HYDRA] Launching 3 jobs locally
[2025-08-24 01:04:04,186][HYDRA] #0 : width=300 height=100
[2025-08-24 01:04:04,320][__main__][INFO] - c6e32107d9584c68843ed05a625e18fd
[2025-08-24 01:04:04,320][__main__][INFO] - {'width': 300, 'height': 100}
[2025-08-24 01:04:04,323][HYDRA] #1 : width=300 height=200
[2025-08-24 01:04:04,418][__main__][INFO] - 1d96078eb8a245ef99834ad766689e24
[2025-08-24 01:04:04,418][__main__][INFO] - {'width': 300, 'height': 200}
[2025-08-24 01:04:04,420][HYDRA] #2 : width=300 height=300
[2025-08-24 01:04:04,510][__main__][INFO] - 39852a762a764f30b4f626e3f3d594f0
[2025-08-24 01:04:04,510][__main__][INFO] - {'width': 300, 'height': 300}
0:00:03 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%
2025/08/24 01:04:06 INFO mlflow.tracking.fluent: Experiment with name 'job_parallel' does not exist. Creating a new experiment.
[2025-08-24 01:04:07,659][HYDRA]
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs
[2025-08-24 01:04:07,659][HYDRA] Launching jobs, sweep output dir :
multirun/01K3CSN833NGJP6549XZEAC05N
[2025-08-24 01:04:07,659][HYDRA] #0 : width=200 height=100
[2025-08-24 01:04:07,659][HYDRA] #1 : width=200 height=200
[2025-08-24 01:04:07,659][HYDRA] #2 : width=200 height=300
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2401_5500f09c85ae41eda1c7d63f7ae44ce1_5b5c9778
751140a58e95e43d2808fa5b for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-cu4dz2gb for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-53ffcyx3 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-91qtrnmd for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-bybp1gqj for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-5vu2x4y5 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-sf_gvl59 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2401_5500f09c85ae41eda1c7d63f7ae44ce1_ba0391e9
e13548e08c99949068d6cebd for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-1n0omtp7 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-peyaocwr for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-cet08gbv for automatic cleanup: unknown
resource type semlock
[2025-08-24 01:04:09,337][__main__][INFO] - a7703d225068472d9048ac37e392e0f4
[2025-08-24 01:04:09,337][__main__][INFO] - {'width': 200, 'height': 200}
[2025-08-24 01:04:09,340][__main__][INFO] - dc762e539aae4e069f1438d0554e86be
[2025-08-24 01:04:09,340][__main__][INFO] - {'width': 200, 'height': 100}
[2025-08-24 01:04:09,559][__main__][INFO] - e069da51d9804950855f1fe65b48f278
[2025-08-24 01:04:09,559][__main__][INFO] - {'width': 200, 'height': 300}
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2401_5500f09c85ae41eda1c7d63f7ae44ce1_ba0391e9
e13548e08c99949068d6cebd for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-cet08gbv for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-peyaocwr for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-1n0omtp7 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2401_5500f09c85ae41eda1c7d63f7ae44ce1_5b5c9778
751140a58e95e43d2808fa5b for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-sf_gvl59 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-5vu2x4y5 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-bybp1gqj for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-91qtrnmd for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-53ffcyx3 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2401-cu4dz2gb for automatic cleanup: unknown
resource type semlock
[2025-08-24 01:04:11,641][HYDRA]
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs
[2025-08-24 01:04:11,642][HYDRA] Launching jobs, sweep output dir :
multirun/01K3CSN833NGJP6549XZEAC05P
[2025-08-24 01:04:11,642][HYDRA] #0 : width=400 height=100
[2025-08-24 01:04:11,642][HYDRA] #1 : width=400 height=200
[2025-08-24 01:04:11,642][HYDRA] #2 : width=400 height=300
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2438_14f977d2ec33401a8ebd822608363f96_d2ea19f4
1c714bb08b90a7ce018e28b2 for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-gop47o67 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-3w5sg2ms for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-l9wuk0y0 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-e3mz_9ck for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-jwcovgob for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-hs2f25wt for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2438_14f977d2ec33401a8ebd822608363f96_224ddeda
6c874da596d7ec46e45271cb for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-rzm1qgsg for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-3i0hawu0 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-39inf5gs for automatic cleanup: unknown
resource type semlock
[2025-08-24 01:04:13,255][__main__][INFO] - d1531dd92942487785246ed4394737bb
[2025-08-24 01:04:13,255][__main__][INFO] - {'width': 400, 'height': 300}
[2025-08-24 01:04:13,363][__main__][INFO] - bf08c8170b70494b92fc102d1ec062d4
[2025-08-24 01:04:13,364][__main__][INFO] - {'width': 400, 'height': 100}
[2025-08-24 01:04:13,534][__main__][INFO] - 6877f1c4996441b9a70527b46ead0e68
[2025-08-24 01:04:13,534][__main__][INFO] - {'width': 400, 'height': 200}
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2438_14f977d2ec33401a8ebd822608363f96_224ddeda
6c874da596d7ec46e45271cb for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-39inf5gs for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-3i0hawu0 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-rzm1qgsg for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register
/dev/shm/joblib_memmapping_folder_2438_14f977d2ec33401a8ebd822608363f96_d2ea19f4
1c714bb08b90a7ce018e28b2 for automatic cleanup: unknown resource type folder
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-hs2f25wt for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-jwcovgob for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-e3mz_9ck for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-l9wuk0y0 for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-3w5sg2ms for automatic cleanup: unknown
resource type semlock
Traceback (most recent call last):
File
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main
raise ValueError(
f'Cannot register {name} for automatic cleanup: '
f'unknown resource type {rtype}')
ValueError: Cannot register /loky-2438-gop47o67 for automatic cleanup: unknown
resource type semlock
0:00:07 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%
2025/08/24 01:04:15 INFO mlflow.tracking.fluent: Experiment with name 'job_submit' does not exist. Creating a new experiment.
[2025-08-24 01:04:16,981][HYDRA] Launching 2 jobs locally
[2025-08-24 01:04:16,981][HYDRA] #0 : width=250 height=150
[2025-08-24 01:04:17,105][__main__][INFO] - c7d4fd05d47f4af9b70303db5735651a
[2025-08-24 01:04:17,105][__main__][INFO] - {'width': 250, 'height': 150}
[2025-08-24 01:04:17,107][HYDRA] #1 : width=250 height=250
[2025-08-24 01:04:17,193][__main__][INFO] - 2d0320e9f0734bb1b161ed37a618f3d8
[2025-08-24 01:04:17,193][__main__][INFO] - {'width': 250, 'height': 250}
[2025-08-24 01:04:18,760][HYDRA] Launching 2 jobs locally
[2025-08-24 01:04:18,760][HYDRA] #0 : width=350 height=150
[2025-08-24 01:04:18,885][__main__][INFO] - ff78cc474b6a45c684b2d81416eca794
[2025-08-24 01:04:18,885][__main__][INFO] - {'width': 350, 'height': 150}
[2025-08-24 01:04:18,888][HYDRA] #1 : width=350 height=250
[2025-08-24 01:04:18,974][__main__][INFO] - 62fa7219124348989c606bcdfa0bfeae
[2025-08-24 01:04:18,974][__main__][INFO] - {'width': 350, 'height': 250}
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=250', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01K3CSNH72KGS8VF4Z7GKX406B']
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=350', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01K3CSNH72KGS8VF4Z7GKX406C']
After running these commands, our project structure looks like this:
.
├── mlruns
│ ├── 0
│ │ └── meta.yaml
│ ├── 457196445157539545
│ │ ├── 2d0320e9f0734bb1b161ed37a618f3d8
│ │ ├── 62fa7219124348989c606bcdfa0bfeae
│ │ ├── c7d4fd05d47f4af9b70303db5735651a
│ │ ├── ff78cc474b6a45c684b2d81416eca794
│ │ └── meta.yaml
│ ├── 625072603073095599
│ │ ├── 1d96078eb8a245ef99834ad766689e24
│ │ ├── 39852a762a764f30b4f626e3f3d594f0
│ │ ├── b8317aa9608a4b0a86d320fcd4d42da7
│ │ ├── b9e961a5585340efa20807dad1695d9b
│ │ ├── c6e32107d9584c68843ed05a625e18fd
│ │ ├── e195530f001e461888f7eb97bbfc0a17
│ │ └── meta.yaml
│ └── 869298363166370650
│ ├── 6877f1c4996441b9a70527b46ead0e68
│ ├── a7703d225068472d9048ac37e392e0f4
│ ├── bf08c8170b70494b92fc102d1ec062d4
│ ├── d1531dd92942487785246ed4394737bb
│ ├── dc762e539aae4e069f1438d0554e86be
│ ├── e069da51d9804950855f1fe65b48f278
│ └── meta.yaml
├── example.py
├── hydraflow.yaml
└── submit.py
The mlruns directory contains all our experiment data.
Let's explore how to access and analyze this data using HydraFlow's API.
Discovering Runs
Finding Run Directories
HydraFlow provides the iter_run_dirs
function to discover runs in your MLflow tracking directory:
>>> from hydraflow import iter_run_dirs
>>> run_dirs = list(iter_run_dirs("mlruns"))
>>> print(len(run_dirs))
>>> for run_dir in run_dirs[:4]:
... print(run_dir)
16
mlruns/457196445157539545/2d0320e9f0734bb1b161ed37a618f3d8
mlruns/457196445157539545/62fa7219124348989c606bcdfa0bfeae
mlruns/457196445157539545/ff78cc474b6a45c684b2d81416eca794
mlruns/457196445157539545/c7d4fd05d47f4af9b70303db5735651a
This function finds all run directories in your MLflow tracking directory, making it easy to collect runs for analysis.
Filtering by Experiment Name
You can filter runs by experiment name to focus on specific experiments:
>>> print(len(list(iter_run_dirs("mlruns", "job_sequential"))))
>>> names = ["job_sequential", "job_parallel"]
>>> print(len(list(iter_run_dirs("mlruns", names))))
>>> print(len(list(iter_run_dirs("mlruns", "job_*"))))
6
12
16
As shown above, you can:
- Filter by a single experiment name
- Provide a list of experiment names
- Use pattern matching with wildcards
Working with Individual Runs
Loading a Run
The Run class represents a single
experiment run in HydraFlow:
>>> from hydraflow import Run
>>> run_dirs = iter_run_dirs("mlruns")
>>> run_dir = next(run_dirs) # run_dirs is an iterator
>>> run = Run(run_dir)
>>> print(run)
>>> print(type(run))
Run('2d0320e9f0734bb1b161ed37a618f3d8')
<class 'hydraflow.core.run.Run'>
You can also use the load
class method, which accepts both string paths and Path objects:
>>> Run.load(str(run_dir))
>>> print(run)
Run('2d0320e9f0734bb1b161ed37a618f3d8')
Accessing Run Information
Each Run instance provides access to run information and configuration:
>>> print(run.info.run_dir)
>>> print(run.info.run_id)
>>> print(run.info.job_name) # Hydra job name = MLflow experiment name
mlruns/457196445157539545/2d0320e9f0734bb1b161ed37a618f3d8
2d0320e9f0734bb1b161ed37a618f3d8
job_submit
The configuration is available through the cfg attribute:
>>> print(run.cfg)
{'width': 250, 'height': 250}
Type-Safe Configuration Access
For better IDE integration and type checking, you can specify the configuration type:
from dataclasses import dataclass
@dataclass
class Config:
width: int = 1024
height: int = 768
>>> run = Run[Config](run_dir)
>>> print(run)
Run('2d0320e9f0734bb1b161ed37a618f3d8')
When you use Run[Config], your IDE will recognize run.cfg as
having the specified type, enabling autocompletion and type checking.
Accessing Configuration Values
The get method provides a unified interface to access values from a run:
>>> print(run.get("width"))
>>> print(run.get("height"))
250
250
Adding Custom Implementations
Basic Implementation
You can extend runs with custom implementation classes to add domain-specific functionality:
from pathlib import Path
class Impl:
root_dir: Path
def __init__(self, root_dir: Path):
self.root_dir = root_dir
def __repr__(self) -> str:
return f"Impl({self.root_dir.stem!r})"
>>> run = Run[Config, Impl](run_dir, Impl)
>>> print(run)
Run[Impl]('2d0320e9f0734bb1b161ed37a618f3d8')
The implementation is lazily initialized when you first access the impl attribute:
>>> print(run.impl)
>>> print(run.impl.root_dir)
Impl('artifacts')
mlruns/457196445157539545/2d0320e9f0734bb1b161ed37a618f3d8/artifacts
Configuration-Aware Implementation
Implementations can also access the run's configuration:
from dataclasses import dataclass, field
@dataclass
class Size:
root_dir: Path = field(repr=False)
cfg: Config
@property
def size(self) -> int:
return self.cfg.width * self.cfg.height
def is_large(self) -> bool:
return self.size > 100000
>>> run = Run[Config, Size].load(run_dir, Size)
>>> print(run)
>>> print(run.impl)
>>> print(run.impl.size)
Run[Size]('2d0320e9f0734bb1b161ed37a618f3d8')
Size(cfg={'width': 250, 'height': 250})
62500
This allows you to define custom analysis methods that use both the run's artifacts and its configuration.
Working with Multiple Runs
Creating a Run Collection
The RunCollection
class helps you analyze multiple runs:
>>> run_dirs = iter_run_dirs("mlruns")
>>> rc = Run[Config, Size].load(run_dirs, Size)
>>> print(rc)
RunCollection(Run[Size], n=16)
The load method automatically creates a RunCollection when
given multiple run directories.
Basic Run Collection Operations
You can perform basic operations on a collection:
>>> print(rc.first())
>>> print(rc.last())
Run[Size]('2d0320e9f0734bb1b161ed37a618f3d8')
Run[Size]('d1531dd92942487785246ed4394737bb')
Filtering Runs
The filter method
lets you select runs based on various criteria:
>>> print(rc.filter(width=400))
RunCollection(Run[Size], n=3)
You can use lists to filter by multiple values (OR logic):
>>> print(rc.filter(height=[100, 300]))
RunCollection(Run[Size], n=8)
Tuples create range filters (inclusive):
>>> print(rc.filter(height=(100, 300)))
RunCollection(Run[Size], n=16)
You can even use custom filter functions:
>>> print(rc.filter(lambda r: r.impl.is_large()))
RunCollection(Run[Size], n=1)
Finding Specific Runs
The get method
returns a single run matching your criteria:
>>> run = rc.get(width=250, height=(100, 200))
>>> print(run)
>>> print(run.impl)
Run[Size]('c7d4fd05d47f4af9b70303db5735651a')
Size(cfg={'width': 250, 'height': 150})
Converting to DataFrames
For data analysis, you can convert runs to a Polars DataFrame:
>>> print(rc.to_frame("width", "height", "size"))
shape: (16, 3)
┌───────┬────────┬────────┐
│ width ┆ height ┆ size │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═══════╪════════╪════════╡
│ 250 ┆ 250 ┆ 62500 │
│ 350 ┆ 250 ┆ 87500 │
│ 350 ┆ 150 ┆ 52500 │
│ 250 ┆ 150 ┆ 37500 │
│ 100 ┆ 100 ┆ 10000 │
│ … ┆ … ┆ … │
│ 400 ┆ 100 ┆ 40000 │
│ 200 ┆ 200 ┆ 40000 │
│ 400 ┆ 200 ┆ 80000 │
│ 200 ┆ 300 ┆ 60000 │
│ 400 ┆ 300 ┆ 120000 │
└───────┴────────┴────────┘
You can add custom columns using callables:
>>> print(rc.to_frame("width", "height", is_large=lambda r: r.impl.is_large()))
shape: (16, 3)
┌───────┬────────┬──────────┐
│ width ┆ height ┆ is_large │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool │
╞═══════╪════════╪══════════╡
│ 250 ┆ 250 ┆ false │
│ 350 ┆ 250 ┆ false │
│ 350 ┆ 150 ┆ false │
│ 250 ┆ 150 ┆ false │
│ 100 ┆ 100 ┆ false │
│ … ┆ … ┆ … │
│ 400 ┆ 100 ┆ false │
│ 200 ┆ 200 ┆ false │
│ 400 ┆ 200 ┆ false │
│ 200 ┆ 300 ┆ false │
│ 400 ┆ 300 ┆ true │
└───────┴────────┴──────────┘
Functions can return lists for multiple values:
>>> def to_list(run: Run) -> list[int]:
... return [2 * run.get("width"), 3 * run.get("height")]
>>> print(rc.to_frame("width", from_list=to_list))
shape: (16, 2)
┌───────┬────────────┐
│ width ┆ from_list │
│ --- ┆ --- │
│ i64 ┆ list[i64] │
╞═══════╪════════════╡
│ 250 ┆ [500, 750] │
│ 350 ┆ [700, 750] │
│ 350 ┆ [700, 450] │
│ 250 ┆ [500, 450] │
│ 100 ┆ [200, 300] │
│ … ┆ … │
│ 400 ┆ [800, 300] │
│ 200 ┆ [400, 600] │
│ 400 ┆ [800, 600] │
│ 200 ┆ [400, 900] │
│ 400 ┆ [800, 900] │
└───────┴────────────┘
Or dictionaries for multiple named columns:
>>> def to_dict(run: Run) -> dict[int, str]:
... width2 = 2 * run.get("width")
... name = f"h{run.get('height')}"
... return {"width2": width2, "name": name}
>>> print(rc.to_frame("width", from_dict=to_dict))
shape: (16, 2)
┌───────┬──────────────┐
│ width ┆ from_dict │
│ --- ┆ --- │
│ i64 ┆ struct[2] │
╞═══════╪══════════════╡
│ 250 ┆ {500,"h250"} │
│ 350 ┆ {700,"h250"} │
│ 350 ┆ {700,"h150"} │
│ 250 ┆ {500,"h150"} │
│ 100 ┆ {200,"h100"} │
│ … ┆ … │
│ 400 ┆ {800,"h100"} │
│ 200 ┆ {400,"h200"} │
│ 400 ┆ {800,"h200"} │
│ 200 ┆ {400,"h300"} │
│ 400 ┆ {800,"h300"} │
└───────┴──────────────┘
Grouping Runs
The group_by
method organizes runs by common attributes:
>>> grouped = rc.group_by("width")
>>> for key, group in grouped.items():
... print(key, group)
250 RunCollection(Run[Size], n=2)
350 RunCollection(Run[Size], n=2)
100 RunCollection(Run[Size], n=3)
300 RunCollection(Run[Size], n=3)
200 RunCollection(Run[Size], n=3)
400 RunCollection(Run[Size], n=3)
You can group by multiple keys:
>>> grouped = rc.group_by("width", "height")
>>> for key, group in grouped.items():
... print(key, group)
(250, 250) RunCollection(Run[Size], n=1)
(350, 250) RunCollection(Run[Size], n=1)
(350, 150) RunCollection(Run[Size], n=1)
(250, 150) RunCollection(Run[Size], n=1)
(100, 100) RunCollection(Run[Size], n=1)
(300, 100) RunCollection(Run[Size], n=1)
(100, 200) RunCollection(Run[Size], n=1)
(100, 300) RunCollection(Run[Size], n=1)
(300, 300) RunCollection(Run[Size], n=1)
(300, 200) RunCollection(Run[Size], n=1)
(200, 100) RunCollection(Run[Size], n=1)
(400, 100) RunCollection(Run[Size], n=1)
(200, 200) RunCollection(Run[Size], n=1)
(400, 200) RunCollection(Run[Size], n=1)
(200, 300) RunCollection(Run[Size], n=1)
(400, 300) RunCollection(Run[Size], n=1)
Adding aggregation functions using the
agg
method transforms the result into a DataFrame:
>>> grouped = rc.group_by("width")
>>> df = grouped.agg(n=lambda runs: len(runs))
>>> print(df)
shape: (6, 2)
┌───────┬─────┐
│ width ┆ n │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═══════╪═════╡
│ 250 ┆ 2 │
│ 350 ┆ 2 │
│ 100 ┆ 3 │
│ 300 ┆ 3 │
│ 200 ┆ 3 │
│ 400 ┆ 3 │
└───────┴─────┘
Summary
In this tutorial, you've learned how to:
- Discover experiment runs in your MLflow tracking directory
- Load and access information from individual runs
- Add custom implementation classes for domain-specific analysis
- Filter, group, and analyze collections of runs
- Convert run data to DataFrames for advanced analysis
These capabilities enable you to efficiently analyze your experiments and extract valuable insights from your machine learning workflows.
Next Steps
Now that you understand HydraFlow's analysis capabilities, you can:
- Dive deeper into the Run Class and Run Collection documentation
- Explore advanced analysis techniques in the Analyzing Results section
- Apply these analysis techniques to your own machine learning experiments