Analyzing Experiment Results
This tutorial demonstrates how to use HydraFlow's powerful analysis capabilities to work with your experiment results.
Prerequisites
Before you begin this tutorial, you should:
- Understand the basic structure of a HydraFlow application (from the Basic Application tutorial)
- Be familiar with the concept of job definitions (from the Automated Workflows tutorial)
Project Setup
We'll start by running several experiments that we can analyze. We'll execute the three jobs defined in the Automated Workflows tutorial:
$ hydraflow run job_sequential
$ hydraflow run job_parallel
$ hydraflow run job_submit
[2025-12-05 23:13:42,669][HYDRA] Launching 3 jobs locally
[2025-12-05 23:13:42,669][HYDRA] #0 : width=100 height=100
2025/12/05 23:13:43 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:43 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:43 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:43 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade ->
451aebb31d03, add metric step
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 451aebb31d03
-> 90e64c465722, migrate user column to tags
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 90e64c465722
-> 181f10493468, allow nulls for metric values
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 181f10493468
-> df50e92ffc5e, Add Experiment Tags Table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade df50e92ffc5e
-> 7ac759974ad8, Update run tags with larger limit
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 7ac759974ad8
-> 89d4b8295536, create latest metrics table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 89d4b8295536
-> 2b4d017a5e9b, add model registry tables to db
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 2b4d017a5e9b
-> cfd24bdc0731, Update run status constraint with killed
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade cfd24bdc0731
-> 0a8213491aaa, drop_duplicate_killed_constraint
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 0a8213491aaa
-> 728d730b5ebd, add registered model tags table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 728d730b5ebd
-> 27a6a02d2cf1, add model version tags table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 27a6a02d2cf1
-> 84291f40a231, add run_link to model_version
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 84291f40a231
-> a8c4a736bde6, allow nulls for run_id
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade a8c4a736bde6
-> 39d1c3be5f05, add_is_nan_constraint_for_metrics_tables_if_necessary
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 39d1c3be5f05
-> c48cb773bb87, reset_default_value_for_is_nan_in_metrics_table_for_mysql
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade c48cb773bb87
-> bd07f7e963c5, create index on run_uuid
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade bd07f7e963c5
-> 0c779009ac13, add deleted_time field to runs table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 0c779009ac13
-> cc1f77228345, change param value length to 500
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade cc1f77228345
-> 97727af70f4d, Add creation_time and last_update_time to experiments table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 97727af70f4d
-> 3500859a5d39, Add Model Aliases table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 3500859a5d39
-> 7f2a7d5fae7d, add datasets inputs input_tags tables
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 7f2a7d5fae7d
-> 2d6e25af4d3e, increase max param val length from 500 to 8000
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 2d6e25af4d3e
-> acf3f17fdcc7, add storage location field to model versions
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade acf3f17fdcc7
-> 867495a8f9d4, add trace tables
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 867495a8f9d4
-> 5b0e9adcef9c, add cascade deletion to trace tables foreign keys
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 5b0e9adcef9c
-> 4465047574b1, increase max dataset schema size
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 4465047574b1
-> f5a4f2784254, increase run tag value limit to 8000
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade f5a4f2784254
-> 0584bdc529eb, add cascading deletion to datasets from experiments
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 0584bdc529eb
-> 400f98739977, add logged model tables
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 400f98739977
-> 6953534de441, add step to inputs table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 6953534de441
-> bda7b8c39065, increase_model_version_tag_value_limit
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade bda7b8c39065
-> cbc13b556ace, add V3 trace schema columns
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade cbc13b556ace
-> 770bee3ae1dd, add assessments table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 770bee3ae1dd
-> a1b2c3d4e5f6, add spans table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade a1b2c3d4e5f6
-> de4033877273, create entity_associations table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade de4033877273
-> 1a0cddfcaa16, Add webhooks and webhook_events tables
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 1a0cddfcaa16
-> 534353b11cbc, add scorer tables
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 534353b11cbc
-> 71994744cf8e, add evaluation datasets
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 71994744cf8e
-> 3da73c924c2f, add outputs to dataset record
2025/12/05 23:13:43 INFO alembic.runtime.migration: Running upgrade 3da73c924c2f
-> bf29a5ff90ea, add jobs table
2025/12/05 23:13:43 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:43 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:43 INFO mlflow.tracking.fluent: Experiment with name
'job_sequential' does not exist. Creating a new experiment.
[2025-12-05 23:13:43,573][__main__][INFO] - d7221d5bbfc5420fa4a8327c7b26da25
[2025-12-05 23:13:43,573][__main__][INFO] - {'width': 100, 'height': 100}
[2025-12-05 23:13:43,582][HYDRA] #1 : width=100 height=200
[2025-12-05 23:13:43,649][__main__][INFO] - ec3f96530ad04faba52248499519b831
[2025-12-05 23:13:43,649][__main__][INFO] - {'width': 100, 'height': 200}
[2025-12-05 23:13:43,655][HYDRA] #2 : width=100 height=300
[2025-12-05 23:13:43,720][__main__][INFO] - 01b87322a28f4eb0b41e85777f5f8476
[2025-12-05 23:13:43,720][__main__][INFO] - {'width': 100, 'height': 300}
[2025-12-05 23:13:45,249][HYDRA] Launching 3 jobs locally
[2025-12-05 23:13:45,249][HYDRA] #0 : width=300 height=100
2025/12/05 23:13:45 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:45 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:45 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:45 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:45 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:45 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
[2025-12-05 23:13:45,712][__main__][INFO] - 53000c83a5d740a68ac7d9e2e4e9e34f
[2025-12-05 23:13:45,712][__main__][INFO] - {'width': 300, 'height': 100}
[2025-12-05 23:13:45,721][HYDRA] #1 : width=300 height=200
[2025-12-05 23:13:45,787][__main__][INFO] - efd39079f47f4e4b9f30303ba15b40ed
[2025-12-05 23:13:45,787][__main__][INFO] - {'width': 300, 'height': 200}
[2025-12-05 23:13:45,794][HYDRA] #2 : width=300 height=300
[2025-12-05 23:13:45,862][__main__][INFO] - e82544637cae4c129690e858d286e8cc
[2025-12-05 23:13:45,862][__main__][INFO] - {'width': 300, 'height': 300}
0:00:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%
[2025-12-05 23:13:48,649][HYDRA]
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs
[2025-12-05 23:13:48,649][HYDRA] Launching jobs, sweep output dir :
multirun/01KBRCP0F371673F9920MJYANH
[2025-12-05 23:13:48,649][HYDRA] #0 : width=200 height=100
[2025-12-05 23:13:48,649][HYDRA] #1 : width=200 height=200
[2025-12-05 23:13:48,650][HYDRA] #2 : width=200 height=300
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0NThfOWNkYjhmNWM0OTk3NGU4ZTljYm
YzMmM4OWUxNTEyZmVfMjI5ZDg3MTFjZjhhNGZkN2E0YzZlMWE3NjVkZjM2NGE="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC16enphZHNkOQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1iamRpMWRmMw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1nbmE3b3gyeQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1kZWVjc2s1dg=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1rNzlmcHl3MA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1icGpoaTBiag=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0NThfOWNkYjhmNWM0OTk3NGU4ZTljYm
YzMmM4OWUxNTEyZmVfZjExYTcxZGVkZmMwNDg2NWJiMmFmMmMwOGNiNWI0NGM="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC03aGR4cDN6aQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC10ZXloM3hncw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC0zcF85a3hlMA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
2025/12/05 23:13:50 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:50 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:50 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:50 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:50 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:50 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:50 INFO mlflow.tracking.fluent: Experiment with name
'job_parallel' does not exist. Creating a new experiment.
[2025-12-05 23:13:50,360][__main__][INFO] - 19aa3d2c1faf4cd9a94afb24cabf8da3
[2025-12-05 23:13:50,360][__main__][INFO] - {'width': 200, 'height': 200}
2025/12/05 23:13:50 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:50 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:50 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:50 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:50 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:50 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
[2025-12-05 23:13:50,762][__main__][INFO] - 1c9b9ec3f27c427a8a0efff1d768577f
[2025-12-05 23:13:50,762][__main__][INFO] - {'width': 200, 'height': 100}
2025/12/05 23:13:51 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:51 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:51 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:51 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:51 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:51 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
[2025-12-05 23:13:51,185][__main__][INFO] - f0538eb972e84ff5bd27bee658b4194d
[2025-12-05 23:13:51,185][__main__][INFO] - {'width': 200, 'height': 300}
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0NThfOWNkYjhmNWM0OTk3NGU4ZTljYm
YzMmM4OWUxNTEyZmVfZjExYTcxZGVkZmMwNDg2NWJiMmFmMmMwOGNiNWI0NGM="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC0zcF85a3hlMA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC10ZXloM3hncw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC03aGR4cDN6aQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0NThfOWNkYjhmNWM0OTk3NGU4ZTljYm
YzMmM4OWUxNTEyZmVfMjI5ZDg3MTFjZjhhNGZkN2E0YzZlMWE3NjVkZjM2NGE="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1icGpoaTBiag=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1rNzlmcHl3MA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1kZWVjc2s1dg=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1nbmE3b3gyeQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC1iamRpMWRmMw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ1OC16enphZHNkOQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
[2025-12-05 23:13:53,144][HYDRA]
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs
[2025-12-05 23:13:53,144][HYDRA] Launching jobs, sweep output dir :
multirun/01KBRCP0F371673F9920MJYANJ
[2025-12-05 23:13:53,144][HYDRA] #0 : width=400 height=100
[2025-12-05 23:13:53,144][HYDRA] #1 : width=400 height=200
[2025-12-05 23:13:53,144][HYDRA] #2 : width=400 height=300
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0OTdfODY4NGI5ZmE3OTg0NGIzN2JhND
BlNWVkOWQ0N2UwMTlfNmY2MjJhODljZGFkNDBhNzk5MDBhN2Y4YTNlOGVmNDA="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny14OGQ0ZW5fZw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny10aHgwdzk0cw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny04ZzFvdnp2dQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1sc2dybmhiYw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1tamg2ZDU5bQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny13NjVyeWM0ZA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0OTdfODY4NGI5ZmE3OTg0NGIzN2JhND
BlNWVkOWQ0N2UwMTlfZTFjNjQzYjIzZmNiNGY5Yzk1MDM3Mzk4MWExNjg5ODE="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1hZGo2MGRwcQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny0xMGRrN3BvaA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1ncm5peXAydA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
2025/12/05 23:13:54 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:54 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:54 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:54 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:54 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:54 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
[2025-12-05 23:13:54,892][__main__][INFO] - fd82c7c9e9494e2195d775acb6ad2b2a
[2025-12-05 23:13:54,892][__main__][INFO] - {'width': 400, 'height': 300}
2025/12/05 23:13:55 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:55 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:55 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:55 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:55 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:55 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
[2025-12-05 23:13:55,277][__main__][INFO] - ca290690d58c4d37a47b4300b401a5cc
[2025-12-05 23:13:55,277][__main__][INFO] - {'width': 400, 'height': 100}
2025/12/05 23:13:55 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...
2025/12/05 23:13:55 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:55 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:55 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
2025/12/05 23:13:55 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:55 INFO alembic.runtime.migration: Will assume
non-transactional DDL.
[2025-12-05 23:13:55,673][__main__][INFO] - aa6a51f3ddb047c588968f9bbd8b0602
[2025-12-05 23:13:55,673][__main__][INFO] - {'width': 400, 'height': 200}
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0OTdfODY4NGI5ZmE3OTg0NGIzN2JhND
BlNWVkOWQ0N2UwMTlfZTFjNjQzYjIzZmNiNGY5Yzk1MDM3Mzk4MWExNjg5ODE="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1ncm5peXAydA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny0xMGRrN3BvaA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1hZGo2MGRwcQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for
automatic cleanup: unknown resource type
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzI0OTdfODY4NGI5ZmE3OTg0NGIzN2JhND
BlNWVkOWQ0N2UwMTlfNmY2MjJhODljZGFkNDBhNzk5MDBhN2Y4YTNlOGVmNDA="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny13NjVyeWM0ZA=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1tamg2ZDU5bQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny1sc2dybmhiYw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny04ZzFvdnp2dQ=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny10aHgwdzk0cw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
Traceback (most recent call last):
File
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main
raise ValueError(
...<4 lines>...
)
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for
automatic cleanup: unknown resource type ("L2xva3ktMjQ5Ny14OGQ0ZW5fZw=="}).
Resource type should be one of the following: ['noop', 'folder', 'file',
'semlock']
0:00:09 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%
[2025-12-05 23:13:58,830][HYDRA] Launching 2 jobs locally
[2025-12-05 23:13:58,830][HYDRA] #0 : width=250 height=150
2025/12/05 23:13:59 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2025/12/05 23:13:59 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:59 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:59 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2025/12/05 23:13:59 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:59 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2025/12/05 23:13:59 INFO mlflow.tracking.fluent: Experiment with name 'job_submit' does not exist. Creating a new experiment.
[2025-12-05 23:13:59,299][__main__][INFO] - bd264aaa2ea244ea9620223ca5031956
[2025-12-05 23:13:59,299][__main__][INFO] - {'width': 250, 'height': 150}
[2025-12-05 23:13:59,308][HYDRA] #1 : width=250 height=250
[2025-12-05 23:13:59,373][__main__][INFO] - 323c9a96ca4f4b74a079364044d6abb7
[2025-12-05 23:13:59,373][__main__][INFO] - {'width': 250, 'height': 250}
[2025-12-05 23:14:00,797][HYDRA] Launching 2 jobs locally
[2025-12-05 23:14:00,797][HYDRA] #0 : width=350 height=150
2025/12/05 23:14:01 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2025/12/05 23:14:01 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:14:01 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:14:01 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2025/12/05 23:14:01 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:14:01 INFO alembic.runtime.migration: Will assume non-transactional DDL.
[2025-12-05 23:14:01,259][__main__][INFO] - 3e5db94281c24e0e817a992ca982fa14
[2025-12-05 23:14:01,260][__main__][INFO] - {'width': 350, 'height': 150}
[2025-12-05 23:14:01,269][HYDRA] #1 : width=350 height=250
[2025-12-05 23:14:01,336][__main__][INFO] - 18db7b8cb7ce4283a0122c1b9f3175c0
[2025-12-05 23:14:01,336][__main__][INFO] - {'width': 350, 'height': 250}
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=250', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01KBRCPAEDKBPT7MFNMV6H5GFJ']
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=350', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01KBRCPAEDKBPT7MFNMV6H5GFK']
After running these commands, our project structure looks like this:
./
├── mlruns/
│ ├── 1/
│ │ ├── 01b87322a28f4eb0b41e85777f5f8476/
│ │ ├── 53000c83a5d740a68ac7d9e2e4e9e34f/
│ │ ├── d7221d5bbfc5420fa4a8327c7b26da25/
│ │ ├── e82544637cae4c129690e858d286e8cc/
│ │ ├── ec3f96530ad04faba52248499519b831/
│ │ └── efd39079f47f4e4b9f30303ba15b40ed/
│ ├── 2/
│ │ ├── 19aa3d2c1faf4cd9a94afb24cabf8da3/
│ │ ├── 1c9b9ec3f27c427a8a0efff1d768577f/
│ │ ├── aa6a51f3ddb047c588968f9bbd8b0602/
│ │ ├── ca290690d58c4d37a47b4300b401a5cc/
│ │ ├── f0538eb972e84ff5bd27bee658b4194d/
│ │ └── fd82c7c9e9494e2195d775acb6ad2b2a/
│ └── 3/
│ ├── 18db7b8cb7ce4283a0122c1b9f3175c0/
│ ├── 323c9a96ca4f4b74a079364044d6abb7/
│ ├── 3e5db94281c24e0e817a992ca982fa14/
│ └── bd264aaa2ea244ea9620223ca5031956/
├── example.py
├── hydraflow.yaml
├── mlflow.db
└── submit.py
The mlruns directory contains all our experiment data.
Let's explore how to access and analyze this data using HydraFlow's API.
Discovering Runs
Finding Run Directories
HydraFlow provides the iter_run_dirs
function to discover runs in your MLflow tracking directory:
>>> from hydraflow import iter_run_dirs
>>> run_dirs = list(iter_run_dirs())
>>> print(len(run_dirs))
>>> for run_dir in run_dirs[:4]:
... print(run_dir)
16
/home/runner/work/hydraflow/hydraflow/examples/mlruns/3/3e5db94281c24e0e817a992ca982fa14
/home/runner/work/hydraflow/hydraflow/examples/mlruns/3/18db7b8cb7ce4283a0122c1b9f3175c0
/home/runner/work/hydraflow/hydraflow/examples/mlruns/3/bd264aaa2ea244ea9620223ca5031956
/home/runner/work/hydraflow/hydraflow/examples/mlruns/3/323c9a96ca4f4b74a079364044d6abb7
This function finds all run directories in your MLflow tracking directory, making it easy to collect runs for analysis.
Filtering by Experiment Name
You can filter runs by experiment name to focus on specific experiments:
>>> print(len(list(iter_run_dirs("job_sequential"))))
>>> names = ["job_sequential", "job_parallel"]
>>> print(len(list(iter_run_dirs(names))))
>>> print(len(list(iter_run_dirs("job_*"))))
6
12
16
As shown above, you can:
- Filter by a single experiment name
- Provide a list of experiment names
- Use pattern matching with wildcards
Working with Individual Runs
Loading a Run
The Run class represents a single
experiment run in HydraFlow:
>>> from hydraflow import Run
>>> run_dirs = iter_run_dirs()
>>> run_dir = next(run_dirs) # run_dirs is an iterator
>>> run = Run(run_dir)
>>> print(run)
>>> print(type(run))
Run('3e5db94281c24e0e817a992ca982fa14')
<class 'hydraflow.core.run.Run'>
You can also use the load
class method, which accepts both string paths and Path objects:
>>> Run.load(str(run_dir))
>>> print(run)
Run('3e5db94281c24e0e817a992ca982fa14')
Accessing Run Information
Each Run instance provides access to run information and configuration:
>>> print(run.info.run_dir)
>>> print(run.info.run_id)
>>> print(run.info.job_name) # Hydra job name = MLflow experiment name
/home/runner/work/hydraflow/hydraflow/examples/mlruns/3/3e5db94281c24e0e817a992ca982fa14
3e5db94281c24e0e817a992ca982fa14
job_submit
The configuration is available through the cfg attribute:
>>> print(run.cfg)
{'width': 350, 'height': 150}
Type-Safe Configuration Access
For better IDE integration and type checking, you can specify the configuration type:
from dataclasses import dataclass
@dataclass
class Config:
width: int = 1024
height: int = 768
>>> run = Run[Config](run_dir)
>>> print(run)
Run('3e5db94281c24e0e817a992ca982fa14')
When you use Run[Config], your IDE will recognize run.cfg as
having the specified type, enabling autocompletion and type checking.
Accessing Configuration Values
The get method provides a unified interface to access values from a run:
>>> print(run.get("width"))
>>> print(run.get("height"))
350
150
Adding Custom Implementations
Basic Implementation
You can extend runs with custom implementation classes to add domain-specific functionality:
from pathlib import Path
class Impl:
root_dir: Path
def __init__(self, root_dir: Path):
self.root_dir = root_dir
def __repr__(self) -> str:
return f"Impl({self.root_dir.stem!r})"
>>> run = Run[Config, Impl](run_dir, Impl)
>>> print(run)
Run[Impl]('3e5db94281c24e0e817a992ca982fa14')
The implementation is lazily initialized when you first access the impl attribute:
>>> print(run.impl)
>>> print(run.impl.root_dir)
Impl('artifacts')
/home/runner/work/hydraflow/hydraflow/examples/mlruns/3/3e5db94281c24e0e817a992ca982fa14/artifacts
Configuration-Aware Implementation
Implementations can also access the run's configuration:
from dataclasses import dataclass, field
@dataclass
class Size:
root_dir: Path = field(repr=False)
cfg: Config
@property
def size(self) -> int:
return self.cfg.width * self.cfg.height
def is_large(self) -> bool:
return self.size > 100000
>>> run = Run[Config, Size].load(run_dir, Size)
>>> print(run)
>>> print(run.impl)
>>> print(run.impl.size)
Run[Size]('3e5db94281c24e0e817a992ca982fa14')
Size(cfg={'width': 350, 'height': 150})
52500
This allows you to define custom analysis methods that use both the run's artifacts and its configuration.
Working with Multiple Runs
Creating a Run Collection
The RunCollection
class helps you analyze multiple runs:
>>> run_dirs = iter_run_dirs()
>>> rc = Run[Config, Size].load(run_dirs, Size)
>>> print(rc)
RunCollection(Run[Size], n=16)
The load method automatically creates a RunCollection when
given multiple run directories.
Basic Run Collection Operations
You can perform basic operations on a collection:
>>> print(rc.first())
>>> print(rc.last())
Run[Size]('3e5db94281c24e0e817a992ca982fa14')
Run[Size]('53000c83a5d740a68ac7d9e2e4e9e34f')
Filtering Runs
The filter method
lets you select runs based on various criteria:
>>> print(rc.filter(width=400))
RunCollection(Run[Size], n=3)
You can use lists to filter by multiple values (OR logic):
>>> print(rc.filter(height=[100, 300]))
RunCollection(Run[Size], n=8)
Tuples create range filters (inclusive):
>>> print(rc.filter(height=(100, 300)))
RunCollection(Run[Size], n=16)
You can even use custom filter functions:
>>> print(rc.filter(lambda r: r.impl.is_large()))
RunCollection(Run[Size], n=1)
Finding Specific Runs
The get method
returns a single run matching your criteria:
>>> run = rc.get(width=250, height=(100, 200))
>>> print(run)
>>> print(run.impl)
Run[Size]('bd264aaa2ea244ea9620223ca5031956')
Size(cfg={'width': 250, 'height': 150})
Converting to DataFrames
For data analysis, you can convert runs to a Polars DataFrame:
>>> print(rc.to_frame("width", "height", "size"))
shape: (16, 3)
┌───────┬────────┬────────┐
│ width ┆ height ┆ size │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═══════╪════════╪════════╡
│ 350 ┆ 150 ┆ 52500 │
│ 350 ┆ 250 ┆ 87500 │
│ 250 ┆ 150 ┆ 37500 │
│ 250 ┆ 250 ┆ 62500 │
│ 400 ┆ 300 ┆ 120000 │
│ … ┆ … ┆ … │
│ 100 ┆ 300 ┆ 30000 │
│ 300 ┆ 300 ┆ 90000 │
│ 100 ┆ 100 ┆ 10000 │
│ 300 ┆ 200 ┆ 60000 │
│ 300 ┆ 100 ┆ 30000 │
└───────┴────────┴────────┘
You can add custom columns using callables:
>>> print(rc.to_frame("width", "height", is_large=lambda r: r.impl.is_large()))
shape: (16, 3)
┌───────┬────────┬──────────┐
│ width ┆ height ┆ is_large │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ bool │
╞═══════╪════════╪══════════╡
│ 350 ┆ 150 ┆ false │
│ 350 ┆ 250 ┆ false │
│ 250 ┆ 150 ┆ false │
│ 250 ┆ 250 ┆ false │
│ 400 ┆ 300 ┆ true │
│ … ┆ … ┆ … │
│ 100 ┆ 300 ┆ false │
│ 300 ┆ 300 ┆ false │
│ 100 ┆ 100 ┆ false │
│ 300 ┆ 200 ┆ false │
│ 300 ┆ 100 ┆ false │
└───────┴────────┴──────────┘
Functions can return lists for multiple values:
>>> def to_list(run: Run) -> list[int]:
... return [2 * run.get("width"), 3 * run.get("height")]
>>> print(rc.to_frame("width", from_list=to_list))
shape: (16, 2)
┌───────┬────────────┐
│ width ┆ from_list │
│ --- ┆ --- │
│ i64 ┆ list[i64] │
╞═══════╪════════════╡
│ 350 ┆ [700, 450] │
│ 350 ┆ [700, 750] │
│ 250 ┆ [500, 450] │
│ 250 ┆ [500, 750] │
│ 400 ┆ [800, 900] │
│ … ┆ … │
│ 100 ┆ [200, 900] │
│ 300 ┆ [600, 900] │
│ 100 ┆ [200, 300] │
│ 300 ┆ [600, 600] │
│ 300 ┆ [600, 300] │
└───────┴────────────┘
Or dictionaries for multiple named columns:
>>> def to_dict(run: Run) -> dict[int, str]:
... width2 = 2 * run.get("width")
... name = f"h{run.get('height')}"
... return {"width2": width2, "name": name}
>>> print(rc.to_frame("width", from_dict=to_dict))
shape: (16, 2)
┌───────┬──────────────┐
│ width ┆ from_dict │
│ --- ┆ --- │
│ i64 ┆ struct[2] │
╞═══════╪══════════════╡
│ 350 ┆ {700,"h150"} │
│ 350 ┆ {700,"h250"} │
│ 250 ┆ {500,"h150"} │
│ 250 ┆ {500,"h250"} │
│ 400 ┆ {800,"h300"} │
│ … ┆ … │
│ 100 ┆ {200,"h300"} │
│ 300 ┆ {600,"h300"} │
│ 100 ┆ {200,"h100"} │
│ 300 ┆ {600,"h200"} │
│ 300 ┆ {600,"h100"} │
└───────┴──────────────┘
Grouping Runs
The group_by
method organizes runs by common attributes:
>>> grouped = rc.group_by("width")
>>> for key, group in grouped.items():
... print(key, group)
350 RunCollection(Run[Size], n=2)
250 RunCollection(Run[Size], n=2)
400 RunCollection(Run[Size], n=3)
200 RunCollection(Run[Size], n=3)
100 RunCollection(Run[Size], n=3)
300 RunCollection(Run[Size], n=3)
You can group by multiple keys:
>>> grouped = rc.group_by("width", "height")
>>> for key, group in grouped.items():
... print(key, group)
(350, 150) RunCollection(Run[Size], n=1)
(350, 250) RunCollection(Run[Size], n=1)
(250, 150) RunCollection(Run[Size], n=1)
(250, 250) RunCollection(Run[Size], n=1)
(400, 300) RunCollection(Run[Size], n=1)
(200, 200) RunCollection(Run[Size], n=1)
(400, 200) RunCollection(Run[Size], n=1)
(400, 100) RunCollection(Run[Size], n=1)
(200, 100) RunCollection(Run[Size], n=1)
(200, 300) RunCollection(Run[Size], n=1)
(100, 200) RunCollection(Run[Size], n=1)
(100, 300) RunCollection(Run[Size], n=1)
(300, 300) RunCollection(Run[Size], n=1)
(100, 100) RunCollection(Run[Size], n=1)
(300, 200) RunCollection(Run[Size], n=1)
(300, 100) RunCollection(Run[Size], n=1)
Adding aggregation functions using the
agg
method transforms the result into a DataFrame:
>>> grouped = rc.group_by("width")
>>> df = grouped.agg(n=lambda runs: len(runs))
>>> print(df)
shape: (6, 2)
┌───────┬─────┐
│ width ┆ n │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═══════╪═════╡
│ 350 ┆ 2 │
│ 250 ┆ 2 │
│ 400 ┆ 3 │
│ 200 ┆ 3 │
│ 100 ┆ 3 │
│ 300 ┆ 3 │
└───────┴─────┘
Summary
In this tutorial, you've learned how to:
- Discover experiment runs in your MLflow tracking directory
- Load and access information from individual runs
- Add custom implementation classes for domain-specific analysis
- Filter, group, and analyze collections of runs
- Convert run data to DataFrames for advanced analysis
These capabilities enable you to efficiently analyze your experiments and extract valuable insights from your machine learning workflows.
Next Steps
Now that you understand HydraFlow's analysis capabilities, you can:
- Dive deeper into the Run Class and Run Collection documentation
- Explore advanced analysis techniques in the Analyzing Results section
- Apply these analysis techniques to your own machine learning experiments