Skip to content

Automating Complex Workflows

This tutorial demonstrates how to use HydraFlow's workflow automation capabilities to define, manage, and execute complex experiment workflows.

Prerequisites

Before you begin this tutorial, you should:

  1. Understand basic HydraFlow applications (from the Creating Your First Application tutorial)
  2. Have a basic understanding of YAML configuration files

Project Structure

First, let's examine our project structure:

./
├── example.py
├── hydraflow.yaml
└── submit.py

In this tutorial, we'll use:

  • example.py: Our basic HydraFlow application
  • hydraflow.yaml: A configuration file to define our experiment workflows
  • submit.py: A helper script for job submission

Understanding Job Definitions

The hydraflow.yaml file allows you to define reusable experiment workflows:

hydraflow.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
jobs:
  job_sequential:
    run: python example.py
    sets:
      - each: width=100,300
        all: height=100:300:100
  job_parallel:
    run: python example.py
    add: >-
      hydra/launcher=joblib
      hydra.launcher.n_jobs=3
    sets:
      - each: width=200,400
        all: height=100:300:100
  job_submit:
    submit: python submit.py example.py
    sets:
      - each: width=250:350:100
        all: height=150,250

This configuration file defines three different types of jobs:

  1. job_sequential: A job that runs sequentially
  2. job_parallel: A job that runs with parallelization
  3. job_submit: A job that uses a submit command for custom execution

Each job demonstrates different execution patterns and parameter combinations.

Using the HydraFlow CLI

HydraFlow provides a command-line interface (CLI) for executing and managing jobs defined in your hydraflow.yaml file. The primary command is hydraflow run, which allows you to execute any job defined in your configuration.

Basic usage: 

hydraflow run <job_name> [overrides]

Where:

  • <job_name> is the name of a job defined in hydraflow.yaml
  • [overrides] are optional Hydra-style parameter overrides

For more details on the CLI, see the Job Configuration documentation.

Previewing Execution with Dry Run

Before executing our workflows, we can preview what will happen using the --dry-run flag:

$ hydraflow run job_sequential --dry-run
python example.py --multirun width=100 height=100,200,300 hydra.job.name=job_sequential hydra.sweep.dir=multirun/01KBRCN10JX78CA8V9JRCRDQJB
python example.py --multirun width=300 height=100,200,300 hydra.job.name=job_sequential hydra.sweep.dir=multirun/01KBRCN10JX78CA8V9JRCRDQJC

From the dry run output, we can observe:

  • 2 jobs will be executed (from the each parameter combinations)
  • Each job contains 3 sweeps (from the all range values)
  • Each job includes additional options:
    • hydra.job.name: The name of the job defined in hydraflow.yaml
    • hydra.sweep.dir: A unique but time-ordered directory for each job created by HydraFlow

Standard Hydra creates directories based on the current date and time, which may cause duplication during parallel execution. HydraFlow solves this problem by creating unique, time-ordered directories for each job.

Running Sequential Jobs

Let's examine the sequential job configuration:

job_sequential:
  run: python example.py
  sets:
    - each: width=100,300
      all: height=100:300:100

This job uses the each and all parameters to run multiple configuration combinations in sequence:

$ hydraflow run job_sequential
[2025-12-05 23:13:17,622][HYDRA] Launching 3 jobs locally                       
[2025-12-05 23:13:17,622][HYDRA]        #0 : width=100 height=100               
2025/12/05 23:13:18 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:18 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:18 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:18 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade  ->         
451aebb31d03, add metric step                                                   
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 451aebb31d03
-> 90e64c465722, migrate user column to tags                                    
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 90e64c465722
-> 181f10493468, allow nulls for metric values                                  
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 181f10493468
-> df50e92ffc5e, Add Experiment Tags Table                                      
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade df50e92ffc5e
-> 7ac759974ad8, Update run tags with larger limit                              
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 7ac759974ad8
-> 89d4b8295536, create latest metrics table                                    
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 89d4b8295536
-> 2b4d017a5e9b, add model registry tables to db                                
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 2b4d017a5e9b
-> cfd24bdc0731, Update run status constraint with killed                       
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade cfd24bdc0731
-> 0a8213491aaa, drop_duplicate_killed_constraint                               
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 0a8213491aaa
-> 728d730b5ebd, add registered model tags table                                
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 728d730b5ebd
-> 27a6a02d2cf1, add model version tags table                                   
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 27a6a02d2cf1
-> 84291f40a231, add run_link to model_version                                  
2025/12/05 23:13:18 INFO alembic.runtime.migration: Running upgrade 84291f40a231
-> a8c4a736bde6, allow nulls for run_id                                         
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade a8c4a736bde6
-> 39d1c3be5f05, add_is_nan_constraint_for_metrics_tables_if_necessary          
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 39d1c3be5f05
-> c48cb773bb87, reset_default_value_for_is_nan_in_metrics_table_for_mysql      
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade c48cb773bb87
-> bd07f7e963c5, create index on run_uuid                                       
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade bd07f7e963c5
-> 0c779009ac13, add deleted_time field to runs table                           
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 0c779009ac13
-> cc1f77228345, change param value length to 500                               
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade cc1f77228345
-> 97727af70f4d, Add creation_time and last_update_time to experiments table    
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 97727af70f4d
-> 3500859a5d39, Add Model Aliases table                                        
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 3500859a5d39
-> 7f2a7d5fae7d, add datasets inputs input_tags tables                          
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 7f2a7d5fae7d
-> 2d6e25af4d3e, increase max param val length from 500 to 8000                 
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 2d6e25af4d3e
-> acf3f17fdcc7, add storage location field to model versions                   
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade acf3f17fdcc7
-> 867495a8f9d4, add trace tables                                               
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 867495a8f9d4
-> 5b0e9adcef9c, add cascade deletion to trace tables foreign keys              
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 5b0e9adcef9c
-> 4465047574b1, increase max dataset schema size                               
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 4465047574b1
-> f5a4f2784254, increase run tag value limit to 8000                           
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade f5a4f2784254
-> 0584bdc529eb, add cascading deletion to datasets from experiments            
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 0584bdc529eb
-> 400f98739977, add logged model tables                                        
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 400f98739977
-> 6953534de441, add step to inputs table                                       
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 6953534de441
-> bda7b8c39065, increase_model_version_tag_value_limit                         
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade bda7b8c39065
-> cbc13b556ace, add V3 trace schema columns                                    
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade cbc13b556ace
-> 770bee3ae1dd, add assessments table                                          
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 770bee3ae1dd
-> a1b2c3d4e5f6, add spans table                                                
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade a1b2c3d4e5f6
-> de4033877273, create entity_associations table                               
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade de4033877273
-> 1a0cddfcaa16, Add webhooks and webhook_events tables                         
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 1a0cddfcaa16
-> 534353b11cbc, add scorer tables                                              
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 534353b11cbc
-> 71994744cf8e, add evaluation datasets                                        
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 71994744cf8e
-> 3da73c924c2f, add outputs to dataset record                                  
2025/12/05 23:13:19 INFO alembic.runtime.migration: Running upgrade 3da73c924c2f
-> bf29a5ff90ea, add jobs table                                                 
2025/12/05 23:13:19 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:19 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:19 INFO mlflow.tracking.fluent: Experiment with name           
'job_sequential' does not exist. Creating a new experiment.                     
[2025-12-05 23:13:19,466][__main__][INFO] - 709e238ca3034815b543e1bce97dd2ef    
[2025-12-05 23:13:19,466][__main__][INFO] - {'width': 100, 'height': 100}       
[2025-12-05 23:13:19,477][HYDRA]        #1 : width=100 height=200               
[2025-12-05 23:13:19,542][__main__][INFO] - 8630b3a939b948d3b5111df1f07dbba3    
[2025-12-05 23:13:19,542][__main__][INFO] - {'width': 100, 'height': 200}       
[2025-12-05 23:13:19,549][HYDRA]        #2 : width=100 height=300               
[2025-12-05 23:13:19,615][__main__][INFO] - 2284373deab6447fb1f18a28f73ba4ea    
[2025-12-05 23:13:19,615][__main__][INFO] - {'width': 100, 'height': 300}       
[2025-12-05 23:13:21,165][HYDRA] Launching 3 jobs locally                       
[2025-12-05 23:13:21,165][HYDRA]        #0 : width=300 height=100               
2025/12/05 23:13:21 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:21 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:21 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:21 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:21 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:21 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
[2025-12-05 23:13:21,665][__main__][INFO] - 7b3273fcfa0e431690ebf79762cc617b    
[2025-12-05 23:13:21,665][__main__][INFO] - {'width': 300, 'height': 100}       
[2025-12-05 23:13:21,699][HYDRA]        #1 : width=300 height=200               
[2025-12-05 23:13:21,803][__main__][INFO] - 20791a7772c04b2cb202b34666581e30    
[2025-12-05 23:13:21,803][__main__][INFO] - {'width': 300, 'height': 200}       
[2025-12-05 23:13:21,845][HYDRA]        #2 : width=300 height=300               
[2025-12-05 23:13:22,010][__main__][INFO] - f388046f15054c4e9fb4870b6fc6bdfd    
[2025-12-05 23:13:22,010][__main__][INFO] - {'width': 300, 'height': 300}       
  0:00:05 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%

Results of execution:

  • An experiment named job_sequential is created
  • 2×3=6 jobs are executed sequentially
  • A progress bar is displayed to track completion

Running Parallel Jobs

Now let's look at our parallel job configuration:

job_parallel:
  run: python example.py
  add: >-
    hydra/launcher=joblib
    hydra.launcher.n_jobs=3
  sets:
    - each: width=200,400
      all: height=100:300:100

This job leverages Hydra's parallel execution features using a joblib launcher via add parameter:

$ hydraflow run job_parallel --dry-run
python example.py --multirun width=200 height=100,200,300 hydra.job.name=job_parallel hydra.sweep.dir=multirun/01KBRCN992WDBDAZP5MQR6J5P9 hydra/launcher=joblib hydra.launcher.n_jobs=3
python example.py --multirun width=400 height=100,200,300 hydra.job.name=job_parallel hydra.sweep.dir=multirun/01KBRCN992WDBDAZP5MQR6J5PA hydra/launcher=joblib hydra.launcher.n_jobs=3

$ hydraflow run job_parallel
[2025-12-05 23:13:26,084][HYDRA]                                                
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs                                              
[2025-12-05 23:13:26,084][HYDRA] Launching jobs, sweep output dir :             
multirun/01KBRCNAEG3403GK8MD2VTCKWS                                             
[2025-12-05 23:13:26,085][HYDRA]        #0 : width=200 height=100               
[2025-12-05 23:13:26,085][HYDRA]        #1 : width=200 height=200               
[2025-12-05 23:13:26,085][HYDRA]        #2 : width=200 height=300               
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for       
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIyODdfZThkOGY4MTQ3NWViNDUyMThmNj
cyYWFhNTFlNTM4OTFfMzE0M2RhNDBlMGNjNDJiZWE1NThlNTg4YjQxODBlNDU="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny0yOXRoN3cxeQ=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny0wYnd6YnFxNA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny11X2F4eGxndg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny00dmx1djBmcA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny10OWJ3dml0dA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny05aXQyMW1rbA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for       
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIyODdfZThkOGY4MTQ3NWViNDUyMThmNj
cyYWFhNTFlNTM4OTFfZjJkNmVmN2UzNmE5NDE5NjlkODVlZDZmYWEwZmE1Njk="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny1mY3Q0aTg5aA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny1fYmRjc2R4Yw=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny1iNDQxMGhqcA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
2025/12/05 23:13:27 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:27 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:27 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:27 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:27 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:27 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:27 INFO mlflow.tracking.fluent: Experiment with name           
'job_parallel' does not exist. Creating a new experiment.                       
[2025-12-05 23:13:27,784][__main__][INFO] - a5063d7de629460797250abcf8ba6f8a    
[2025-12-05 23:13:27,784][__main__][INFO] - {'width': 200, 'height': 200}       
2025/12/05 23:13:28 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:28 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:28 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:28 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:28 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:28 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
[2025-12-05 23:13:28,241][__main__][INFO] - de2b28980f824a46b93e35ddf83c9d01    
[2025-12-05 23:13:28,241][__main__][INFO] - {'width': 200, 'height': 100}       
2025/12/05 23:13:28 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:28 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:28 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:28 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:28 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:28 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
[2025-12-05 23:13:28,644][__main__][INFO] - c1e23b4157194861bf020f395418c1c9    
[2025-12-05 23:13:28,644][__main__][INFO] - {'width': 200, 'height': 300}       
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for     
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIyODdfZThkOGY4MTQ3NWViNDUyMThmNj
cyYWFhNTFlNTM4OTFfZjJkNmVmN2UzNmE5NDE5NjlkODVlZDZmYWEwZmE1Njk="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny1iNDQxMGhqcA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny1fYmRjc2R4Yw=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny1mY3Q0aTg5aA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for     
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIyODdfZThkOGY4MTQ3NWViNDUyMThmNj
cyYWFhNTFlNTM4OTFfMzE0M2RhNDBlMGNjNDJiZWE1NThlNTg4YjQxODBlNDU="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny05aXQyMW1rbA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny10OWJ3dml0dA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny00dmx1djBmcA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny11X2F4eGxndg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny0wYnd6YnFxNA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjI4Ny0yOXRoN3cxeQ=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
[2025-12-05 23:13:30,644][HYDRA]                                                
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs                                              
[2025-12-05 23:13:30,644][HYDRA] Launching jobs, sweep output dir :             
multirun/01KBRCNAEG3403GK8MD2VTCKWT                                             
[2025-12-05 23:13:30,644][HYDRA]        #0 : width=400 height=100               
[2025-12-05 23:13:30,644][HYDRA]        #1 : width=400 height=200               
[2025-12-05 23:13:30,644][HYDRA]        #2 : width=400 height=300               
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for       
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIzMjdfYWQyZjZiMjhhNDgyNDg3OTg3Zm
RkOTg0OTRiMDZjODlfODlhMDk2MTQ0ZWNhNGNlNmI5YzdmNzJjYTlkZGVlZTU="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy14aHN6dW5raA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0zY2p1Z2Q2dA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy1ybzg5X25nZg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy1fbmh0YWtyag=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy14a2R6djg3cA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0ya3pxaHU1aQ=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"folder","base64_name" for       
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIzMjdfYWQyZjZiMjhhNDgyNDg3OTg3Zm
RkOTg0OTRiMDZjODlfNzI5MWM5NjVmNzEzNDExNmEyM2E1MTJiZGYxZjlmOTk="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0zcTN5dXdsMw=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy02bXFhNm84eg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "REGISTER","rtype":"semlock","base64_name" for      
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0yNHBiejlveg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
2025/12/05 23:13:32 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:32 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:32 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:32 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:32 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:32 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
[2025-12-05 23:13:32,363][__main__][INFO] - 15dde034c21b4ebc9ff1b6cd22af7f11    
[2025-12-05 23:13:32,363][__main__][INFO] - {'width': 400, 'height': 300}       
2025/12/05 23:13:32 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:32 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:32 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:32 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:32 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:32 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
[2025-12-05 23:13:32,739][__main__][INFO] - c287a6122e384cb38d52a7456866ab12    
[2025-12-05 23:13:32,739][__main__][INFO] - {'width': 400, 'height': 200}       
2025/12/05 23:13:33 INFO mlflow.store.db.utils: Creating initial MLflow database
tables...                                                                       
2025/12/05 23:13:33 INFO mlflow.store.db.utils: Updating database tables        
2025/12/05 23:13:33 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:33 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
2025/12/05 23:13:33 INFO alembic.runtime.migration: Context impl SQLiteImpl.    
2025/12/05 23:13:33 INFO alembic.runtime.migration: Will assume                 
non-transactional DDL.                                                          
[2025-12-05 23:13:33,159][__main__][INFO] - 112ac058f97347d9a59d4a68819945c5    
[2025-12-05 23:13:33,159][__main__][INFO] - {'width': 400, 'height': 100}       
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for     
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIzMjdfYWQyZjZiMjhhNDgyNDg3OTg3Zm
RkOTg0OTRiMDZjODlfNzI5MWM5NjVmNzEzNDExNmEyM2E1MTJiZGYxZjlmOTk="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0yNHBiejlveg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy02bXFhNm84eg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0zcTN5dXdsMw=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"folder","base64_name" for     
automatic cleanup: unknown resource type                                        
("L2Rldi9zaG0vam9ibGliX21lbW1hcHBpbmdfZm9sZGVyXzIzMjdfYWQyZjZiMjhhNDgyNDg3OTg3Zm
RkOTg0OTRiMDZjODlfODlhMDk2MTQ0ZWNhNGNlNmI5YzdmNzJjYTlkZGVlZTU="}). Resource type
should be one of the following: ['noop', 'folder', 'file', 'semlock']           
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0ya3pxaHU1aQ=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy14a2R6djg3cA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy1fbmh0YWtyag=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy1ybzg5X25nZg=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy0zY2p1Z2Q2dA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
Traceback (most recent call last):                                              
  File                                                                          
"/home/runner/work/hydraflow/hydraflow/.venv/lib/python3.13/site-packages/joblib
/externals/loky/backend/resource_tracker.py", line 297, in main                 
    raise ValueError(                                                           
    ...<4 lines>...                                                             
    )                                                                           
ValueError: Cannot register "UNREGISTER","rtype":"semlock","base64_name" for    
automatic cleanup: unknown resource type ("L2xva3ktMjMyNy14aHN6dW5raA=="}).     
Resource type should be one of the following: ['noop', 'folder', 'file',        
'semlock']                                                                      
  0:00:09 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%

Results of execution:

  • An experiment named job_parallel is created
  • The same Python script is used but with a different experiment name
  • 2 Python commands are executed sequentially
  • Each Python command runs 3 jobs in parallel (using the hydra/launcher=joblib configuration)

This demonstrates how HydraFlow makes Hydra's powerful parallel execution features easily accessible.

Using the Submit Command

For more complex execution patterns, HydraFlow provides the submit command. Here's our submit job configuration:

job_submit:
  submit: python submit.py example.py
  sets:
    - each: width=250:350:100
      all: height=150,250

The submit command requires two key components:

  1. Your HydraFlow application (example.py in this case)
  2. A command or script that will receive and process a parameter file

Here's our implementation of the submit handler:

submit.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from __future__ import annotations

import shlex
import subprocess
import sys
from pathlib import Path


def main() -> None:
    app_file, opt_file = sys.argv[1:]
    text = Path(opt_file).read_text()

    for line in text.splitlines():
        opts = shlex.split(line)
        args = [sys.executable, app_file, *opts]
        print(args)
        subprocess.run(args, check=True)


if __name__ == "__main__":
    main()

How the submit command works:

  1. HydraFlow generates all parameter combinations based on your job configuration
  2. It writes these combinations to a temporary text file (one combination per line)
  3. It runs the command specified in the submit field of your hydraflow.yaml
  4. It appends the temporary file path as the last argument to your command

For example, with submit: python submit.py example.py in your configuration, the actual executed command will be something like: 

python submit.py example.py /tmp/hydraflow_parameters_12345.txt

Let's see it in action with a dry run:

$ hydraflow run job_submit --dry-run
python submit.py example.py /home/runner/work/hydraflow/hydraflow/examples/tmp5yxwauxc
--multirun width=250 height=150,250 hydra.job.name=job_submit hydra.sweep.dir=multirun/01KBRCNMD4VK2Z4AKWVEYHQGQM
--multirun width=350 height=150,250 hydra.job.name=job_submit hydra.sweep.dir=multirun/01KBRCNMD4VK2Z4AKWVEYHQGQN

And now let's run it:

$ hydraflow run job_submit
[2025-12-05 23:13:37,420][HYDRA] Launching 2 jobs locally
[2025-12-05 23:13:37,421][HYDRA]    #0 : width=250 height=150
2025/12/05 23:13:37 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2025/12/05 23:13:37 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:37 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:37 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2025/12/05 23:13:37 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:37 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2025/12/05 23:13:37 INFO mlflow.tracking.fluent: Experiment with name 'job_submit' does not exist. Creating a new experiment.
[2025-12-05 23:13:37,894][__main__][INFO] - 6fd0adbbbcba42e9a8f4250103350434
[2025-12-05 23:13:37,894][__main__][INFO] - {'width': 250, 'height': 150}
[2025-12-05 23:13:37,903][HYDRA]    #1 : width=250 height=250
[2025-12-05 23:13:37,968][__main__][INFO] - 855fd7bfa2ca4431b799cdd8a2611c2e
[2025-12-05 23:13:37,969][__main__][INFO] - {'width': 250, 'height': 250}
[2025-12-05 23:13:39,417][HYDRA] Launching 2 jobs locally
[2025-12-05 23:13:39,417][HYDRA]    #0 : width=350 height=150
2025/12/05 23:13:39 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2025/12/05 23:13:39 INFO mlflow.store.db.utils: Updating database tables
2025/12/05 23:13:39 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:39 INFO alembic.runtime.migration: Will assume non-transactional DDL.
2025/12/05 23:13:39 INFO alembic.runtime.migration: Context impl SQLiteImpl.
2025/12/05 23:13:39 INFO alembic.runtime.migration: Will assume non-transactional DDL.
[2025-12-05 23:13:39,879][__main__][INFO] - 223c396d26144583b1d6f8fa4f4594e2
[2025-12-05 23:13:39,879][__main__][INFO] - {'width': 350, 'height': 150}
[2025-12-05 23:13:39,888][HYDRA]    #1 : width=350 height=250
[2025-12-05 23:13:39,954][__main__][INFO] - 896a97798d314aeda54cc52c84f87a5c
[2025-12-05 23:13:39,954][__main__][INFO] - {'width': 350, 'height': 250}
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=250', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01KBRCNNH60P34X356HPJKWBVJ']
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=350', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01KBRCNNH60P34X356HPJKWBVK']

Our submit.py script implements a simple processor that:

  1. Accepts two arguments: the application file (example.py) and the parameter file
  2. Reads each line from the parameter file
  3. Runs the application with each set of parameters sequentially

In real-world scenarios, you could customize this handler to:

  • Submit jobs to compute clusters (SLURM, PBS, etc.)
  • Implement custom scheduling logic
  • Distribute workloads based on resource requirements

Reviewing Results

With HydraFlow, all important data is stored in MLflow, so we can safely delete the Hydra output directories:

$ rm -rf multirun

Let's check the directory structure:

./
├── mlruns/
│   ├── 1/
│   │   ├── 20791a7772c04b2cb202b34666581e30/
│   │   ├── 2284373deab6447fb1f18a28f73ba4ea/
│   │   ├── 709e238ca3034815b543e1bce97dd2ef/
│   │   ├── 7b3273fcfa0e431690ebf79762cc617b/
│   │   ├── 8630b3a939b948d3b5111df1f07dbba3/
│   │   └── f388046f15054c4e9fb4870b6fc6bdfd/
│   ├── 2/
│   │   ├── 112ac058f97347d9a59d4a68819945c5/
│   │   ├── 15dde034c21b4ebc9ff1b6cd22af7f11/
│   │   ├── a5063d7de629460797250abcf8ba6f8a/
│   │   ├── c1e23b4157194861bf020f395418c1c9/
│   │   ├── c287a6122e384cb38d52a7456866ab12/
│   │   └── de2b28980f824a46b93e35ddf83c9d01/
│   └── 3/
│       ├── 223c396d26144583b1d6f8fa4f4594e2/
│       ├── 6fd0adbbbcba42e9a8f4250103350434/
│       ├── 855fd7bfa2ca4431b799cdd8a2611c2e/
│       └── 896a97798d314aeda54cc52c84f87a5c/
├── example.py
├── hydraflow.yaml
├── mlflow.db
└── submit.py

After cleanup, we can observe:

  • There are three experiments (one for each job type)
  • Each experiment contains multiple runs
  • A total of 16 runs were executed across all jobs

Summary

In this tutorial, you've learned how to:

  1. Define different types of experiment workflows in a hydraflow.yaml file
  2. Execute sequential and parallel job runs
  3. Use the submit command for custom execution patterns
  4. Preview jobs with dry runs
  5. Manage and organize experiment outputs

These workflow automation capabilities allow you to efficiently manage complex experiment configurations, making your machine learning research more organized and reproducible.

Next Steps

Now that you've learned about workflow automation, try:

  • Defining your own custom workflows
  • Exploring more complex parameter sweep combinations
  • Learning how to Analyze Results from your experiments

For more detailed information, refer to: