Skip to content

Automating Complex Workflows

This tutorial demonstrates how to use HydraFlow's workflow automation capabilities to define, manage, and execute complex experiment workflows.

Prerequisites

Before you begin this tutorial, you should:

  1. Understand basic HydraFlow applications (from the Creating Your First Application tutorial)
  2. Have a basic understanding of YAML configuration files

Project Structure

First, let's examine our project structure:

.
├── example.py
├── hydraflow.yaml
└── submit.py

In this tutorial, we'll use:

  • example.py: Our basic HydraFlow application
  • hydraflow.yaml: A configuration file to define our experiment workflows
  • submit.py: A helper script for job submission

Understanding Job Definitions

The hydraflow.yaml file allows you to define reusable experiment workflows:

hydraflow.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
jobs:
  job_sequential:
    run: python example.py
    sets:
      - each: width=100,300
        all: height=100:300:100
  job_parallel:
    run: python example.py
    add: >-
      hydra/launcher=joblib
      hydra.launcher.n_jobs=3
    sets:
      - each: width=200,400
        all: height=100:300:100
  job_submit:
    submit: python submit.py example.py
    sets:
      - each: width=250:350:100
        all: height=150,250

This configuration file defines three different types of jobs:

  1. job_sequential: A job that runs sequentially
  2. job_parallel: A job that runs with parallelization
  3. job_submit: A job that uses a submit command for custom execution

Each job demonstrates different execution patterns and parameter combinations.

Using the HydraFlow CLI

HydraFlow provides a command-line interface (CLI) for executing and managing jobs defined in your hydraflow.yaml file. The primary command is hydraflow run, which allows you to execute any job defined in your configuration.

Basic usage: 

hydraflow run <job_name> [overrides]

Where:

  • <job_name> is the name of a job defined in hydraflow.yaml
  • [overrides] are optional Hydra-style parameter overrides

For more details on the CLI, see the Job Configuration documentation.

Previewing Execution with Dry Run

Before executing our workflows, we can preview what will happen using the --dry-run flag:

$ hydraflow run job_sequential --dry-run
python example.py --multirun width=100 height=100,200,300 hydra.job.name=job_sequential hydra.sweep.dir=multirun/01K3CSMCKWY9M38F9BB8TMQRX4
python example.py --multirun width=300 height=100,200,300 hydra.job.name=job_sequential hydra.sweep.dir=multirun/01K3CSMCKWY9M38F9BB8TMQRX5

From the dry run output, we can observe:

  • 2 jobs will be executed (from the each parameter combinations)
  • Each job contains 3 sweeps (from the all range values)
  • Each job includes additional options:
    • hydra.job.name: The name of the job defined in hydraflow.yaml
    • hydra.sweep.dir: A unique but time-ordered directory for each job created by HydraFlow

Standard Hydra creates directories based on the current date and time, which may cause duplication during parallel execution. HydraFlow solves this problem by creating unique, time-ordered directories for each job.

Running Sequential Jobs

Let's examine the sequential job configuration:

job_sequential:
  run: python example.py
  sets:
    - each: width=100,300
      all: height=100:300:100

This job uses the each and all parameters to run multiple configuration combinations in sequence:

$ hydraflow run job_sequential
2025/08/24 01:03:39 INFO mlflow.tracking.fluent: Experiment with name 'job_sequential' does not exist. Creating a new experiment.
[2025-08-24 01:03:40,911][HYDRA] Launching 3 jobs locally                       
[2025-08-24 01:03:40,911][HYDRA]        #0 : width=100 height=100               
[2025-08-24 01:03:41,206][__main__][INFO] - 60fa8395d5b14ddf9bded3be44847fe0    
[2025-08-24 01:03:41,206][__main__][INFO] - {'width': 100, 'height': 100}       
[2025-08-24 01:03:41,365][HYDRA]        #1 : width=100 height=200               
[2025-08-24 01:03:41,453][__main__][INFO] - 2053bd6ddcab4465963146967b396c33    
[2025-08-24 01:03:41,453][__main__][INFO] - {'width': 100, 'height': 200}       
[2025-08-24 01:03:41,456][HYDRA]        #2 : width=100 height=300               
[2025-08-24 01:03:41,539][__main__][INFO] - 8ac4dd4406f045378b0ac592d987b7c3    
[2025-08-24 01:03:41,539][__main__][INFO] - {'width': 100, 'height': 300}       
[2025-08-24 01:03:43,237][HYDRA] Launching 3 jobs locally                       
[2025-08-24 01:03:43,237][HYDRA]        #0 : width=300 height=100               
[2025-08-24 01:03:43,364][__main__][INFO] - 08ec0b26237c4b32ad30550d1f552802    
[2025-08-24 01:03:43,364][__main__][INFO] - {'width': 300, 'height': 100}       
[2025-08-24 01:03:43,367][HYDRA]        #1 : width=300 height=200               
[2025-08-24 01:03:43,458][__main__][INFO] - 731ca75db9bb47bdae68c7a5f08c9d4a    
[2025-08-24 01:03:43,458][__main__][INFO] - {'width': 300, 'height': 200}       
[2025-08-24 01:03:43,461][HYDRA]        #2 : width=300 height=300               
[2025-08-24 01:03:43,550][__main__][INFO] - b61455bf287442448a172e3f95eee283    
[2025-08-24 01:03:43,550][__main__][INFO] - {'width': 300, 'height': 300}       
  0:00:04 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%

Results of execution:

  • An experiment named job_sequential is created
  • 2×3=6 jobs are executed sequentially
  • A progress bar is displayed to track completion

Running Parallel Jobs

Now let's look at our parallel job configuration:

job_parallel:
  run: python example.py
  add: >-
    hydra/launcher=joblib
    hydra.launcher.n_jobs=3
  sets:
    - each: width=200,400
      all: height=100:300:100

This job leverages Hydra's parallel execution features using a joblib launcher via add parameter:

$ hydraflow run job_parallel --dry-run
python example.py --multirun width=200 height=100,200,300 hydra.job.name=job_parallel hydra.sweep.dir=multirun/01K3CSMJWBBSNQ45BS4YYXE5Q0 hydra/launcher=joblib hydra.launcher.n_jobs=3
python example.py --multirun width=400 height=100,200,300 hydra.job.name=job_parallel hydra.sweep.dir=multirun/01K3CSMJWBBSNQ45BS4YYXE5Q1 hydra/launcher=joblib hydra.launcher.n_jobs=3

$ hydraflow run job_parallel
2025/08/24 01:03:45 INFO mlflow.tracking.fluent: Experiment with name 'job_parallel' does not exist. Creating a new experiment.
[2025-08-24 01:03:47,311][HYDRA]                                                
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs                                              
[2025-08-24 01:03:47,311][HYDRA] Launching jobs, sweep output dir :             
multirun/01K3CSMM611DKJJM59Y4VZNXPT                                             
[2025-08-24 01:03:47,311][HYDRA]        #0 : width=200 height=100               
[2025-08-24 01:03:47,311][HYDRA]        #1 : width=200 height=200               
[2025-08-24 01:03:47,311][HYDRA]        #2 : width=200 height=300               
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2247_2d1cf93582614429b70b6af8a0d0a984_159a4b71
7f64437a85e0c06613ff82a1 for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-9er72j1s for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-pnp2v73w for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-o06miog5 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-v7zuqj8z for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-i2ud2v0j for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-ab03j21l for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2247_2d1cf93582614429b70b6af8a0d0a984_be53c9ce
20a6493384c842043136613e for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-addwoprj for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-9zwz2md4 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-07jo306d for automatic cleanup: unknown  
resource type semlock                                                           
[2025-08-24 01:03:48,942][__main__][INFO] - 5ff0225193d64880b09909544bef2f44    
[2025-08-24 01:03:48,943][__main__][INFO] - {'width': 200, 'height': 200}       
[2025-08-24 01:03:48,970][__main__][INFO] - f3a49081b00a4b12b663c7235d1e265d    
[2025-08-24 01:03:48,970][__main__][INFO] - {'width': 200, 'height': 100}       
[2025-08-24 01:03:49,234][__main__][INFO] - 059ad676deb84ea4954fd3b069e81c66    
[2025-08-24 01:03:49,234][__main__][INFO] - {'width': 200, 'height': 300}       
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2247_2d1cf93582614429b70b6af8a0d0a984_be53c9ce
20a6493384c842043136613e for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-07jo306d for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-9zwz2md4 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-addwoprj for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2247_2d1cf93582614429b70b6af8a0d0a984_159a4b71
7f64437a85e0c06613ff82a1 for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-ab03j21l for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-i2ud2v0j for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-v7zuqj8z for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-o06miog5 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-pnp2v73w for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2247-9er72j1s for automatic cleanup: unknown  
resource type semlock                                                           
[2025-08-24 01:03:51,288][HYDRA]                                                
Joblib.Parallel(n_jobs=3,backend=loky,prefer=processes,require=None,verbose=0,ti
meout=None,pre_dispatch=2*n_jobs,batch_size=auto,temp_folder=None,max_nbytes=Non
e,mmap_mode=r) is launching 3 jobs                                              
[2025-08-24 01:03:51,288][HYDRA] Launching jobs, sweep output dir :             
multirun/01K3CSMM611DKJJM59Y4VZNXPV                                             
[2025-08-24 01:03:51,288][HYDRA]        #0 : width=400 height=100               
[2025-08-24 01:03:51,288][HYDRA]        #1 : width=400 height=200               
[2025-08-24 01:03:51,288][HYDRA]        #2 : width=400 height=300               
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2283_776ec74644b544ad8357a4ca61130cfa_e2778365
14c947be90c1686c6e6f76aa for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-0krpq1u_ for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-x4y87e6b for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-26paw93b for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-7n1zgi6l for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-9bfxv28f for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-k5t8xq16 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2283_776ec74644b544ad8357a4ca61130cfa_ed1e43d9
0ca4421a90b65b85af0ba48a for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-2vuyut6r for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-11xeicvw for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-ihq9w8r2 for automatic cleanup: unknown  
resource type semlock                                                           
[2025-08-24 01:03:52,675][__main__][INFO] - eae88e32bb274665bffb161b8b037c28    
[2025-08-24 01:03:52,675][__main__][INFO] - {'width': 400, 'height': 100}       
[2025-08-24 01:03:53,125][__main__][INFO] - 7cc2d0a5eace4552add1913ab2695979    
[2025-08-24 01:03:53,125][__main__][INFO] - d5173eb349cb4249a4addcc250153e71    
[2025-08-24 01:03:53,125][__main__][INFO] - {'width': 400, 'height': 200}       
[2025-08-24 01:03:53,125][__main__][INFO] - {'width': 400, 'height': 300}       
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2283_776ec74644b544ad8357a4ca61130cfa_ed1e43d9
0ca4421a90b65b85af0ba48a for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-ihq9w8r2 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-11xeicvw for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-2vuyut6r for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register                                                     
/dev/shm/joblib_memmapping_folder_2283_776ec74644b544ad8357a4ca61130cfa_e2778365
14c947be90c1686c6e6f76aa for automatic cleanup: unknown resource type folder    
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-k5t8xq16 for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-9bfxv28f for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-7n1zgi6l for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-26paw93b for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-x4y87e6b for automatic cleanup: unknown  
resource type semlock                                                           
Traceback (most recent call last):                                              
  File                                                                          
"/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/multiprocessing/resource_
tracker.py", line 295, in main                                                  
    raise ValueError(                                                           
        f'Cannot register {name} for automatic cleanup: '                       
        f'unknown resource type {rtype}')                                       
ValueError: Cannot register /loky-2283-0krpq1u_ for automatic cleanup: unknown  
resource type semlock                                                           
  0:00:07 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 2/2 100%

Results of execution:

  • An experiment named job_parallel is created
  • The same Python script is used but with a different experiment name
  • 2 Python commands are executed sequentially
  • Each Python command runs 3 jobs in parallel (using the hydra/launcher=joblib configuration)

This demonstrates how HydraFlow makes Hydra's powerful parallel execution features easily accessible.

Using the Submit Command

For more complex execution patterns, HydraFlow provides the submit command. Here's our submit job configuration:

job_submit:
  submit: python submit.py example.py
  sets:
    - each: width=250:350:100
      all: height=150,250

The submit command requires two key components:

  1. Your HydraFlow application (example.py in this case)
  2. A command or script that will receive and process a parameter file

Here's our implementation of the submit handler:

submit.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import shlex
import subprocess
import sys
from pathlib import Path


def main() -> None:
    app_file, opt_file = sys.argv[1:]
    text = Path(opt_file).read_text()

    for line in text.splitlines():
        opts = shlex.split(line)
        args = [sys.executable, app_file, *opts]
        print(args)
        subprocess.run(args, check=True)


if __name__ == "__main__":
    main()

How the submit command works:

  1. HydraFlow generates all parameter combinations based on your job configuration
  2. It writes these combinations to a temporary text file (one combination per line)
  3. It runs the command specified in the submit field of your hydraflow.yaml
  4. It appends the temporary file path as the last argument to your command

For example, with submit: python submit.py example.py in your configuration, the actual executed command will be something like: 

python submit.py example.py /tmp/hydraflow_parameters_12345.txt

Let's see it in action with a dry run:

$ hydraflow run job_submit --dry-run
python submit.py example.py /home/runner/work/hydraflow/hydraflow/examples/tmp7tgk_2v3
--multirun width=250 height=150,250 hydra.job.name=job_submit hydra.sweep.dir=multirun/01K3CSMWJM2Y93E2SFD0HVWY2Y
--multirun width=350 height=150,250 hydra.job.name=job_submit hydra.sweep.dir=multirun/01K3CSMWJM2Y93E2SFD0HVWY2Z

And now let's run it:

$ hydraflow run job_submit
2025/08/24 01:03:55 INFO mlflow.tracking.fluent: Experiment with name 'job_submit' does not exist. Creating a new experiment.
[2025-08-24 01:03:57,125][HYDRA] Launching 2 jobs locally
[2025-08-24 01:03:57,125][HYDRA]    #0 : width=250 height=150
[2025-08-24 01:03:57,248][__main__][INFO] - 5e5698fd07ec4bdd990d150706c5886a
[2025-08-24 01:03:57,248][__main__][INFO] - {'width': 250, 'height': 150}
[2025-08-24 01:03:57,251][HYDRA]    #1 : width=250 height=250
[2025-08-24 01:03:57,337][__main__][INFO] - c79f6ae566824016ae5aa13d3dda7b6c
[2025-08-24 01:03:57,337][__main__][INFO] - {'width': 250, 'height': 250}
[2025-08-24 01:03:58,888][HYDRA] Launching 2 jobs locally
[2025-08-24 01:03:58,888][HYDRA]    #0 : width=350 height=150
[2025-08-24 01:03:59,012][__main__][INFO] - 6b312cd744e14cb7bd9b1b69dca1ee7e
[2025-08-24 01:03:59,012][__main__][INFO] - {'width': 350, 'height': 150}
[2025-08-24 01:03:59,015][HYDRA]    #1 : width=350 height=250
[2025-08-24 01:03:59,101][__main__][INFO] - fc7a7fc605c34fc6b8cb080915c6cab4
[2025-08-24 01:03:59,101][__main__][INFO] - {'width': 350, 'height': 250}
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=250', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01K3CSMXVENCMMQ991005M6CXF']
['/home/runner/work/hydraflow/hydraflow/.venv/bin/python', 'example.py', '--multirun', 'width=350', 'height=150,250', 'hydra.job.name=job_submit', 'hydra.sweep.dir=multirun/01K3CSMXVENCMMQ991005M6CXG']

Our submit.py script implements a simple processor that:

  1. Accepts two arguments: the application file (example.py) and the parameter file
  2. Reads each line from the parameter file
  3. Runs the application with each set of parameters sequentially

In real-world scenarios, you could customize this handler to:

  • Submit jobs to compute clusters (SLURM, PBS, etc.)
  • Implement custom scheduling logic
  • Distribute workloads based on resource requirements

Reviewing Results

With HydraFlow, all important data is stored in MLflow, so we can safely delete the Hydra output directories:

$ rm -rf multirun

Let's check the directory structure:

.
├── mlruns
│   ├── 0
│   │   └── meta.yaml
│   ├── 316773436696952934
│   │   ├── 5e5698fd07ec4bdd990d150706c5886a
│   │   ├── 6b312cd744e14cb7bd9b1b69dca1ee7e
│   │   ├── c79f6ae566824016ae5aa13d3dda7b6c
│   │   ├── fc7a7fc605c34fc6b8cb080915c6cab4
│   │   └── meta.yaml
│   ├── 571006024065909081
│   │   ├── 08ec0b26237c4b32ad30550d1f552802
│   │   ├── 2053bd6ddcab4465963146967b396c33
│   │   ├── 60fa8395d5b14ddf9bded3be44847fe0
│   │   ├── 731ca75db9bb47bdae68c7a5f08c9d4a
│   │   ├── 8ac4dd4406f045378b0ac592d987b7c3
│   │   ├── b61455bf287442448a172e3f95eee283
│   │   └── meta.yaml
│   └── 957879147273956178
│       ├── 059ad676deb84ea4954fd3b069e81c66
│       ├── 5ff0225193d64880b09909544bef2f44
│       ├── 7cc2d0a5eace4552add1913ab2695979
│       ├── d5173eb349cb4249a4addcc250153e71
│       ├── eae88e32bb274665bffb161b8b037c28
│       ├── f3a49081b00a4b12b663c7235d1e265d
│       └── meta.yaml
├── example.py
├── hydraflow.yaml
└── submit.py

After cleanup, we can observe:

  • There are three experiments (one for each job type)
  • Each experiment contains multiple runs
  • A total of 16 runs were executed across all jobs

Summary

In this tutorial, you've learned how to:

  1. Define different types of experiment workflows in a hydraflow.yaml file
  2. Execute sequential and parallel job runs
  3. Use the submit command for custom execution patterns
  4. Preview jobs with dry runs
  5. Manage and organize experiment outputs

These workflow automation capabilities allow you to efficiently manage complex experiment configurations, making your machine learning research more organized and reproducible.

Next Steps

Now that you've learned about workflow automation, try:

  • Defining your own custom workflows
  • Exploring more complex parameter sweep combinations
  • Learning how to Analyze Results from your experiments

For more detailed information, refer to: