1. Sparkle user guide
1.1. Quick start
Note
Sparkle currently relies on Slurm, but in some cases works locally as well. Sparkle also relies on RunSolver, which is a Linux based program. Thus Sparkle can only run on Linux based systems in many cases.
Follow these steps:
Prepare an algorithm configuration or an algorithm selection.
1.2. Installing Sparkle
Note
The installation process use the conda
command available in Anaconda or Miniconda to manage some dependencies.
1.2.1. Get a copy of Sparkle
To get a copy of Sparkle you can clone the repository.
If git
is available on your system, this will clone the Sparkle repository and create a subdirectory named sparkle
:
$ git clone https://github.com/ADA-research/Sparkle
You can also download the stable version here:
1.2.2. Install dependencies
Sparkle depends on Python 3.9+, swig 3.0, gnuplot, LaTeX, and multiple Python packages. An easy way to install most of what is needed is to use the conda
package manager (https://docs.conda.io/en/latest/miniconda.html).
Note
LaTeX is used to create the reports and the documentation and must be installed manually on the system. If you don’t plan to use the reports or recreate the documentations, you can skip it.
You can install the base requirements with
$ conda env create -f environment.yml
This will create an environment named sparkle
that contains everything needed to run Sparkle, except LaTeX that needs to be installed manually.
To activate the environment in the current shell, execute:
$ conda activate sparkle
Note
You will need to reactivate the environment every time you log in, before using Sparkle.
The file environment.yml
contains a tested list of Python packages with fixed versions required to execute Sparkle. We recommended using it.
The file environment-dev.txt
contains unpinned packages and the dependencies are not resolved. It is used for development and may cause problems.
The two environments can be created in parallel since one is named sparkle
and the other sparkle-dev
. If you want to update an environment it is better to do a clean installation by removing and recreating it. For example:
$ conda deactivate
$ conda env remove -n sparkle
$ conda env create -f environment.yml
$ conda activate sparkle
This should be fast as both conda
and pip
use local cache for the packages.
1.2.3. Configure Sparkle/Slurm
Before running Sparkle, you probably want to have a look at the settings described in Section 1.9. In particular, the default Slurm settings are set to work with the Grace cluster in Leiden University and should be adapted to your specific cluster.
1.3. Algorithm Configuration
Configuring an algorithm has the following minimal requirements for the algorithm (for an example of a solver directory see Section 1.6.2):
A working solver executable
An algorithm wrapper called
sprakle_smac_wrapper.py
A PCS (parameter configuration space) file
Further, training and testing instance sets are needed (for an example of an instances directory see Section 1.6.1). For the purpose of testing whether your configuration setup works with Sparkle, it is advised to primarily use instances that are solved (relatively) quickly even with the default parameters.
Note
See the example page for a walk-through on how to perform configuration with Sparkle.
1.3.1. Creating a wrapper for your algorithm
A template for the wrapper that connects your algorithm with Sparkle is
available at Examples/Resources/Solvers/template/sparkle_smac_wrapper.py
. Within
this template a number of TODO
s are indicated where you are likely
to need to make changes for your specific algorithm. You can also
compare the different example solvers to get an idea for what kind of
changes are needed.
1.3.2. Parameter configuration space (PCS) file
The PCS (parameter configuration space) format [1] is used to pass the
possible parameter ranges of an algorithm to Sparkle in a .pcs
file.
For an example see e.g.
Examples/Resources/Solvers/PbO-CCSAT-Generic/PbO-CCSAT-params_test.pcs
.
In this file you should enter all configurable parameters of your algorithm. Note that parameters such as the random seed used by the algorithm should not be configured and therefore should also not be included in the PCS file.
1.4. Algorithm Selection
Creating a portfolio selector requires multiple algorithms with the following minimal requirements (for an example of a solver directory see Section 1.6.3):
A working solver executable
An algorithm wrapper called
sparkle_run_default_wrapper.py
Further, training and testing instance sets are needed (for an example of an instances directory see Section 1.6.1). For the purpose of testing whether your selection setup works with Sparkle, it is advised to primarily use instances that are solved (relatively) quickly.
Note
See the example page for a walk-through on how to perform selection with Sparkle.
1.4.1. Creating a wrapper for your algorithm
A template for the wrapper that connects your algorithm with Sparkle is
available at
Examples/Resources/Solvers/template/sparkle_run_default_wrapper.py
.
Within this template a number of TODO
s are indicated where you are
likely to need to make changes for your specific algorithm. You can also
compare the different example solvers to get an idea for what kind of
changes are needed.
1.5. Executing commands
Executing commands in Sparkle is as simple as running them in the top directory of Sparkle, for example:
CLI/initialise.py
Do note that when running on a cluster additional arguments may be needed, for instance under the Slurm workload manager the above command would change to something like:
srun -N1 -n1 -c1 CLI/initialise.py
In the Examples/
directory a number of common command sequences are
given. For instance, for configuration with specified training and
testing sets see e.g. Examples/configuration.md
for an example of a
sequence of commands to execute. Note that some command run in the
background and need time to complete before the next command is
executed. To see whether a command is still running the Slurm command
squeue
can be used.
In the Output/
directory paths to generated scripts and logs are
gathered per executed command.
1.6. File structure
1.6.1. A typical instance directory
An instance directory should look something like this:
Instances/
Example_Instance_Set/
instance_a.cnf
instance_b.cnf
... ...
instance_z.cnf
This directory simply contains a collection of instances, as example here SAT instances in the CNF format are given.
For instances consisting of multiple files one additional file called sparkle_instance_list.txt
should be
included in the Example_Instance_Set
directory, describing which
files together form an instance. The format is a single instance per
line with each file separated by a space, as shown below.
instance_a_part_one.abc instance_a_part_two.xyz
instance_b_part_one.abc instance_b_part_two.xyz
... ...
instance_z_part_one.abc instance_z_part_two.xyz
1.6.2. A typical solver directory (configuration)
A solver directory should look something like this:
Solver/
Example_Solver/
solver
sparkle_smac_wrapper.py
parameters.pcs
Here solver
is a binary executable of the solver that is to be
configured. The sprakle_smac_wrapper.py
is a wrapper that Sparkle
should call to run the solver with specific settings, and then returns a
result for the configurator. In parameters.pcs
the configurable
parameters are described in the PCS format. Finally, when importing your
Solver into Sparkle, a binary executable of the runsolver tool runsolver
is added.
This allows Sparkle to make fair time measurements for all configuration experiments.
1.6.3. A typical solver directory (selection)
A solver directory should look something like this:
Solver/
Example_Solver/
solver
sparkle_run_default_wrapper.py
Here solver
is a binary executable of a solver that is to be
included in a portfolio selector. The sprakle_run_default_wrapper.py
is a wrapper that Sparkle should call to run the solver on a specific
instance.
1.6.4. The output directory
Note
This section describes a desirable behaviour but has not been implemented fully yet.
The output directory is located at the root of the Sparkle directory. Its structure is as follows:
Output/
Logs/
Common/
Raw_Data/
Analysis/
Configuration/
Raw_Data/
run_<alias>/
related files
Analysis/
Parallel_Portfolio/
Raw_Data/
run_<alias>/
related files
Analysis/
Selection/
Raw_Data/
run_<alias>/
related files
Analysis/
The alias
is based on the command and a timestamp.
The Logs
directory should contain the history of commands and their output such that one can easily know what has been done in which order and find enough pointers to debug unwanted behaviour.
Other directories are cut into two subdirectories: Raw_Data
contains the data produced by the main command, often time consuming to generate, handle with care; Analysis
contains information extracted from the raw data, easy to generate, plots and reports.
For each type of task run by Sparkle, the related files
differ. The aim is always to have all required files for reproducibility. A copy of the sparkle configuration file at the time of the run and of all files relevant to the run, a copy of any log or error file that could help with debugging or a link to it, and the output of the executed task.
For configuration the configuration trajectory if available, the training and testing sets, the default configuration and the final found configuration. The performance of those will be in the Analysis folder.
For parallel portfolio the resulting portfolio and its components. The performance of the portfolio will be in the Analysis folder.
For selection the algorithms and their performance on the training set, the model(s) generated if available and the resulting selector. The performance evaluation of the selector will be in the Analysis folder.
For analysis a link to the folder on which the analysis was performed (configuration, portfolio or selection), the performance evaluation from it and the report if it was generated.
1.7. Wrappers
1.7.1. sparkle_run_default_wrapper.py
The sparkle_run_default_wrapper.py
has two functions that need to be
implemented for each algorithm:
print_command(instance_file, seed_str: str, cutoff_time_str: str)
print_output(terminal_output_file: str)
print_command(...)
should print a command line call that Sparkle can
use to run the algorithm on a given instance file. Ideally, for
reproducibility purposes, the seed provided by Sparkle should also be
passed to the algorithm. If the algorithm requires this, the cutoff time
can also be passed to the algorithm. However, in this case the cutoff
time should be made very large. For instance by multiplying by ten with:
cutoff_time_str = str(int(cutoff_time_str) * 10)
. This is necessary
to ensure Sparkle stops the algorithm after the cutoff time, rather than
the algorithm itself. By doing this it is ensured runtime measurements
are always done by Sparkle, and thus consistent between algorithms that
might measure time differently.
print_output(...)
should process the algorithm output. If the
performance measure is RUNTIME
, this function only needs to output
the algorithm status. For all QUALITY
performance measures both the
algorithm status and the solution quality have to be given. Sparkle
internally measures RUNTIME
, while it can be overwritten by the user
if desired, for consistent runtime measurements between solvers this is
not recommended. The output should be printed and formatted as in the
example below.
quality 8734
status SUCCESS
Status can hold the following values {SUCCESS, TIMEOUT, CRASHED}
. If
the status is not known, reporting SUCCESS
will allow Sparkle to
continue, but may mean that Sparkle does not know when the algorithm
crashed, and continues with faulty results.
1.8. Commands
Currently the commands below are available in Sparkle (listed
alphabetically). Every command can be called with the –help
option
to get a description of the required arguments and other options.
Note
Arguments in [square brackets] are optional, arguments without brackets are mandatory. Input in <chevrons> indicate required text input, {curly brackets} indicate a set of inputs to choose from.
1.8.1. about_cli.py
usage: about_cli.py [-h]
- -h, --help
show this help message and exit
1.8.2. add_feature_extractor.py
usage: add_feature_extractor.py [-h]
[--run-extractor-now | --run-extractor-later]
[--nickname NICKNAME] [--parallel]
extractor-path
- extractor-path
path or nickname of the feature extractor
- -h, --help
show this help message and exit
- --run-extractor-now
immediately run the feature extractor(s) on all the instances
- --run-extractor-later
do not immediately run the feature extractor(s) on all the instances (default)
- --nickname <nickname>
set a nickname for the feature extractor
- --parallel
Run the command in parallel
1.8.3. add_instances.py
usage: add_instances.py [-h] [--run-extractor-now | --run-extractor-later]
[--run-solver-now | --run-solver-later]
[--nickname NICKNAME] [--parallel]
instances-path
- instances-path
path to the instance set
- -h, --help
show this help message and exit
- --run-extractor-now
immediately run the feature extractor(s) on all the instances
- --run-extractor-later
do not immediately run the feature extractor(s) on all the instances (default)
- --run-solver-now
immediately run the solver(s) on all instances
- --run-solver-later
do not immediately run the solver(s) on all instances (default)
- --nickname <nickname>
set a nickname for the instance set
- --parallel
Run the command in parallel
1.8.4. add_solver.py
Add a solver to the Sparkle platform.
usage: add_solver.py [-h] --deterministic {0,1}
[--run-solver-now | --run-extractor-later]
[--nickname NICKNAME] [--parallel]
[--solver-variations SOLVER_VARIATIONS]
[--run-on {Runner.LOCAL,Runner.SLURM}] [--skip-checks]
solver-path
- solver-path
path to the solver
- -h, --help
show this help message and exit
- --deterministic {0,1}
indicate whether the solver is deterministic or not
- --run-solver-now
immediately run the solver(s) on all instances
- --run-extractor-later
do not immediately run the solver(s) on all instances (default)
- --nickname <nickname>
set a nickname for the solver
- --parallel
Run the command in parallel
- --solver-variations <solver_variations>
Use this option to add multiple variations of the solver by using a different random seed for each varation.
- --run-on
On which computer or cluster environment to execute the calculation.
- --skip-checks
Checks the solver’s functionality by testing it on an instance and the pcs file, when applicable.
1.8.5. cleanup_temporary_files.py
usage: cleanup_temporary_files.py [-h]
- -h, --help
show this help message and exit
1.8.6. compute_features.py
usage: compute_features.py [-h] [--recompute] [--parallel]
[--settings-file SETTINGS_FILE]
[--run-on {Runner.LOCAL,Runner.SLURM}]
- -h, --help
show this help message and exit
- --recompute
Re-run feature extractor for instances with previously computed features
- --parallel
Run the command in parallel
- --settings-file
Specify the settings file to use in case you want to use one other than the default
- --run-on
On which computer or cluster environment to execute the calculation.
1.8.7. compute_marginal_contribution.py
usage: compute_marginal_contribution.py [-h] (--perfect | --actual)
[--recompute]
[--selector-timeout SELECTOR_TIMEOUT]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--settings-file SETTINGS_FILE]
- -h, --help
show this help message and exit
- --perfect
compute the marginal contribution for the perfect selector
- --actual
compute the marginal contribution for the actual selector
- --recompute
force marginal contribution to be recomputed even when it already exists in file for the current selector
- --selector-timeout <selector_timeout>
Cuttoff time (in seconds) for the algorithmselector construction
- --performance-measure
the performance measure, e.g. runtime
- --settings-file
Specify the settings file to use in case you want to use one other than the default
1.8.8. configure_solver.py
Configure a solver in the Sparkle platform.
usage: configure_solver.py [-h] [--configurator CONFIGURATOR] --solver SOLVER
--instance-set-train INSTANCE_SET_TRAIN
[--instance-set-test INSTANCE_SET_TEST]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--target-cutoff-time TARGET_CUTOFF_TIME]
[--wallclock-time WALLCLOCK_TIME]
[--cpu-time CPU_TIME] [--solver-calls SOLVER_CALLS]
[--number-of-runs NUMBER_OF_RUNS]
[--settings-file SETTINGS_FILE] [--use-features]
[--validate] [--ablation]
[--run-on {Runner.LOCAL,Runner.SLURM}]
- -h, --help
show this help message and exit
- --configurator <configurator>
path to configurator
- --solver <solver>
path to solver
- --instance-set-train <instance_set_train>
path to training instance set
- --instance-set-test <instance_set_test>
path to test instance set (only for validating)
- --performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}
the performance measure, e.g. runtime
- --target-cutoff-time <target_cutoff_time>
cutoff time per target algorithm run in seconds
- --wallclock-time <wallclock_time>
configuration budget per configurator run in seconds (wallclock)
- --cpu-time <cpu_time>
configuration budget per configurator run in seconds (cpu)
- --solver-calls <solver_calls>
number of solver calls to execute
- --number-of-runs <number_of_runs>
number of configuration runs to execute
- --settings-file
Specify the settings file to use in case you want to use one other than the default
- --use-features
use the training set’s features for configuration
- --validate
validate after configuration
- --ablation
run ablation after configuration
- --run-on
On which computer or cluster environment to execute the calculation.
Note that the test instance set is only used if the --ablation
or --validation
flags are given
1.8.9. construct_sparkle_portfolio_selector.py
usage: construct_sparkle_portfolio_selector.py [-h]
[--recompute-portfolio-selector]
[--recompute-marginal-contribution]
[--selector-timeout SELECTOR_TIMEOUT]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
- -h, --help
show this help message and exit
- --recompute-portfolio-selector
force the construction of a new portfolio selector even when it already exists for the current feature and performance data. NOTE: This will also result in the computation of the marginal contributions of solvers to the new portfolio selector.
- --recompute-marginal-contribution
force marginal contribution to be recomputed even when it already exists in file for the current selector
- --selector-timeout <selector_timeout>
Cuttoff time (in seconds) for the algorithmselector construction
- --performance-measure
the performance measure, e.g. runtime
1.8.10. generate_report.py
Without any arguments a report for the most recent algorithm selection or algorithm configuration procedure is generated.
usage: generate_report.py [-h] [--solver SOLVER]
[--instance-set-train INSTANCE_SET_TRAIN]
[--instance-set-test INSTANCE_SET_TEST]
[--no-ablation [FLAG_ABLATION]] [--selection]
[--test-case-directory TEST_CASE_DIRECTORY]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--settings-file SETTINGS_FILE]
- -h, --help
show this help message and exit
- --solver <solver>
path to solver for an algorithm configuration report
- --instance-set-train <instance_set_train>
path to training instance set included in Sparkle for an algorithm configuration report
- --instance-set-test <instance_set_test>
path to testing instance set included in Sparkle for an algorithm configuration report
- --no-ablation <flag_ablation>
turn off reporting on ablation for an algorithm configuration report
- --selection
set to generate a normal selection report
- --test-case-directory <test_case_directory>
Path to test case directory of an instance set for a selection report
- --performance-measure
the performance measure, e.g. runtime
- --settings-file
Specify the settings file to use in case you want to use one other than the default
Note that if a test instance set is given, the training instance set must also be given.
1.8.11. initialise.py
Initialise the Sparkle platform, this command does not have any arguments.
usage: initialise.py [-h]
- -h, --help
show this help message and exit
1.8.12. load_snapshot.py
usage: load_snapshot.py [-h] snapshot-file-path
- snapshot-file-path
path to the snapshot file
- -h, --help
show this help message and exit
1.8.13. remove_feature_extractor.py
usage: remove_feature_extractor.py [-h] extractor-path
- extractor-path
path or nickname of the feature extractor
- -h, --help
show this help message and exit
1.8.14. remove_instances.py
usage: remove_instances.py [-h] instances-path
- instances-path
path to or nickname of the instance set
- -h, --help
show this help message and exit
1.8.15. remove_solver.py
usage: remove_solver.py [-h] solver
- solver
name, path to or nickname of the solver
- -h, --help
show this help message and exit
1.8.16. run_ablation.py
Runs parameter importance between the default and configured parameters with ablation. This command requires a finished configuration for the solver instance pair.
usage: run_ablation.py [-h] [--solver SOLVER]
[--instance-set-train INSTANCE_SET_TRAIN]
[--instance-set-test INSTANCE_SET_TEST]
[--ablation-settings-help]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--target-cutoff-time TARGET_CUTOFF_TIME]
[--wallclock-time WALLCLOCK_TIME]
[--number-of-runs NUMBER_OF_RUNS] [--racing RACING]
[--settings-file SETTINGS_FILE]
[--run-on {Runner.LOCAL,Runner.SLURM}]
- -h, --help
show this help message and exit
- --solver <solver>
path to solver
- --instance-set-train <instance_set_train>
path to training instance set
- --instance-set-test <instance_set_test>
path to test instance set
- --ablation-settings-help
Prints a list of setting that can be used for the ablation analysis
- --performance-measure
the performance measure, e.g. runtime
- --target-cutoff-time
cutoff time per target algorithm run in seconds
- --wallclock-time <wallclock_time>
configuration budget per configurator run in seconds (wallclock)
- --number-of-runs
Number of configuration runs to execute
- --racing
Performs abaltion analysis with racing
- --settings-file
Specify the settings file to use in case you want to use one other than the default
- --run-on
On which computer or cluster environment to execute the calculation.
Note that if no test instance set is given, the validation is performed on the training set.
1.8.17. run_configured_solver.py
usage: run_configured_solver.py [-h] [--settings-file SETTINGS_FILE]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--parallel]
[--run-on {Runner.LOCAL,Runner.SLURM}]
instance_path [instance_path ...]
- instance_path
Path(s) to instance file(s) (when multiple files are given, it is assumed this is a multi-file instance) or instance directory.
- -h, --help
show this help message and exit
- --settings-file
Specify the settings file to use in case you want to use one other than the default
- --performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}
the performance measure, e.g. runtime
- --parallel
Run the command in parallel
- --run-on
On which computer or cluster environment to execute the calculation.
1.8.18. run_solvers.py
usage: run_solvers.py [-h] [--recompute] [--parallel]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--target-cutoff-time TARGET_CUTOFF_TIME]
[--also-construct-selector-and-report]
[--verifier {NONE,SAT}]
[--run-on {Runner.LOCAL,Runner.SLURM}]
[--settings-file SETTINGS_FILE]
- -h, --help
show this help message and exit
- --recompute
recompute the performance of all solvers on all instances
- --parallel
Run the command in parallel
- --performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}
the performance measure, e.g. runtime
- --target-cutoff-time <target_cutoff_time>
cutoff time per target algorithm run in seconds
- --also-construct-selector-and-report
after running the solvers also construct the selector and generate the report
- --verifier {NONE,SAT}
problem specific verifier that should be used to verify solutions found by a target algorithm
- --run-on
On which computer or cluster environment to execute the calculation.
- --settings-file
Specify the settings file to use in case you want to use one other than the default
1.8.19. run_parallel_portfolio.py
usage: run_parallel_portfolio.py [-h] --instance-paths PATH [PATH ...]
[--portfolio-name PORTFOLIO_NAME]
[--solvers SOLVERS [SOLVERS ...]]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--cutoff-time CUTOFF_TIME]
[--run-on {Runner.LOCAL,Runner.SLURM}]
[--settings-file SETTINGS_FILE]
- -h, --help
show this help message and exit
- --instance-paths <path>
Specify the instance_path(s) on which the portfolio will run. This can be a space-separated list of instances containing instance sets and/or singular instances. For example –instance-paths Instances/PTN/Ptn-7824-b01.cnf Instances/PTN2/
- --portfolio-name <portfolio_name>
Specify a name of the portfolio. If none is given, one will be generated.
- --solvers <solvers>
Specify the list of solvers to be used. If not specifed, all solvers known in Sparkle will be used.
- --performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}
the performance measure, e.g. runtime
- --cutoff-time <cutoff_time>
The duration the portfolio will run before the solvers within the portfolio will be stopped (default: 60)
- --run-on
On which computer or cluster environment to execute the calculation.
- --settings-file
Specify the settings file to use in case you want to use one other than the default
1.8.20. run_sparkle_portfolio_selector.py
usage: run_sparkle_portfolio_selector.py [-h]
[--run-on {Runner.LOCAL,Runner.SLURM}]
[--settings-file SETTINGS_FILE]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
instance_path [instance_path ...]
- instance_path
Path to instance or instance directory
- -h, --help
show this help message and exit
- --run-on
On which computer or cluster environment to execute the calculation.
- --settings-file
Specify the settings file to use in case you want to use one other than the default
- --performance-measure
the performance measure, e.g. runtime
1.8.21. run_status.py
usage: run_status.py [-h] [--verbose]
- -h, --help
show this help message and exit
- --verbose, -v
output status in verbose mode
1.8.22. save_snapshot.py
usage: save_snapshot.py [-h]
- -h, --help
show this help message and exit
1.8.23. wait.py
usage: wait.py [-h]
[--job-id JOB_ID | --command {ABOUT,ADD_FEATURE_EXTRACTOR,ADD_INSTANCES,ADD_SOLVER,CLEANUP_CURRENT_SPARKLE_PLATFORM,CLEANUP_TEMPORARY_FILES,COMPUTE_FEATURES,COMPUTE_MARGINAL_CONTRIBUTION,CONFIGURE_SOLVER,CONFIGURE_SOLVER_CALLBACK,CONSTRUCT_SPARKLE_PORTFOLIO_SELECTOR,GENERATE_REPORT,INITIALISE,LOAD_SNAPSHOT,REMOVE_FEATURE_EXTRACTOR,REMOVE_INSTANCES,REMOVE_SOLVER,RUN_ABLATION,RUN_ABLATION_VALIDATION,ABLATION_CALLBACK,ABLATION_VALIDATION_CALLBACK,RUN_SOLVERS,RUN_SPARKLE_PORTFOLIO_SELECTOR,RUN_STATUS,SAVE_SNAPSHOT,SPARKLE_WAIT,SYSTEM_STATUS,VALIDATE_CONFIGURED_VS_DEFAULT,RUN_CONFIGURED_SOLVER,RUN_PARALLEL_PORTFOLIO,CSV_MERGE,VALIDATION}]
- -h, --help
show this help message and exit
- --job-id <job_id>
job ID to wait for
- --command {ABOUT,ADD_FEATURE_EXTRACTOR,ADD_INSTANCES,ADD_SOLVER,CLEANUP_CURRENT_SPARKLE_PLATFORM,CLEANUP_TEMPORARY_FILES,COMPUTE_FEATURES,COMPUTE_MARGINAL_CONTRIBUTION,CONFIGURE_SOLVER,CONFIGURE_SOLVER_CALLBACK,CONSTRUCT_SPARKLE_PORTFOLIO_SELECTOR,GENERATE_REPORT,INITIALISE,LOAD_SNAPSHOT,REMOVE_FEATURE_EXTRACTOR,REMOVE_INSTANCES,REMOVE_SOLVER,RUN_ABLATION,RUN_ABLATION_VALIDATION,ABLATION_CALLBACK,ABLATION_VALIDATION_CALLBACK,RUN_SOLVERS,RUN_SPARKLE_PORTFOLIO_SELECTOR,RUN_STATUS,SAVE_SNAPSHOT,SPARKLE_WAIT,SYSTEM_STATUS,VALIDATE_CONFIGURED_VS_DEFAULT,RUN_CONFIGURED_SOLVER,RUN_PARALLEL_PORTFOLIO,CSV_MERGE,VALIDATION}
command you want to run. Sparkle will wait for dependencies of this command to be completed
1.8.24. system_status.py
usage: system_status.py [-h] [--verbose]
- -h, --help
show this help message and exit
- --verbose, -v
output status in verbose mode
1.8.25. validate_configured_vs_default.py
Test the performance of the configured solver and the default solver by doing validation experiments on the training and test sets.
usage: validate_configured_vs_default.py [-h] --solver SOLVER
--instance-set-train
INSTANCE_SET_TRAIN
[--instance-set-test INSTANCE_SET_TEST]
[--configurator CONFIGURATOR]
[--performance-measure {ERR,RUNTIME,QUALITY_ABSOLUTE,QUALITY_ABSOLUTE_MINIMISATION,QUALITY_ABSOLUTE_MAXIMISATION}]
[--target-cutoff-time TARGET_CUTOFF_TIME]
[--settings-file SETTINGS_FILE]
[--run-on {Runner.LOCAL,Runner.SLURM}]
- -h, --help
show this help message and exit
- --solver <solver>
path to solver
- --instance-set-train <instance_set_train>
path to training instance set
- --instance-set-test <instance_set_test>
path to test instance set (only for validating)
- --configurator <configurator>
path to configurator
- --performance-measure
the performance measure, e.g. runtime
- --target-cutoff-time
cutoff time per target algorithm run in seconds
- --settings-file
Specify the settings file to use in case you want to use one other than the default
- --run-on
On which computer or cluster environment to execute the calculation.
1.9. Sparkle settings
Most settings can be controlled through
Settings/sparkle_settings.ini
. Possible settings are summarised per
category in Section 1.9.2. For any settings
that are not provided the defaults will be used. Meaning, in the extreme
case, that if the settings file is empty (and nothing is set through the
command line) everything will run with default values.
For convenience after every command Settings/latest.ini
is written
with the used settings. This can, for instance, be used to provide the
same settings to the next command in a chain. E.g. for
validate_configured_vs_default
after configure_solver
. The used
settings are also recorded in the relevant Output/
subdirectory.
Note that when writing settings Sparkle always uses the name, and not an
alias.
1.9.1. Example sparkle_settings.ini
This is a short example to show the format, see the settings file in
Settings/sparkle_settings.ini
for more.
[general]
performance_measure = RUNTIME
target_cutoff_time = 60
[configuration]
number_of_runs = 25
[slurm]
number_of_runs_in_parallel = 25
1.9.2. Names and possible values
[general]
performance_measure
aliases:
smac_run_obj
values:
{RUNTIME, QUALITY_ABSOLUTE
(also:QUALITY}
)
RUNTIME
focuses on runtime the solver requires,
QUALITY_ABSOLUTE
andQUALITY
focuses on the average absolute improvements on the instancesdescription: The type of performance measure that sparkle uses.
target_cutoff_time
aliases:
smac_each_run_cutoff_time
,cutoff_time_each_performance_computation
values: integer
description: The time a solver is allowed to run before it is terminated.
extractor_cutoff_time
aliases:
cutoff_time_each_feature_computation
values: integer
description: The time a feature extractor is allowed to run before it is terminated. In case of multiple feature extractors this budget is divided equally.
penalty_multiplier
aliases:
penalty_number
values: integer
description: In case of not solving an instance within the cutoff time the runtime is set to be the
penalty_multiplier * cutoff_time
.
solution_verifier
aliases: N/A
values:
{NONE, SAT}
note: Only available for SAT solving.
[configuration]
wallclock_time
aliases:
smac_whole_time_budget
values: integer
description: The wallclock time one configuration run is allowed to use for finding configurations.
cpu_time
aliases:
smac_cpu_time_budget
values: integer
description: The cpu time one configuration run is allowed to use for finding configurations.
solver_calls
aliases:
smac_solver_calls_budget
values: integer
description: The number of solver calls one configuration run is allowed to use for finding configurations.
number_of_runs
aliases:
num_of_smac_runs
values: integer
description: The number of separate configurations runs.
[smac]
target_cutoff_length
aliases:
smac_each_run_cutoff_length
values:
{max}
(other values: whatever is allowed by SMAC)
[ablation]
racing
aliases:
ablation_racing
values: boolean
description: Use racing when performing the ablation analysis between the default and configured parameters
[slurm]
number_of_runs_in_parallel
aliases:
smac_run_obj
values: integer
description: The number of configuration runs that can run in parallel.
max_parallel_runs_per_node
aliases:
clis_per_node
values: integer
note: Not really a Slurm option, will likely be moved to another section.
description: The number of parallel processes that can be run on one compute node. In case a node has 32 cores and each solver uses 2 cores, the
max_parallel_runs_per_node
is at most 16.
1.9.3. Priorities
Sparkle has a large flexibility with passing along settings. Settings provided through different channels have different priorities as follows:
Default –- Default values will be overwritten if a value is given through any other mechanism;
File –- Settings form the
Settings/sparkle_settings.ini
overwrite default values, but are overwritten by settings given through the command line;Command line file -– Settings files provided through the command line, overwrite default values and other settings files.
Command line –- Settings given through the command line overwrite all other settings, including settings files provided through the command line.
1.9.4. Slurm (focused on Grace)
Slurm settings can be specified in the
Settings/sparkle_slurm_settings.txt
file. Currently these settings
are inserted as is in any srun
or sbatch
calls done by
Sparkle. This means that any options exclusive to one or the other
currently should not be used (see
Section 1.9.4.2).
To overwrite the default settings specific to the cluster Grace in Leiden, you should set the option “–partition” with a valid value on your cluster. Also, you might have to adapt “–mem-per-cpu” to your system.
1.9.4.1. Tested options
Below a list of tested Slurm options for srun
and sbatch
is
included. Most other options for these commands should also be safe to
use (given they are valid), but have not been explicitly tested. Note
that any options related to commands other than srun
and sbatch
should not be used with Sparkle, and should not be included in
Settings/sparkle_slurm_settings.txt
.
-–partition / -p
-–exclude
-–nodelist
1.9.4.2. Disallowed options
The options below are exclusive to sbatch
and are thus disallowed:
-–array
-–clusters
-–wrap
The options below are exclusive to srun
and are thus disallowed:
-–label
1.9.4.3. Nested srun
calls
A number of Sparkle commands internally call the srun
command, and
for those commands the provided settings need to match the restrictions
of your call to a Sparkle command. Take for instance the following
command:
srun -N1 -n1 -p graceTST CLI/configure_solver.py --solver Solvers/PbO-CCSAT-Generic --instances-train Instances/PTN/
This call restricts itself to the graceTST
partition (the
graceTST
partition only consists of node 22). So if the settings
file contains the setting –exclude=ethnode22
, all available nodes
are excluded, and the command cannot execute any internal srun
commands it may have.
Finally, Slurm ignores nested partition settings for srun
, but not
for sbatch
. This means that if you specify the graceTST
partition (as above) in your command, but the graceADA
partition in
the settings file, Slurm will still execute any nested srun
commands
on the graceTST
partition only.
1.10. Required packages
1.10.1. Sparkle on Grace
Grace is the computing cluster of the ADA group [2] at LIACS, Leiden University. Since not all packages required by Sparkle are installed on the system, some have to be installed local to the user.
1.10.1.1. Making your algorithm run on Grace
Shell and Python scripts should work as is. If a compiled binary does not work, you may have to compile it on Grace and manually install packages on Grace that are needed by your algorithm.
1.10.1.2. General requirements
Other software used by Sparkle:
pdflatex
latex
bibtex
gnuplot
gnuplot-x11
To manually install gnuplot
see for instance the instructions on
their website http://www.gnuplot.info/development/
1.11. Installation and compilation of examples
1.11.1. Solvers
1.11.1.1. CSCCSat
CSCCSat can be recompiled as follows in the
Examples/Resources/Solvers/CSCCSat/
directory:
unzip src.zip
cd src/CSCCSat_source_codes/
make
cp CSCCSat ../../
1.11.1.2. MiniSAT
MiniSAT can be recompiled as follows in the
Examples/Resources/Solvers/MiniSAT/
directory:
unzip src.zip
cd minisat-master/
make
cp build/release/bin/minisat ../
1.11.1.3. PbO-CCSAT
PbO-CCSAT can be recompiled as follows in the
Examples/Resources/Solvers/PbO-CCSAT-Generic/
directory:
unzip src.zip
cd PbO-CCSAT-master/PbO-CCSAT_process_oriented_version_source_code/
make
cp PbO-CCSAT ../../
1.11.1.4. TCA and FastCA
The TCA and FastCA solvers, require GLIBCXX_3.4.21
. This library
comes with GCC 5.1.0
(or greater). Following installation you may
have to update environment variables such as
LD_LIBRARY_PATH, LD_RUN_PATH, CPATH
to point to your installation
directory.
TCA can be recompiled as follows in the
Examples/Resources/CCAG/Solvers/TCA/
directory:
unzip src.zip
cd TCA-master/
make clean
make
cp TCA ../
FastCA can be recompiled as follows in the
Examples/Resources/CCAG/Solvers/FastCA/
directory:
unzip src.zip
cd fastca-master/fastCA/
make clean
make
cp FastCA ../../
1.11.1.5. VRP_SISRs
VRP_SISRs can be recompiled as follows in the
Examples/Resources/CVRP/Solvers/VRP_SISRs/
directory:
unzip src.zip
cd src/
make
cp VRP_SISRs ../