Commands

Executing commands

Executing commands in Sparkle is as simple as running them terminal for example:

sparkle initialise

Do note that when running on a cluster additional arguments may be needed, for instance under the Slurm workload manager. All CLI entries are placed in the sparkle package sparkle/CLI/$COMMANDNAME$.py the above command would change to something like:

srun -N1 -n1 -c1 path/to/package/sparkle/CLI/initialise.py

In the Examples/ directory a number of common command sequences are given. For instance, for configuration with specified training and testing sets see e.g. Examples/configuration_runtime.sh for an example of a sequence of commands to execute. Note that some command run in the background and need time to complete before the next command is executed. To see whether a command is still running the wait command can be used.

In the Output/ directory paths to generated scripts and logs are gathered per executed command.

Note

When typing a sparkle command name that consists of multiple words, both spaces and underscores are accepted as seperators.

List of Commands

Currently the commands below are available in Sparkle (listed alphabetically). Every command can be called with the –help option to get a description of the required arguments and other options.

Note

Arguments in [square brackets] are optional, arguments without brackets are mandatory. Input in <chevrons> indicate required text input, {curly brackets} indicate a set of inputs to choose from.

about

usage: about [-h]

-h, --help: show this help message and exit

add_feature_extractor

Add a feature extractor to the platform.

usage: add_feature_extractor [-h] [--run-extractor-now] [--nickname NICKNAME]
                             extractor-path

extractor-path: path or nickname of the feature extractor

-h, --help: show this help message and exit

--run-extractor-now: immediately run the feature extractor(s) on all the instances

--nickname <nickname>: set a nickname for the feature extractor

add_instances

Add instances to the platform.

usage: add_instances [-h] [--run-extractor-now] [--run-solver-now]
                     [--nickname NICKNAME]
                     [--run-on {Runner.LOCAL,Runner.SLURM}]
                     instances-path

instances-path: path to the instance set

-h, --help: show this help message and exit

--run-extractor-now: immediately run the feature extractor(s) on all the instances

--run-solver-now: immediately run the solver(s) on all instances

--nickname <nickname>: set a nickname for the instance set

--run-on: On which computer or cluster environment to execute the calculation.

add_solver

Add a solver to the Sparkle platform.

usage: add_solver [-h] [--deterministic] [--run-solver-now]
                  [--nickname NICKNAME] [--run-on {Runner.LOCAL,Runner.SLURM}]
                  [--skip-checks]
                  solver-path

solver-path: path to the solver

-h, --help: show this help message and exit

--deterministic: Flag indicating the solver is deterministic

--run-solver-now: immediately run the solver(s) on all instances

--nickname <nickname>: set a nickname for the solver

--run-on: On which computer or cluster environment to execute the calculation.

--skip-checks: Checks the solver’s functionality by testing it on an instance and the pcs file, when applicable.

cancel

Command to cancel running jobs.

usage: cancel [-h] [--job-ids JOB_IDS [JOB_IDS ...]] [--all]

-h, --help: show this help message and exit

--job-ids <job_ids>: job ID(s) to use for the command

--all: use all known job ID(s) for the command

cleanup

Command to clean files from the platform.

usage: cleanup [-h] [--all] [--remove]

-h, --help: show this help message and exit

--all: clean all output files

--remove: remove all files in the platform, including user data such as InstanceSets and Solvers

compute_features

Sparkle command to Compute features for instances using added extractors and instances.

usage: compute_features [-h] [--recompute] [--settings-file SETTINGS_FILE]
                        [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--recompute: Re-run feature extractor for instances with previously computed features

--settings-file: Specify the settings file to use in case you want to use one other than the default

--run-on: On which computer or cluster environment to execute the calculation.

compute_marginal_contribution

Command to compute the marginal contribution of solvers to the portfolio.

usage: compute_marginal_contribution [-h] [--perfect] [--actual]
                                     [--objectives OBJECTIVES]
                                     [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--perfect: compute the marginal contribution for the perfect selector

--actual: compute the marginal contribution for the actual selector

--objectives <objectives>: the comma seperated objective(s) to use.

--settings-file: Specify the settings file to use in case you want to use one other than the default

configure_solver

Configure a solver in the platform.

usage: configure_solver [-h] [--configurator CONFIGURATOR] --solver SOLVER
                        --instance-set-train INSTANCE_SET_TRAIN
                        [--instance-set-test INSTANCE_SET_TEST]
                        [--objectives OBJECTIVES]
                        [--target-cutoff-time TARGET_CUTOFF_TIME]
                        [--solver-calls SOLVER_CALLS]
                        [--number-of-runs NUMBER_OF_RUNS]
                        [--settings-file SETTINGS_FILE] [--use-features]
                        [--validate] [--ablation]
                        [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--configurator <configurator>: name of the configurator

--solver <solver>: path to solver

--instance-set-train <instance_set_train>: path to training instance set

--instance-set-test <instance_set_test>: path to test instance set (only for validating)

--objectives <objectives>: the comma seperated objective(s) to use.

--target-cutoff-time <target_cutoff_time>: cutoff time per target algorithm run in seconds

--solver-calls <solver_calls>: number of solver calls to execute

--number-of-runs <number_of_runs>: number of configuration runs to execute

--settings-file: Specify the settings file to use in case you want to use one other than the default

--use-features: use the training set’s features for configuration

--validate: validate after configuration

--ablation: run ablation after configuration

--run-on: On which computer or cluster environment to execute the calculation.

Note that the test instance set is only used if the --ablation or --validation flags are given

construct_portfolio_selector

Command to construct a portfolio selector over all known features solver performances.

usage: construct_portfolio_selector [-h] [--recompute-portfolio-selector]
                                    [--selector-timeout SELECTOR_TIMEOUT]
                                    [--objectives OBJECTIVES]
                                    [--solver-ablation]
                                    [--run-on {Runner.LOCAL,Runner.SLURM}]
                                    [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--recompute-portfolio-selector: force the construction of a new portfolio selector even when it already exists for the current feature and performance data. NOTE: This will also result in the computation of the marginal contributions of solvers to the new portfolio selector.

--selector-timeout <selector_timeout>: Cuttoff time (in seconds) for the algorithmselector construction

--objectives <objectives>: the comma seperated objective(s) to use.

--solver-ablation: construct a selector for each solver ablation combination

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

generate_report

Without any arguments a report for the most recent algorithm selection or algorithm configuration procedure is generated.

usage: generate_report [-h] [--solver SOLVER]
                       [--instance-set-train INSTANCE_SET_TRAIN]
                       [--instance-set-test INSTANCE_SET_TEST]
                       [--no-ablation [FLAG_ABLATION]] [--selection]
                       [--test-case-directory TEST_CASE_DIRECTORY]
                       [--objectives OBJECTIVES]
                       [--settings-file SETTINGS_FILE] [--only-json ONLY_JSON]

-h, --help: show this help message and exit

--solver <solver>: path to solver for an algorithm configuration report

--instance-set-train <instance_set_train>: path to training instance set included in Sparkle for an algorithm configuration report

--instance-set-test <instance_set_test>: path to testing instance set included in Sparkle for an algorithm configuration report

--no-ablation <flag_ablation>: turn off reporting on ablation for an algorithm configuration report

--selection: set to generate a normal selection report

--test-case-directory <test_case_directory>: Path to test case directory of an instance set for a selection report

--objectives <objectives>: the comma seperated objective(s) to use.

--settings-file: Specify the settings file to use in case you want to use one other than the default

--only-json <only_json>: if set to True, only generate machine readable output

Note that if a test instance set is given, the training instance set must also be given.

initialise

Initialise the Sparkle platform in the current directory.

usage: initialise [-h] [--download-examples | --no-download-examples]

-h, --help: show this help message and exit

--download-examples, --no-download-examples: Download the Examples into the directory. (default: False)

load_snapshot

Load a platform from a zip file.

usage: load_snapshot [-h] snapshot-file-path

snapshot-file-path: path to the snapshot file

-h, --help: show this help message and exit

remove_feature_extractor

Remove a feature extractor from the platform.

usage: remove_feature_extractor [-h] extractor-path

extractor-path: path or nickname of the feature extractor

-h, --help: show this help message and exit

remove_instances

Remove instances from the platform.

usage: remove_instances [-h] instances-path

instances-path: path to or nickname of the instance set

-h, --help: show this help message and exit

remove_solver

Remove a solver from the platform.

usage: remove_solver [-h] solver

solver: name, path to or nickname of the solver

-h, --help: show this help message and exit

run_ablation

Runs parameter importance between the default and configured parameters with ablation. This command requires a finished configuration for the solver instance pair.

usage: run_ablation [-h] [--solver SOLVER]
                    [--instance-set-train INSTANCE_SET_TRAIN]
                    [--instance-set-test INSTANCE_SET_TEST]
                    [--objectives OBJECTIVES]
                    [--target-cutoff-time TARGET_CUTOFF_TIME]
                    [--wallclock-time WALLCLOCK_TIME]
                    [--number-of-runs NUMBER_OF_RUNS] [--racing RACING]
                    [--settings-file SETTINGS_FILE]
                    [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--solver <solver>: path to solver

--instance-set-train <instance_set_train>: path to training instance set

--instance-set-test <instance_set_test>: path to test instance set

--objectives <objectives>: the comma seperated objective(s) to use.

--target-cutoff-time: cutoff time per target algorithm run in seconds

--wallclock-time <wallclock_time>: configuration budget per configurator run in seconds (wallclock)

--number-of-runs: Number of configuration runs to execute

--racing: Performs abaltion analysis with racing

--settings-file: Specify the settings file to use in case you want to use one other than the default

--run-on: On which computer or cluster environment to execute the calculation.

Note that if no test instance set is given, the validation is performed on the training set.

run_configured_solver

Command to run a configured solver on an instance (set).

usage: run_configured_solver [-h] [--settings-file SETTINGS_FILE]
                             [--objectives OBJECTIVES]
                             [--run-on {Runner.LOCAL,Runner.SLURM}]
                             instance_path

instance_path: Path to an instance (set)

-h, --help: show this help message and exit

--settings-file: Specify the settings file to use in case you want to use one other than the default

--objectives <objectives>: the comma seperated objective(s) to use.

--run-on: On which computer or cluster environment to execute the calculation.

run_parallel_portfolio

Run a portfolio of solvers on an instance set in parallel.

usage: run_parallel_portfolio [-h] --instance-path INSTANCE_PATH
                              [--portfolio-name PORTFOLIO_NAME]
                              [--solvers SOLVERS [SOLVERS ...]]
                              [--objectives OBJECTIVES]
                              [--cutoff-time CUTOFF_TIME]
                              [--solver-seeds SOLVER_SEEDS]
                              [--run-on {Runner.LOCAL,Runner.SLURM}]
                              [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--instance-path <instance_path>: Path to an instance (set)

--portfolio-name <portfolio_name>: Specify a name of the portfolio. If none is given, one will be generated.

--solvers <solvers>: Specify the list of solvers to be used. If not specifed, all solvers known in Sparkle will be used.

--objectives <objectives>: the comma seperated objective(s) to use.

--cutoff-time <cutoff_time>: The duration the portfolio will run before the solvers within the portfolio will be stopped (default: 60)

--solver-seeds <solver_seeds>: number of random seeds per solver to execute

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

run_portfolio_selector

Run a portfolio selector on instance (set), determine which solver is most likely to perform well and run it on the instance (set).

usage: run_portfolio_selector [-h] [--run-on {Runner.LOCAL,Runner.SLURM}]
                              [--settings-file SETTINGS_FILE]
                              [--objectives OBJECTIVES]
                              instance_path

instance_path: Path to an instance (set)

-h, --help: show this help message and exit

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

--objectives <objectives>: the comma seperated objective(s) to use.

run_solvers

Run all solvers on all instances to get their performance data.

usage: run_solvers [-h] [--recompute] [--objectives OBJECTIVES]
                   [--target-cutoff-time TARGET_CUTOFF_TIME]
                   [--also-construct-selector-and-report]
                   [--run-on {Runner.LOCAL,Runner.SLURM}]
                   [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--recompute: recompute the performance of all solvers on all instances

--objectives <objectives>: the comma seperated objective(s) to use.

--target-cutoff-time <target_cutoff_time>: cutoff time per target algorithm run in seconds

--also-construct-selector-and-report: after running the solvers also construct the selector and generate the report

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

save_snapshot

Save the current platform in a .zip file.

usage: save_snapshot [-h] [--name NAME]

-h, --help: show this help message and exit

--name <name>: name of the snapshot

Can be loaded later with the load snapshot command.

status

Display the status of the platform.

usage: status [-h] [--verbose]

-h, --help: show this help message and exit

--verbose, -v: output status in verbose mode

validate_configured_vs_default

Test the performance of the configured solver and the default solver by doing validation experiments on the training and test sets.

usage: validate_configured_vs_default [-h] --solver SOLVER
                                      --instance-set-train INSTANCE_SET_TRAIN
                                      [--instance-set-test INSTANCE_SET_TEST]
                                      [--configurator CONFIGURATOR]
                                      [--objectives OBJECTIVES]
                                      [--target-cutoff-time TARGET_CUTOFF_TIME]
                                      [--settings-file SETTINGS_FILE]
                                      [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--solver <solver>: path to solver

--instance-set-train <instance_set_train>: path to training instance set

--instance-set-test <instance_set_test>: path to test instance set (only for validating)

--configurator <configurator>: name of the configurator

--objectives <objectives>: the comma seperated objective(s) to use.

--target-cutoff-time: cutoff time per target algorithm run in seconds

--settings-file: Specify the settings file to use in case you want to use one other than the default

--run-on: On which computer or cluster environment to execute the calculation.

wait

Wait for async jobs to finish. Gives periodic updates in table format about each job.

usage: wait [-h] [--job-ids JOB_IDS [JOB_IDS ...]]

-h, --help: show this help message and exit

--job-ids <job_ids>: job ID(s) to use for the command