Commands

Executing commands

Executing commands in Sparkle is as simple as running them terminal for example:

sparkle initialise

Do note that when running on a cluster additional arguments may be needed, for instance under the Slurm workload manager. All CLI entries are placed in the sparkle package sparkle/CLI/$COMMANDNAME$.py the above command would change to something like:

srun -N1 -n1 -c1 path/to/package/sparkle/CLI/initialise.py

In the Examples/ directory a number of common command sequences are given. For instance, for configuration with specified training and testing sets see e.g. Examples/configuration_runtime.sh for an example of a sequence of commands to execute. Note that some command run in the background and need time to complete before the next command is executed. To see whether a command is still running the jobs command can be used.

In the Output/ directory paths to generated scripts and logs are gathered per executed command.

Note

When typing a sparkle command name that consists of multiple words, both spaces and underscores are accepted as seperators.

List of Commands

Currently the commands below are available in Sparkle (listed alphabetically). Every command can be called with the –help option to get a description of the required arguments and other options.

Note

Arguments in [square brackets] are optional, arguments without brackets are mandatory. Input in <chevrons> indicate required text input, {curly brackets} indicate a set of inputs to choose from.

about

usage: about [-h]

-h, --help: show this help message and exit

add_feature_extractor

Add a feature extractor to the platform.

usage: add_feature_extractor [-h] [--nickname NICKNAME] [--no-copy]
                             extractor-path

extractor-path: path or nickname of the feature extractor

-h, --help: show this help message and exit

--nickname <nickname>: set a nickname for the feature extractor

--no-copy: do not copy the source directory to the platform directory, but create a symbolic link instead

add_instances

Add instances to the platform.

usage: add_instances [-h] [--nickname NICKNAME] [--no-copy] instances-path

instances-path: path to the instance set

-h, --help: show this help message and exit

--nickname <nickname>: set a nickname for the instance set

--no-copy: do not copy the source directory to the platform directory, but create a symbolic link instead

add_solver

Add a solver to the Sparkle platform.

usage: add_solver [-h] [--deterministic]
                  [--solution-verifier SOLUTION_VERIFIER]
                  [--nickname NICKNAME] [--skip-checks] [--no-copy]
                  solver-path

solver-path: path to the solver

-h, --help: show this help message and exit

--deterministic: Flag indicating the solver is deterministic

--solution-verifier <solution_verifier>: the class name of the solution verifier to use for the Solver. If it is a Path, will resolve as a SolutionFileVerifier class with the specified Path instead.

--nickname <nickname>: set a nickname for the solver

--skip-checks: Checks the solver’s functionality by testing it on an instance and the pcs file, when applicable.

--no-copy: do not copy the source directory to the platform directory, but create a symbolic link instead

cleanup

Command to clean files from the platform.

usage: cleanup [-h] [--all] [--remove] [--performance-data]

-h, --help: show this help message and exit

--all: clean all output files

--remove: remove all files in the platform, including user data such as InstanceSets and Solvers

--performance-data: clean performance data from empty lines

compute_features

Sparkle command to Compute features for instances using added extractors and instances.

usage: compute_features [-h] [--recompute] [--settings-file SETTINGS_FILE]
                        [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--recompute: Re-run feature extractor for instances with previously computed features

--settings-file: Specify the settings file to use in case you want to use one other than the default

--run-on: On which computer or cluster environment to execute the calculation.

compute_marginal_contribution

Command to compute the marginal contribution of solvers to the portfolio.

usage: compute_marginal_contribution [-h] [--perfect] [--actual]
                                     [--objectives OBJECTIVES]
                                     [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--perfect: compute the marginal contribution for the perfect selector

--actual: compute the marginal contribution for the actual selector

--objectives <objectives>: the comma seperated objective(s) to use.

--settings-file: Specify the settings file to use in case you want to use one other than the default

configure_solver

Configure a solver in the platform.

usage: configure_solver [-h] [--configurator CONFIGURATOR] --solver SOLVER
                        --instance-set-train INSTANCE_SET_TRAIN
                        [--instance-set-test INSTANCE_SET_TEST]
                        [--test-set-run-all-configurations]
                        [--objectives OBJECTIVES]
                        [--target-cutoff-time TARGET_CUTOFF_TIME]
                        [--solver-calls SOLVER_CALLS]
                        [--number-of-runs NUMBER_OF_RUNS]
                        [--settings-file SETTINGS_FILE] [--use-features]
                        [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--configurator <configurator>: name of the configurator

--solver <solver>: path to solver

--instance-set-train <instance_set_train>: path to training instance set

--instance-set-test <instance_set_test>: path to test instance set (only for validating)

--test-set-run-all-configurations: run all found configurations on the test set

--objectives <objectives>: the comma seperated objective(s) to use.

--target-cutoff-time, --solver-cutoff-time: cutoff time per Solver run in seconds

--solver-calls <solver_calls>: number of solver calls to execute

--number-of-runs <number_of_runs>: number of configuration runs to execute

--settings-file: Specify the settings file to use in case you want to use one other than the default

--use-features: use the training set’s features for configuration

--run-on: On which computer or cluster environment to execute the calculation.

Note that the test instance set is only used if the --ablation or --validation flags are given

construct_portfolio_selector

Command to construct a portfolio selector over all known features solver performances.

usage: construct_portfolio_selector [-h] [--recompute-portfolio-selector]
                                    [--selector-timeout SELECTOR_TIMEOUT]
                                    [--objective OBJECTIVE]
                                    [--solver-ablation]
                                    [--run-on {Runner.LOCAL,Runner.SLURM}]
                                    [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--recompute-portfolio-selector: force the construction of a new portfolio selector even when it already exists for the current feature and performance data. NOTE: This will also result in the computation of the marginal contributions of solvers to the new portfolio selector.

--selector-timeout <selector_timeout>: Cuttoff time (in seconds) for the algorithmselector construction

--objective <objective>: the objective to use.

--solver-ablation: construct a selector for each solver ablation combination

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

generate_report

Without any arguments a report for the most recent algorithm selection or algorithm configuration procedure is generated.

usage: generate_report [-h] [--solver SOLVER]
                       [--instance-set-train INSTANCE_SET_TRAIN]
                       [--instance-set-test INSTANCE_SET_TEST]
                       [--no-ablation [FLAG_ABLATION]] [--selection]
                       [--test-case-directory TEST_CASE_DIRECTORY]
                       [--objectives OBJECTIVES]
                       [--settings-file SETTINGS_FILE] [--only-json ONLY_JSON]

-h, --help: show this help message and exit

--solver <solver>: path to solver for an algorithm configuration report

--instance-set-train <instance_set_train>: path to training instance set included in Sparkle for an algorithm configuration report

--instance-set-test <instance_set_test>: path to testing instance set included in Sparkle for an algorithm configuration report

--no-ablation <flag_ablation>: turn off reporting on ablation for an algorithm configuration report

--selection: set to generate a normal selection report

--test-case-directory <test_case_directory>: Path to test case directory of an instance set for a selection report

--objectives <objectives>: the comma seperated objective(s) to use.

--settings-file: Specify the settings file to use in case you want to use one other than the default

--only-json <only_json>: if set to True, only generate machine readable output

Note that if a test instance set is given, the training instance set must also be given.

initialise

Initialise the Sparkle platform in the current directory.

usage: initialise [-h] [--download-examples] [--no-save] [--rebuild-runsolver]

-h, --help: show this help message and exit

--download-examples: Download the Examples into the directory.

--no-save: do not save the platform upon re-initialisation.

--rebuild-runsolver: Clean the RunSolver executable and rebuild it.

jobs

Command to interact with async jobs.

usage: jobs [-h] [--cancel] [--job-ids JOB_IDS [JOB_IDS ...]] [--all]

-h, --help: show this help message and exit

--cancel: cancel the job(s) with the given ID(s)

--job-ids <job_ids>: job ID(s) to use for the command

--all: use all known job ID(s) for the command

load_snapshot

Load a platform from a zip file.

usage: load_snapshot [-h] snapshot-file-path

snapshot-file-path: path to the snapshot file

-h, --help: show this help message and exit

remove_feature_extractor

Remove a feature extractor from the platform.

usage: remove_feature_extractor [-h] extractor-path

extractor-path: path or nickname of the feature extractor

-h, --help: show this help message and exit

remove_instances

Remove instances from the platform.

usage: remove_instances [-h] instances-path

instances-path: path to or nickname of the instance set

-h, --help: show this help message and exit

remove_solver

Remove a solver from the platform.

usage: remove_solver [-h] solver

solver: name, path to or nickname of the solver

-h, --help: show this help message and exit

run_ablation

Runs parameter importance between the default and configured parameters with ablation. This command requires a finished configuration for the solver instance pair.

usage: run_ablation [-h] [--solver SOLVER]
                    [--instance-set-train INSTANCE_SET_TRAIN]
                    [--instance-set-test INSTANCE_SET_TEST]
                    [--objectives OBJECTIVES]
                    [--target-cutoff-time TARGET_CUTOFF_TIME]
                    [--wallclock-time WALLCLOCK_TIME]
                    [--number-of-runs NUMBER_OF_RUNS] [--racing RACING]
                    [--settings-file SETTINGS_FILE]
                    [--run-on {Runner.LOCAL,Runner.SLURM}]

-h, --help: show this help message and exit

--solver <solver>: path to solver

--instance-set-train <instance_set_train>: path to training instance set

--instance-set-test <instance_set_test>: path to test instance set

--objectives <objectives>: the comma seperated objective(s) to use.

--target-cutoff-time, --solver-cutoff-time: cutoff time per Solver run in seconds

--wallclock-time <wallclock_time>: configuration budget per configurator run in seconds (wallclock)

--number-of-runs: Number of configuration runs to execute

--racing: Performs abaltion analysis with racing

--settings-file: Specify the settings file to use in case you want to use one other than the default

--run-on: On which computer or cluster environment to execute the calculation.

Note that if no test instance set is given, the validation is performed on the training set.

run_parallel_portfolio

Run a portfolio of solvers on an instance set in parallel.

usage: run_parallel_portfolio [-h]
                              [--instance-path INSTANCE_PATH [INSTANCE_PATH ...]]
                              [--portfolio-name PORTFOLIO_NAME]
                              [--solvers SOLVERS [SOLVERS ...]]
                              [--objectives OBJECTIVES]
                              [--cutoff-time CUTOFF_TIME]
                              [--solver-seeds SOLVER_SEEDS]
                              [--run-on {Runner.LOCAL,Runner.SLURM}]
                              [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--instance-path <instance_path>, --instance-set-path <instance_path>, --instance <instance_path>, --instance-set <instance_path>, --instances <instance_path>, --instance-sets <instance_path>, --instance-paths <instance_path>, --instance-set-paths <instance_path>: Path to an instance (set)

--portfolio-name <portfolio_name>: Specify a name of the portfolio. If none is given, one will be generated.

--solvers <solvers>, --solver-paths <solvers>, --solver <solvers>, --solver-path <solvers>: Specify the list of solvers to be used. If not specifed, all solvers known in Sparkle will be used.

--objectives <objectives>: the comma seperated objective(s) to use.

--cutoff-time <cutoff_time>: The duration the portfolio will run before the solvers within the portfolio will be stopped (default: 60)

--solver-seeds <solver_seeds>: number of random seeds per solver to execute

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

run_portfolio_selector

Run a portfolio selector on instance (set), determine which solver is most likely to perform well and run it on the instance (set).

usage: run_portfolio_selector [-h] [--run-on {Runner.LOCAL,Runner.SLURM}]
                              [--settings-file SETTINGS_FILE]
                              instance_path

instance_path: Path to an instance (set)

-h, --help: show this help message and exit

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

run_solvers

Run solvers on instances to get their performance data.

usage: run_solvers [-h] [--solvers SOLVERS [SOLVERS ...]]
                   [--instance-path INSTANCE_PATH [INSTANCE_PATH ...]]
                   [--configuration CONFIGURATION | --best-configuration [BEST_CONFIGURATION]]
                   [--objective OBJECTIVE] [--performance-data-jobs]
                   [--recompute] [--target-cutoff-time TARGET_CUTOFF_TIME]
                   [--run-on {Runner.LOCAL,Runner.SLURM}]
                   [--settings-file SETTINGS_FILE]

-h, --help: show this help message and exit

--solvers <solvers>, --solver-paths <solvers>, --solver <solvers>, --solver-path <solvers>: Specify the list of solvers to be used. If not specifed, all solvers known in Sparkle will be used.

--instance-path <instance_path>, --instance-set-path <instance_path>, --instance <instance_path>, --instance-set <instance_path>, --instances <instance_path>, --instance-sets <instance_path>, --instance-paths <instance_path>, --instance-set-paths <instance_path>: Path to an instance (set)

--configuration <configuration>: The run index of which configuration to use

--best-configuration <best_configuration>: Paths to instance(s) or instanceset(s) over which to determine the best configuration. If empty, all known instances are used.

--objective <objective>: the objective to use.

--performance-data-jobs: compute the remaining jobs in the Performance DataFrame

--recompute: recompute the performance of all solvers on all instances

--target-cutoff-time, --solver-cutoff-time: cutoff time per Solver run in seconds

--run-on: On which computer or cluster environment to execute the calculation.

--settings-file: Specify the settings file to use in case you want to use one other than the default

save_snapshot

Save the current platform in a .zip file.

usage: save_snapshot [-h] [--name NAME]

-h, --help: show this help message and exit

--name <name>: name of the snapshot

Can be loaded later with the load snapshot command.

status

Display the status of the platform.

usage: status [-h] [--verbose]

-h, --help: show this help message and exit

--verbose, -v: output status in verbose mode