Skip to content

Exp IBP

Exp-IBP wrapper using the common expressive IBP training and HPO infrastructure.

ExpIBPModelWrapper

Bases: ExpressiveIBPModelWrapper

Wrap a classifier for Exp-IBP training using a geometric combination of batch-mean adversarial and IBP losses.

The shared expressive-loss implementation handles attacks, bound propagation and schedules, while this subclass selects the loss combination. Alpha weights the IBP component. Evaluation and hyperparameter optimisation use the common wrapper interface.

See De Palma et al. (2024): https://arxiv.org/abs/2305.13991.

Source code in CTRAIN/model_wrappers/certified/exp_ibp_model_wrapper.py 
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
class ExpIBPModelWrapper(ExpressiveIBPModelWrapper):
    """Wrap a classifier for Exp-IBP training using a geometric combination of batch-mean
    adversarial and IBP losses.

    The shared expressive-loss implementation handles attacks, bound propagation
    and schedules, while this subclass selects the loss combination. Alpha
    weights the IBP component. Evaluation and hyperparameter optimisation use
    the common wrapper interface.

    See De Palma et al. (2024): https://arxiv.org/abs/2305.13991.
    """
    _loss_type = 'exp_ibp'
    _alpha_name = 'exp_ibp_alpha'
    _loss_func = staticmethod(get_exp_ibp_loss)

    def __init__(self, model, input_shape, eps, num_epochs, train_eps_factor=1,
                 optimizer_func=torch.optim.Adam, lr=0.0005, warm_up_epochs=1, ramp_up_epochs=70,
                 lr_scheduler_func=torch.optim.lr_scheduler.MultiStepLR,
                 lr_decay_kwargs=dict(milestones=(80, 90)), gradient_clip=10, l1_reg_weight=0.000001,
                 shi_reg_weight=.5, shi_reg_decay=True, pgd_steps=1, pgd_alpha=10, pgd_restarts=1,
                 pgd_early_stopping=False, pgd_alpha_decay_factor=.1, pgd_decay_milestones=(),
                 pgd_eps_factor=1, exp_ibp_alpha=.5, checkpoint_save_path=None, checkpoint_save_interval=10,
                 bound_opts=dict(conv_mode='patches', relu='adaptive'), device=torch.device('cuda'),
                 population_bn=False, relu_upper_retention=1.0):
        """Configure the Exp-IBP geometric mixture and shared expressive training options.

        Args:
            model (torch.nn.Module): PyTorch model to train or evaluate.
            input_shape (tuple[int, ...]): Shape of a model input, excluding the batch dimension unless supplied
                explicitly.
            eps (float or torch.Tensor): Target perturbation radius in raw input units, before loader normalisation.
            num_epochs (int): Total training budget, including warm-up and ramp-up.
            train_eps_factor (float): Factor applied to the training radius; the nominal evaluation
                radius remains eps.
            optimizer_func (callable): Optimiser constructor accepting the model parameters and a learning
                rate.
            lr (float): Learning rate passed to the optimiser.
            warm_up_epochs (int): Initial epochs using only the natural task loss.
            ramp_up_epochs (int): Epochs over which the perturbation radius increases to its training
                target.
            lr_scheduler_func (callable): Constructor for the learning-rate scheduler.
            lr_decay_kwargs (dict): Scheduler arguments, such as milestones and gamma for MultiStepLR.
            gradient_clip (float): Maximum parameter-gradient norm. None disables clipping where
                supported.
            l1_reg_weight (float): Weight of the L1 parameter regulariser.
            shi_reg_weight (float): Weight of the interval-stability regulariser proposed by Shi et al.
                (2021).
            shi_reg_decay (bool): Whether to reduce the Shi regulariser as the perturbation radius
                increases.
            pgd_steps (int): Number of projected gradient updates in the training attack.
            pgd_alpha (float): PGD step-size factor relative to the search-region radius.
            pgd_restarts (int): Number of random starts for the training attack.
            pgd_early_stopping (bool): Whether to stop searching an input once a counterexample has
                been found.
            pgd_alpha_decay_factor (float): Multiplier applied to the PGD step size at each decay
                milestone.
            pgd_decay_milestones (Sequence[int]): Attack iterations at which to reduce the PGD step size.
            pgd_eps_factor (float): Positive factor scaling the PGD search range, independently of the
                nominal bound region.
            exp_ibp_alpha (float): Weight of the IBP loss in the Exp-IBP geometric mixture.
            checkpoint_save_path (str or pathlib.Path, optional): Optional directory for epoch checkpoints containing the
                training trajectory.
            checkpoint_save_interval (int): Save an epoch checkpoint at this interval and at the end
                of the training call.
            bound_opts (dict): Bound-propagation options passed to auto_LiRPA.
            device (str or torch.device): Device on which to run the model and tensor computations.
            population_bn (bool): Whether to recalibrate BatchNorm on training data after each epoch,
                before model selection and saving.
            relu_upper_retention (float): Fraction of unstable ReLU upper bounds retained during
                training, in (0, 1]. Values below one give an unsound training
                surrogate.
        """
        super().__init__(
            model, input_shape, eps, num_epochs, train_eps_factor, optimizer_func, lr,
            warm_up_epochs, ramp_up_epochs, lr_scheduler_func, lr_decay_kwargs, gradient_clip,
            l1_reg_weight, shi_reg_weight, shi_reg_decay, pgd_steps, pgd_alpha, pgd_restarts,
            pgd_early_stopping, pgd_alpha_decay_factor, pgd_decay_milestones, pgd_eps_factor,
            exp_ibp_alpha, checkpoint_save_path, checkpoint_save_interval, bound_opts, device,
            population_bn, relu_upper_retention)

__init__(model, input_shape, eps, num_epochs, train_eps_factor=1, optimizer_func=torch.optim.Adam, lr=0.0005, warm_up_epochs=1, ramp_up_epochs=70, lr_scheduler_func=torch.optim.lr_scheduler.MultiStepLR, lr_decay_kwargs=dict(milestones=(80, 90)), gradient_clip=10, l1_reg_weight=1e-06, shi_reg_weight=0.5, shi_reg_decay=True, pgd_steps=1, pgd_alpha=10, pgd_restarts=1, pgd_early_stopping=False, pgd_alpha_decay_factor=0.1, pgd_decay_milestones=(), pgd_eps_factor=1, exp_ibp_alpha=0.5, checkpoint_save_path=None, checkpoint_save_interval=10, bound_opts=dict(conv_mode='patches', relu='adaptive'), device=torch.device('cuda'), population_bn=False, relu_upper_retention=1.0)

Configure the Exp-IBP geometric mixture and shared expressive training options.

Parameters:

Name Type Description Default
model Module

PyTorch model to train or evaluate.

 required
input_shape tuple[int, ...]

Shape of a model input, excluding the batch dimension unless supplied explicitly.

 required
eps float or Tensor

Target perturbation radius in raw input units, before loader normalisation.

 required
num_epochs int

Total training budget, including warm-up and ramp-up.

 required
train_eps_factor float

Factor applied to the training radius; the nominal evaluation radius remains eps.

 1
optimizer_func callable

Optimiser constructor accepting the model parameters and a learning rate.

 Adam
lr float

Learning rate passed to the optimiser.

 0.0005
warm_up_epochs int

Initial epochs using only the natural task loss.

 1
ramp_up_epochs int

Epochs over which the perturbation radius increases to its training target.

 70
lr_scheduler_func callable

Constructor for the learning-rate scheduler.

 MultiStepLR
lr_decay_kwargs dict

Scheduler arguments, such as milestones and gamma for MultiStepLR.

 dict(milestones=(80, 90))
gradient_clip float

Maximum parameter-gradient norm. None disables clipping where supported.

 10
l1_reg_weight float

Weight of the L1 parameter regulariser.

 1e-06
shi_reg_weight float

Weight of the interval-stability regulariser proposed by Shi et al. (2021).

 0.5
shi_reg_decay bool

Whether to reduce the Shi regulariser as the perturbation radius increases.

 True
pgd_steps int

Number of projected gradient updates in the training attack.

 1
pgd_alpha float

PGD step-size factor relative to the search-region radius.

 10
pgd_restarts int

Number of random starts for the training attack.

 1
pgd_early_stopping bool

Whether to stop searching an input once a counterexample has been found.

 False
pgd_alpha_decay_factor float

Multiplier applied to the PGD step size at each decay milestone.

 0.1
pgd_decay_milestones Sequence[int]

Attack iterations at which to reduce the PGD step size.

 ()
pgd_eps_factor float

Positive factor scaling the PGD search range, independently of the nominal bound region.

 1
exp_ibp_alpha float

Weight of the IBP loss in the Exp-IBP geometric mixture.

 0.5
checkpoint_save_path str or Path

Optional directory for epoch checkpoints containing the training trajectory.

 None
checkpoint_save_interval int

Save an epoch checkpoint at this interval and at the end of the training call.

 10
bound_opts dict

Bound-propagation options passed to auto_LiRPA.

 dict(conv_mode='patches', relu='adaptive')
device str or device

Device on which to run the model and tensor computations.

 device('cuda')
population_bn bool

Whether to recalibrate BatchNorm on training data after each epoch, before model selection and saving.

 False
relu_upper_retention float

Fraction of unstable ReLU upper bounds retained during training, in (0, 1]. Values below one give an unsound training surrogate.

 1.0
Source code in CTRAIN/model_wrappers/certified/exp_ibp_model_wrapper.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
def __init__(self, model, input_shape, eps, num_epochs, train_eps_factor=1,
             optimizer_func=torch.optim.Adam, lr=0.0005, warm_up_epochs=1, ramp_up_epochs=70,
             lr_scheduler_func=torch.optim.lr_scheduler.MultiStepLR,
             lr_decay_kwargs=dict(milestones=(80, 90)), gradient_clip=10, l1_reg_weight=0.000001,
             shi_reg_weight=.5, shi_reg_decay=True, pgd_steps=1, pgd_alpha=10, pgd_restarts=1,
             pgd_early_stopping=False, pgd_alpha_decay_factor=.1, pgd_decay_milestones=(),
             pgd_eps_factor=1, exp_ibp_alpha=.5, checkpoint_save_path=None, checkpoint_save_interval=10,
             bound_opts=dict(conv_mode='patches', relu='adaptive'), device=torch.device('cuda'),
             population_bn=False, relu_upper_retention=1.0):
    """Configure the Exp-IBP geometric mixture and shared expressive training options.

    Args:
        model (torch.nn.Module): PyTorch model to train or evaluate.
        input_shape (tuple[int, ...]): Shape of a model input, excluding the batch dimension unless supplied
            explicitly.
        eps (float or torch.Tensor): Target perturbation radius in raw input units, before loader normalisation.
        num_epochs (int): Total training budget, including warm-up and ramp-up.
        train_eps_factor (float): Factor applied to the training radius; the nominal evaluation
            radius remains eps.
        optimizer_func (callable): Optimiser constructor accepting the model parameters and a learning
            rate.
        lr (float): Learning rate passed to the optimiser.
        warm_up_epochs (int): Initial epochs using only the natural task loss.
        ramp_up_epochs (int): Epochs over which the perturbation radius increases to its training
            target.
        lr_scheduler_func (callable): Constructor for the learning-rate scheduler.
        lr_decay_kwargs (dict): Scheduler arguments, such as milestones and gamma for MultiStepLR.
        gradient_clip (float): Maximum parameter-gradient norm. None disables clipping where
            supported.
        l1_reg_weight (float): Weight of the L1 parameter regulariser.
        shi_reg_weight (float): Weight of the interval-stability regulariser proposed by Shi et al.
            (2021).
        shi_reg_decay (bool): Whether to reduce the Shi regulariser as the perturbation radius
            increases.
        pgd_steps (int): Number of projected gradient updates in the training attack.
        pgd_alpha (float): PGD step-size factor relative to the search-region radius.
        pgd_restarts (int): Number of random starts for the training attack.
        pgd_early_stopping (bool): Whether to stop searching an input once a counterexample has
            been found.
        pgd_alpha_decay_factor (float): Multiplier applied to the PGD step size at each decay
            milestone.
        pgd_decay_milestones (Sequence[int]): Attack iterations at which to reduce the PGD step size.
        pgd_eps_factor (float): Positive factor scaling the PGD search range, independently of the
            nominal bound region.
        exp_ibp_alpha (float): Weight of the IBP loss in the Exp-IBP geometric mixture.
        checkpoint_save_path (str or pathlib.Path, optional): Optional directory for epoch checkpoints containing the
            training trajectory.
        checkpoint_save_interval (int): Save an epoch checkpoint at this interval and at the end
            of the training call.
        bound_opts (dict): Bound-propagation options passed to auto_LiRPA.
        device (str or torch.device): Device on which to run the model and tensor computations.
        population_bn (bool): Whether to recalibrate BatchNorm on training data after each epoch,
            before model selection and saving.
        relu_upper_retention (float): Fraction of unstable ReLU upper bounds retained during
            training, in (0, 1]. Values below one give an unsound training
            surrogate.
    """
    super().__init__(
        model, input_shape, eps, num_epochs, train_eps_factor, optimizer_func, lr,
        warm_up_epochs, ramp_up_epochs, lr_scheduler_func, lr_decay_kwargs, gradient_clip,
        l1_reg_weight, shi_reg_weight, shi_reg_decay, pgd_steps, pgd_alpha, pgd_restarts,
        pgd_early_stopping, pgd_alpha_decay_factor, pgd_decay_milestones, pgd_eps_factor,
        exp_ibp_alpha, checkpoint_save_path, checkpoint_save_interval, bound_opts, device,
        population_bn, relu_upper_retention)