Model Evaluation with CTRAIN¶

In this example, we evaluate the MNIST network saved by the certified-training notebook. We assess natural, certified and adversarial accuracy against ℓ∞ perturbations with radius ε = 0.1. Run the training notebook first, or supply a compatible checkpoint in the loading cell.

First, we import PyTorch and the required functions from CTRAIN.

In [ ]:

import torch

from CTRAIN.data_loaders import load_mnist
from CTRAIN.model_definitions import CNN7_Shi
from CTRAIN.model_wrappers import ShiIBPModelWrapper

import torch from CTRAIN.data_loaders import load_mnist from CTRAIN.model_definitions import CNN7_Shi from CTRAIN.model_wrappers import ShiIBPModelWrapper

Now, we load MNIST and recreate the network architecture used for training.

In [ ]:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
input_shape = (1, 28, 28)
train_loader, test_loader = load_mnist(batch_size=128, val_split=False, data_root="./data")
model = CNN7_Shi(in_shape=input_shape, n_classes=10)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") input_shape = (1, 28, 28) train_loader, test_loader = load_mnist(batch_size=128, val_split=False, data_root="./data") model = CNN7_Shi(in_shape=input_shape, n_classes=10)

We wrap the network with the same model wrapper and evaluation radius. The training schedule is included to match the training example; evaluation does not train the model.

In [ ]:

wrapped_model = ShiIBPModelWrapper(
    model=model,
    input_shape=input_shape,
    eps=0.1,
    num_epochs=70,
    warm_up_epochs=1,
    ramp_up_epochs=40,
    lr_decay_kwargs={"milestones": [50, 60], "gamma": 0.2},
    device=device,
)

wrapped_model = ShiIBPModelWrapper( model=model, input_shape=input_shape, eps=0.1, num_epochs=70, warm_up_epochs=1, ramp_up_epochs=40, lr_decay_kwargs={"milestones": [50, 60], "gamma": 0.2}, device=device, )

Next, we load the saved weights and set the model to evaluation mode.

In [ ]:

wrapped_model.load_state_dict(
    torch.load("mnist_ibp.pt", map_location=device, weights_only=True),
)
wrapped_model.eval()

wrapped_model.load_state_dict( torch.load("mnist_ibp.pt", map_location=device, weights_only=True), ) wrapped_model.eval()

Finally, we evaluate the first 1,000 test images. The default adaptive evaluation applies IBP, CROWN-IBP and CROWN to establish certificates and uses PGD to search for adversarial examples. The returned accuracies are fractions.

In [ ]:

std_acc, cert_acc, adv_acc = wrapped_model.evaluate(test_loader, test_samples=1000)

print(f"Natural accuracy: {std_acc:.2%}")
print(f"Certified accuracy: {cert_acc:.2%}")
print(f"Adversarial accuracy: {adv_acc:.2%}")

std_acc, cert_acc, adv_acc = wrapped_model.evaluate(test_loader, test_samples=1000) print(f"Natural accuracy: {std_acc:.2%}") print(f"Certified accuracy: {cert_acc:.2%}") print(f"Adversarial accuracy: {adv_acc:.2%}")

Certified accuracy is a lower bound on robust accuracy, while adversarial accuracy is an upper bound after the attacks found by this evaluation. A gap between them represents unresolved examples.

For tighter incomplete bounds, select eval_method="ALPHA-CROWN". Complete verification with abCROWN or NeuralSAT is available through evaluate_complete; the complete-verification guide explains backend setup, timeouts and result files.