I’m fascinated by system failures and oddities, especially when systems are operating under normal conditions. A while back I saw a slide by Ian Goodfellow that made me laugh. It was random noise fed into a pre-trained network and the network did its job classifying the noise as a known object. That begged a few questions. Will different pre-trained networks see the same thing? Different things? What percentage of the time? What’s the highest level of confidence of a network that the random noise is a particular object? On top of these other questions, what is the neural network actually “seeing?”
This post and the associated code is the result of my curiosity toward these unexpected results. Fortunately, PyTorch makes experiments like this easy. To visualize why the network classifies objects a certain way, we’ll use Captum a model interpretability framework. The full code can be seen in the following Github repository.
Before we begin, you may wonder why any of this is relevant. In many cases, developers aren’t building models from scratch. They are reaching for frameworks and pre-trained networks obtained from a model zoo as a starting points. This activity saves time as you didn’t need to collect the data and do the legwork of the initial training. However, this also means that unexpected problems have a way of cropping up in strange places. Depending on the model’s use and function, this could have both security and safety impacts.
Pre-trained models are easy to instantiate and allow you to quickly send them data for classification. With these models, you don’t have to specify the model definition or perform the training on them, this is done for you ahead of time meaning they are ready for use after you have instantiated them. The pre-trained models in the Torchvision library were trained on the Imagenet dataset which consists of 1000 categories. The thing to remember here is that this training was for a single object in an image, not a complex image with multiple objects, which makes for some interesting results, but that’s a topic for another time. Pulling in a pre-trained model from PyTorch’s Torchvision library is easy. It’s a matter of importing the selected model with the
pretrained parameter set to
True. I also set the model to evaluation mode, since there won’t be any training happening during these tests, just inference.
To start with, I have a line of code that sets the device to
cpu depending on if a GPU is available. A GPU isn’t necessary for these simple tests, but since I have one in my machine, I use it.
device = "cuda" if torch.cuda.is_available() else "cpu"
import torchvision.models as models vgg16 = models.vgg16(pretrained=True) vgg16.eval() vgg16.to(device)
A list of the Torchvision pretrained models is available here. For my tests, I didn’t want to use all of the pre-trained networks, because that would get excessive. I selected the following 5 networks.
I didn’t employ any particular methodology in my selection of these networks. Vgg16 and Inception are used quite a bit in examples and they are all different, which was the biggest factor.
Generating Images with Noise
We need a way to automatically generate images of noise that we can feed the neural network. For this task, I used a combination of the
PIL libraries and wrote a small function to return an image containing random noise.
import numpy as np from PIL import Image def gen_image(): image = (np.random.standard_normal([256, 256, 3]) * 255).astype(np.uint8) im = Image.fromarray(image) return im
The result of this function is an image like the one below.
Next we have to perform image transformations on our noise, convert it to a tensor, and normalize it. The following code is intended to be used on not only our random noise, but also any other image we might want to feed into our pre-trained networks to test (hence the Resize and CenterCrop values).
def xform_image(image): transform = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) new_image = transform(image).to(device) new_image = new_image.unsqueeze_(0) return new_image
Once we have our transformed images, it’s easy to get predictions from an instantiated model. In this case, we assume the return from our
xform_image function is called
image_xform. In the code I used for testing, I broke these up into two different functions, but for simplicity, I’ll lump it together here. Basically, we feed the transformed image into the network, run the result through a softmax function, use the
topk function to retrieve the score, and predicted label ID of a singular top result.
with torch.no_grad(): vgg16_res = vgg16(image_xform) vgg16_output = F.softmax(vgg16_res, dim=1) vgg16score, pred_label_idx = torch.topk(vgg16_output, 1)
So, now that we have a basic idea of how to generate these images of noise and feed them to a pre-trained network, what were the results? For this test, I decided to generate 1000 images of noise, run them through the 5 selected pre-trained networks and put them in a Pandas dataframe for quick analysis. The results were pretty interesting and a bit unexpected.
As you can see from the results, some of these networks predicted that the noise was something with a pretty high level of confidence, both resnet18 and densenet had max values over 50%. This is all well and good, but “what” are these networks seeing in the noise? Interesting enough, they weren’t seeing the same things.
Vgg16 Results: stole 978 jellyfish 14 coral_reef 7 poncho 1 Resnet18 Results: jellyfish 1000 Alexnet Results: poncho 942 dishrag 58 Densenet Results: chainlink_fence 893 window_screen 37 chain_mail 33 doormat 20 tile_roof 16 space_heater 1 Inception Results: switch 155 magpie 123 jigsaw_puzzle 102 pillow 85 jean 83 indigo_bunting 81 birdhouse 69 honeycomb 32 poncho 26 carton 25 mousetrap 24 sarong 18 corn 16 chain_mail 16 vacuum 12 window_screen 12 cardigan 11 American_egret 9 broccoli 9 wallet 8 sandbar 7 bee_eater 5 ice_lolly 5 dishwasher 5 hammerhead 5 chainlink_fence 4 apiary 4 nail 4 rain_barrel 4 ashcan 3 jersey 3 bib 3 little_blue_heron 3 cockroach 3 envelope 2 stingray 2 shower_curtain 2 apron 2 starfish 2 miniskirt 1 mitten 1 Italian_greyhound 1 matchstick 1 binder 1 loudspeaker 1 bucket 1 ear 1 shoe_shop 1 handkerchief 1 tray 1 walking_stick 1 sweatshirt 1 dishrag 1 centipede 1 kimono 1
All of these networks saw something different. Resnet18 was absolutely sure it was a jellyfish 100% of the time while Inception on the other hand, had very low confidence in any of its predictions, but saw far more objects than any of the other networks.
Just for fun, I decided to see what kind of a caption Microsoft would add to the image of noise I had near the beginning of this blog post. For this test, I went with the least amount of friction and used the Office 365 version of PowerPoint. The result is interesting because, unlike the imagenet models which are trying to classify a single object, the PowerPoint is trying to classify multiple objects to create an accurate description for the caption.
The result does not disappoint. To me, it looks like the image of noise was interpreted as a circus.
This leads us to another question, what is the neural network seeing that makes it think noise is an object? For this, we can use a tool focused on model interpretability to give us an idea of what the network is “seeing”. Captum is a model interpretability framework for PyTorch. I didn’t do anything too fancy for this and just used the code provided in the tutorials section of the website for my examples. I did add the
internal_batch_size parameter with a value of 50 due to the fact that I ran out of memory rather quickly on my GPU without batching.
For these visualizations, I used two gradient based attributions and an occlusion based attribution. With these visualizations, we are trying to see what was important to the classifier in an attempt to “see” what the network sees. I also used my pre-trained resnet model, but you can change the code to any of the other pre-trained models.
Before we get to an image of noise, for a visual reference, I used an image of a daisy for as a demonstration since these features would be easy to identify.
result = resnet18(image_xform) result = F.softmax(result, dim=1) score, pred_label_idx = torch.topk(result, 1) integrated_gradients = IntegratedGradients(resnet18) attributions_ig = integrated_gradients.attribute(image_xform, target=pred_label_idx, internal_batch_size=50, n_steps=200) default_cmap = LinearSegmentedColormap.from_list('custom blue', [(0, '#ffffff'), (0.25, '#000000'), (1, '#000000')], N=256) _ = viz.visualize_image_attr(np.transpose(attributions_ig.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(image_xform.squeeze().cpu().detach().numpy(), (1,2,0)), method='heat_map', cmap=default_cmap, show_colorbar=True, sign='positive', outlier_perc=1)
noise_tunnel = NoiseTunnel(integrated_gradients) attributions_ig_nt = noise_tunnel.attribute(image_xform, n_samples=10, nt_type='smoothgrad_sq', target=pred_label_idx, internal_batch_size=50) _ = viz.visualize_image_attr_multiple(np.transpose(attributions_ig_nt.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(image_xform.squeeze().cpu().detach().numpy(), (1,2,0)), ["original_image", "heat_map"], ["all", "positive"], cmap=default_cmap, show_colorbar=True)
occlusion = Occlusion(resnet18) attributions_occ = occlusion.attribute(image_xform, strides = (3, 8, 8), target=pred_label_idx, sliding_window_shapes=(3,15, 15), baselines=0) _ = viz.visualize_image_attr_multiple(np.transpose(attributions_occ.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(image_xform.squeeze().cpu().detach().numpy(), (1,2,0)), ["original_image", "heat_map"], ["all", "positive"], show_colorbar=True, outlier_perc=2, )
Now that we’ve seen the previous images generated with a daisy, now it’s time to look at what it looks like with our random noise.
For reference, we are using the resnet18 pre-trained network, and for this particular image, it is 40% certain, it is a jellyfish. I won’t repeat the code here, but the code for the visualizations is the same as above.
As we can see from these visualizations, as humans, it’s still not clear why the network thought this was a jellyfish. There are areas where the network placed more importance, but it’s not nearly as defined as we saw in the daisy example. Unlike a daisy, jellyfish are amorphus and vary in levels of transparency.
You might be wondering, what would these visualizations look like on an image of a jellyfish? With the code I’ve provided in the Github repository, it would be easy to see and compare.
From this post, it’s easy to see how easy it can be to fool neural networks by feeding them unexpected inputs. To these networks’ credit, they served their purpose and returned a result as best they could. We could also see from the results that merely filtering out low confidence predictions may not be a valid countermeasure since some of the predictions had reasonably high confidence. We need to be mindful of situations in which systems fail so easily when implemented into applications so they do not catch us off guard and feed our systems strange and unexpected input, something security professionals have been doing for quite some time.