Vision Language Models are Biased
Large language models (LLMs) memorize a vast amount of prior knowledge from the Internet that help them on downstream tasks...
Computer Vision
TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Models
Multi-head self-attention (MHSA) is a key component of Transformers, a widely popular architecture in both language and vision. Multiple heads...
Improving zero-shot object-level change detection by incorporating visual correspondence
Detecting object-level changes between two images across possibly different views is a core task in many applications that involve visual...
Understanding Generative AI Capabilities in Everyday Image Editing Tasks
Generative AI (GenAI) holds significant promise for automating everyday image editing tasks, especially following the recent release of GPT-4o on...
Fast and Interpretable Face Identification for Out-Of-Distribution Data Using Vision Transformers
Most face identification approaches employ a Siamese neural network to compare two images at the image embedding level. Yet, this...
PCNN: Probable-Class Nearest-Neighbor Explanations Improve Fine-Grained Image Classification Accuracy for AIs and Humans
Nearest neighbors (NN) are traditionally used to compute final decisions, e.g., in Support Vector Machines or k-NN classifiers, and to...
Vision Language Models Are Blind
While large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini-1.5 Pro, are powering various image-text applications and scoring...
PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck
* Equal contribution. CLIP-based classifiers rely on the prompt containing a {class name} that is known to the text encoder....
GlitchBench: Can large multimodal models detect video game glitches?
Large multimodal models (LMMs) have evolved from large language models (LLMs) to integrate multiple input modalities, such as visual inputs....
ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classification
Image classifiers are information-discarding machines, by design. Yet, how these models discard information remains mysterious. We hypothesize that one way...
DeepFace-EMD: Re-ranking Using Patch-wise Earth Mover’s Distance Improves Out-Of-Distribution Face Identification
Face identification (FI) is ubiquitous and drives many high-stake decisions made by law enforcement. State-of-the-art FI approaches compare two images...
gScoreCAM: What objects is CLIP looking at?
Large-scale, multimodal models trained on web data such as OpenAI’s CLIP are becoming the foundation of many applications. Yet, they...
Visual correspondence-based explanations improve AI robustness and human-AI team accuracy
Explaining artificial intelligence (AI) predictions is increasingly important and even imperative in many high-stakes applications where humans are the ultimate...
How explainable are adversarially-robust CNNs?
Three important criteria of existing convolutional neural networks (CNNs) are (1) test-set accuracy; (2) out-of-distribution accuracy; and (3) explainability. While...
Inverting Adversarially Robust Networks for Image Synthesis
Recent research in adversarially robust classifiers suggests their representations tend to be aligned with human perception, which makes them attractive...
The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
Explaining the decisions of an Artificial Intelligence (AI) model is increasingly critical in many real-world, high-stake applications. Hundreds of papers...
Explaining image classifiers by removing input features using generative models
Interpretability methods often measure the contribution of an input feature to an image classifier’s decisions by heuristically removing it via...
The shape and simplicity biases of adversarially robust ImageNet-trained CNNs
* All authors contributed equally. Adversarial training has been the topic of dozens of studies and a leading method for...
SAM: The Sensitivity of Attribution Methods to Hyperparameters
* Equal contributions. Attribution methods can provide powerful insights into the reasons for a classifier’s decision. We argue that a...
Strike (with) a Pose: Neural networks are easily fooled by strange poses of familiar objects
Despite excellent performance on stationary test sets, deep neural networks (DNNs) can fail to generalize to out-of-distribution (OoD) inputs, including...
A cost-effective method for improving and re-purposing large, pre-trained GANs by fine-tuning their class-embeddings
Large, pre-trained generative models have been increasingly popular and useful to both the research and wider communities. Specifically, BigGANs a...
Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning
Motion-sensor cameras in natural habitats offer the opportunity to inexpensively and unobtrusively gather vast amounts of data on animals in...
VectorDefense: Vectorization as a Defense to Adversarial Examples
Training deep neural networks on images represented as grids of pixels has brought to light an interesting phenomenon known as...
Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting...
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Deep neural networks (DNNs) have demonstrated state-of-the-art results on many pattern recognition tasks, especially vision classification problems. Understanding the inner...
Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
We can better understand deep neural networks by identifying which features each of their neurons have learned to detect. To...
Deep Neural Network are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Deep neural networks (DNNs) have recently been achieving state-of-the-art performance on a variety of pattern-recognition tasks, most notably visual classification...
Innovation Engine: Automated Creativity and Improving Stochastic Optimization via Deep Learning
The Achilles Heel of stochastic optimization algorithms is getting trapped on local optima. Novelty Search avoids this problem by encouraging...
Understanding Neural Networks Through Deep Visualization
Recent years have produced great advances in training large, deep neural networks (DNNs), including notable successes in training convolutional neural...