Detecting natural disasters, damage, and incidents in the wild


Paper & Code

Responding to natural disasters, such as earthquakes, floods, and wildfires, is a laborious task performed by on-the-ground emergency responders and analysts. Social media has emerged as a low-latency data source to quickly understand disaster situations. While most studies on social media are limited to text, images offer more information for understanding disaster and incident scenes. However, no large-scale image datasets for incident detection exists. In this work, we present the Incidents Dataset, which contains 446,684 images annotated by humans that cover 43 incidents across a variety of scenes. We employ a baseline classification model that mitigates false-positive errors and we perform image filtering experiments on millions of social media images from Flickr and Twitter. Through these experiments, we show how the Incidents Dataset can be used to detect images with incidents in the wild.

Image
Incidents detection map. Here we illustrate a map that could be created with our Incidents Model. By detecting and geo-locating incidents in tweets, relevant maps can be created to help inform in response efforts.

Use the following links for access to the paper, code, and data. The GitHub page contains code and final model weights, along with an explanation of how to use it. It also includes instructions for downloading the dataset.

Incidents Dataset

The large-scale Incidents Dataset consists of 446,684 scene-centric class-positive images (annotated by humans) related to natural disasters, types of damage or specific events that can require human attention or assistance, like traffic jams or car accidents. We use the term incidents to refer to the 43 categories covered by our dataset. 49 places are used to add diversity to the images. An additional set of 697,464 class-negative images are part of the dataset and used to train our final model to mitigate false-positive predictions. See the paper for more details.

Image
Incidents Dataset composition. The number of positive (blue) and negative (orange) labeled incident images is shown on the left and the distribution of images for (incident, place) combinations is shown on the right. The dataset contains incidents in many different scenes. White cells indicate the unlikley (incident, place) combinations for which it is hard to find images (e.g., ``car accident in volcano'').

Below we show some example images from the dataset:

Experiments

We examine how our incident detection model, trained with class-negative loss, performs in different real-world scenarios using millions of images collected from two popular social media platforms: Twitter and Flickr.

Image
Using the class-negative loss. (Top) Top confidence images for "airplane accident", "on fire", and "bicycle accident" categories when training without (left) and with (right) the class-negative loss. (Bottom) We report incident AP increments achieved by our model over the baseline model.
Image
Finding peaks in earthquake tweets. (Top) Histogram of tweets obtained from Twitter using natural disaster keywords from 2017-2018. Black lines indicate periods of time when our data collection server was inactive. (Bottom) Number of tweets with earthquake images per day after filtering with at least 50% confidence. For significant earthquakes (above 6.5 magnitude), we notice an increase in earthquake images immediately after the event. Furthermore, we notice a spike on July 20, 2018 not reported in the NOAA database. We manually checked the tweets and found images referring to a severe flood in Japan, indicating that the flood damage may resemble to earthquake damage.
Filtering Flickr images. We show an interface where we filter Flickr images with 95% confidence and show them plotted on the map. More concretely, 41,667,045 geo-located images are filtered to only 26,123 high confidence incident images. You can try the inferface here.
Filtering YouTube videos. We show a video where we filter YouTube videos and show the most confident incident prediction.

Demo

Upload an image to see the incident and place prediction!