Problem Description

Retail stores need to quickly identify when shelf slots are empty or running low on inventory. However, manually checking shelf images or walking through store aisles is time-consuming, inconsistent, and difficult to scale across many products and locations. Being able to automatically classify shelf conditions can help retailers restock faster, reduce missed sales, and improve inventory visibility.

The Classify task on the Abilities tab can analyze store shelf imagery and assign one of several inventory status labels based on the visible shelf condition.

For example, a shelf image should be classified as Empty if the expected product area is completely empty and no units are visible in the slot. A shelf should be classified as Out_of_Stock if only a few products of the same target product remain, the slot has large visible gaps, or the product is clearly close to running out. A shelf should be classified as OK if the product slot is sufficiently stocked and products are visibly present across most of the shelf space.

We will need to separate true empty shelf slots from low-inventory cases and confusing cases such as messy shelves, partially hidden products, reflections, price tags, shadows, shelf dividers, or nearby products from a different shelf slot.

Our expected inputs are images of retail store shelves, and the expected output will be a single classification label: Empty, Out_of_Stock, or OK.

UI Tutorial

Step 1: Create an Ability

Go to the Abilities tab and select the button Create Ability.

Fill out basic information about the ability such as its name and the description of the task itself. Since we are classifying an image, select the Task Type as Classify.

Name: retail-shelf-stock
Description: Classify a retail shelf image as empty, out of stock / low inventory, or OK

Step 2: Task Configuration

To configure the task, we need to select a dataset for the specific task. If you have already uploaded your retail shelf images in a dataset, simply select the name of your dataset.

However, if you haven't already done so, select <New Dataset>, upload your shelf images, and create the labels Empty, Out_of_Stock, and OK.

When labeling the dataset, use Empty for shelf slots where the target product is fully missing, Out_of_Stock for shelf slots where only a small number of the target product remain or the shelf space has large visible gaps, and OK for shelf slots that are reasonably stocked.

Step 3: Configuration

Our next step is to configure the prompt, select the model, and image size. For this use case, we recommend using the below prompt and settings for highest accuracy and best results.

Prompt:

Determine the primary content of the image and assign exactly one label: ['Empty', 'Out_of_Stock', 'OK']. Choose 'Empty' only if the image's main focus is a retail shelf slot, shelf bay, or assigned product area with zero visible units of the target product remaining. The target shelf space must be completely bare. If there is even one visible unit of the target product remaining in the main target shelf area, do not choose 'Empty'. Choose 'Out_of_Stock' if the image's main focus is a retail shelf slot, shelf bay, or assigned product area where at least one visible unit of the target product remains, but the shelf space is clearly low inventory. This includes cases where only 1 to 5 units remain, products are sparse, pushed to one side, widely separated, isolated in a mostly empty shelf bay, or surrounded by large empty gaps where more of the same product should be. If the target area is mostly empty but still has at least one product unit remaining, choose 'Out_of_Stock' instead of 'Empty'. Choose 'OK' only if the image's main focus is a retail shelf slot, shelf bay, or assigned product area that is sufficiently stocked with the target product. Products should be visible across most of the target shelf space, with no large obvious gaps and no immediate restocking needed. Ignore neighboring stocked shelves, background products, price tags, shelf dividers, shadows, reflections, advertisements, and signs unless they are part of the main target product area. Do not rely on written words like 'empty' or 'out of stock'; classify based on the visible shelf condition. Return only the single best-fitting label.

We chose Max New Tokens to be 10 since outputting one classification label will not require many new tokens. In addition, we use an image size of 640x640 to preserve enough detail to distinguish empty slots, sparse product rows, and sufficiently stocked shelves. Since this is an image classification task rather than a video event detection task, FPS is not applicable.

Step 4: Run Evaluation

To check how well the prompt does against the dataset, our next step is to run the evaluation. If needed, review the examples in your dataset to ensure all necessary images are correctly labeled and can be used in the evaluation.

Step 5: Check Evaluation

All evaluations can be reviewed in the Abilities tab by clicking the dropdown arrow next to the associated ability-alias. Evaluations can take around 15-20 minutes to complete based on the size of the dataset.

In addition to the performance, recall, and precision percentages on the Abilities tab, you can see a visualization of what the model predicted by revisiting the dataset. Click on the three dots and select "Go to reference dataset". Select one of the images in the dataset and click on the review button. After running the evaluation, you can see what the model labeled as Empty, Out_of_Stock, or OK and compare it to your source of truth. With this, you can improve your prompts and thus improve your accuracy.

Tips for Accuracy

1. Explicit "Negative" Case

Telling the model exactly what not to look for is just as important as telling it what to look for. This is because if you only define Empty, the model might classify messy or partially stocked shelves as empty too often.

In our prompt, the explicit non-empty cases are Out_of_Stock and OK. Out_of_Stock covers shelf slots that are not fully empty but still need restocking, while OK covers shelves that are sufficiently stocked. This helps prevent the model from treating every visible gap as a full empty-shelf issue.

2. Define "Edge Cases"

The key to high accuracy is a deep understanding of your specific acceptance criteria. In a retail shelf context, the line between Out_of_Stock and OK can be thin. You must be explicitly clear about where that line is drawn.

Examples of edge cases include shelves with products pushed to one side, products hidden behind shelf dividers, shelves with mixed product types, packaging that blends into the shelf background, shadows that look like empty space, or angled images where the shelf depth is hard to judge.

3. Use Balanced Examples

Make sure the dataset has enough examples of Empty, Out_of_Stock, and OK shelves. If the dataset has too many OK examples and not enough Empty or Out_of_Stock examples, the model may become biased toward predicting OK.

4. Use Synthetic Images Carefully

Synthetic images from Nano Banana 2 can help create more examples of empty shelves, sparse shelves, and fully stocked shelves. However, the synthetic images should still look like realistic store imagery. Include variations in shelf angle, lighting, product type, shelf spacing, and store environment so the model does not overfit to one visual style.

For Empty examples, the target product slot should have zero visible units. For Out_of_Stock examples, the target product slot should still have at least one visible unit remaining but clearly show low inventory. For OK examples, the product slot should be sufficiently stocked with no large obvious gaps.