Deep Dive into Zero-Shot Image Classification Model in a Lambda Container
Exploring how a CLIP-based model classifies collectible cards in AWS Lambda without retraining or using GPUs.
Stock photo for illustration only, not from the actual event
- The image classifier operates with no training data and no GPU required.
- Utilizes a model from the Contrastive Language-Image Pre-training (CLIP) family.
- Runs inside a Lambda container requiring 2048 MB of memory using onnxruntime.
- New card types can be added instantly simply by writing descriptions by hand.
This article dives deep into the core of the image processing pipeline, specifically the lightweight image classification model residing inside the lambda container. Because the upload bucket is public and anonymous, incoming images might not exclusively consist of AWS Builder Cards. They could range from cat photos and Pokémon cards to poker cards and sports collectibles. Telling them apart poses a challenge for an un-fine-tuned machine learning model.
The eye-brain pathway serves as the key concept here. Just as the human brain distinguishes AWS Builder Cards through specific features, we can instruct the model to do the same using plain English descriptions. The winning model for this implementation is open_clip ViT-B/32, pretrained on laion2b_s34b_b79k, capable of performing image classification within 2 GB of memory efficiently.
CLIP works fascinatingly by embedding images and text into the same vector space, bringing corresponding concepts close together. The model compares photos against a fixed set of predefined labels, and softmax turns those scores into percentages summing up to 100%. The categorization consists of three main groups:
- AWS_LABELS: Describing cloud computing trading cards with pixel-art icons and QR codes.
- COMPETITOR_LABELS: Describing competing trading cards such as Pokémon or Magic: The Gathering.
- NONCARD_LABELS: Describing photos of people or selfies.
Stock photo for illustration only, not from the actual event
Architecturally, deploying as an AWS Lambda function via an ECR container instead of utilizing Amazon Bedrock or SageMaker allows precise resource management under a 10 GB memory limit. Furthermore, stripping out PyTorch during inference in favor of onnxruntime significantly optimizes cold starts and container footprint within strict hardware boundaries.
Interestingly, the initial version utilized a generic label like trading card, which proved disastrous as every card scored roughly 1.0. The fix involved assigning distinct labels to competitor cards so their probabilities land in their own categories, driving the AWS score down to nearly zero, measured at 0.016%.
"No training, no training data and no GPU. This classifier's whole brain is a list of English sentences I typed by hand - and I can teach it a new card type by writing one more."
AWS Builders
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment