Among the hobbies my family pursued over the years was stamp collecting, resulting in a collection that I have always enjoyed looking through.
I haven't contributed much to it myself, but whenever I browse through the pages, I cannot help but think about the history behind it: hundreds of small artifacts, each one having gone through the same process of being collected from envelopes (sometimes even torn from postcards), separated with water, dried, and eventually sorted. Whatever happened to the original letters these stamps once travelled on remains unknown, though.
In 2019, I decided to digitize part of the collection. This simply involved taking photos and turning pages. I mainly did this to preserve it from destruction, but I also had another idea in mind. While a physical collection cannot be replaced, it would be interesting to build a digital inventory out of it. This requires identifying each stamp and entering it into some spreadsheet or specialized software. This is a cumbersome process and probably not very efficient.
Could it be automated?
Pattern matching approach
Back then, I tried using OpenCV:
- Load all stamps ever created (from some public catalogue)
- For each catalogue stamp image, find the corresponding pattern in my collection
I tested two different methods. The first one, a simple template matching algorithm. The limitation is that it is very sensitive to distortions and small variations.
I then tested keypoint methods, in particular SIFT and ORB. The algorithm works by identifying local structural features in the image, represented as points, and then matching these points. This is much more robust to transformations and local defects, but is computationally demanding.
While those actually worked for finding a few stamps, they scale extremely poorly. The total number of stamps issued in France is a few thousand, and a single book of my collection contains 32 pages. These leads to a cartesian product between the two sets of images. Moreover, finding a pattern in an image is itself a rather expensive task (and even more so if rotation and shearing are included).
I put the problem aside and moved on.
Deep-learning approach
Recently, I decided to give this problem another shot. With my new hardware, I had no computational excuse.
My new idea was the following:
- Instead of searching for a specific stamp in the photo, search for stamps in the photo. This is an object detection problem.
- Once all stamps are detected, we obtain a similarity search problem against the catalogue stamps.
While this creates two problems instead of one, each is much simpler. Object detection is independent of the identity of the stamp, while retrieval only has to compare already-cropped images against the catalog.
Data source
The main source of data that I decided to use is timbres-de-france.com. This website, maintained by Michel Vanème and made available free of charge, is a manually curated catalog containing information about French stamps, including images and metadata. While it is not a readily usable dataset, it provides a convenient reference collection of the stamps that could potentially appear in my albums that contain French stamps. The goal is therefore not to identify stamps from scratch, but rather to match the detected stamps against this existing catalog.
To build the local dataset used for this project, I collected the relevant information from the website using a small script based on Beautiful Soup. This was done solely for personal use; the data itself is not being republished, only the results of the identification process.
Object detection
Object detection consists of finding all occurrences of an object in an image. The format in which the results are returned can vary, but it is often represented as simple rectangular bounding boxes around each detected object.
YOLO
The YOLO model series is a family of real-time object detection models capable of predicting bounding boxes, classes, and confidence scores in a single pass. The nice thing is that they can be fine-tuned effectively with relatively few labeled examples.
I used the model YOLO26n, the smallest model in the series.
Using Label Studio, you can quickly create manual training labels for your dataset. My labels are quite simple in structure. For each image:
- The objects belong to a single class ("stamp")
- The objects are captured using axis-aligned bounding boxes
- There can be any number of objects in an image
This is the dataset configuration I used:
<View>
<Image name="image" value="$image"/>
<RectangleLabels name="label" toName="image" canRotate="false">
<Label value="stamp" background="#FFFFFF"/>
</RectangleLabels>
</View>The software allows you to export the labels to many formats, including YOLO, which is basically a simple .txt file containing normalized image coordinates.
The model does, in fact, converge very quickly. I labeled a single image (which contained about 30 stamps), trained for 200 epochs using the augmented dataset (which took about 2 minutes), and predicted the results on the other images. I then used these predictions to re-label three more images, only performing small fixes and adjustments. This is far quicker than labeling an image from scratch. I re-trained the entire model on this new dataset of four images, and after looking at the results determined that it was good enough.
The (surprisingly) quick convergence in this case could be explained by the task being well-defined and having consistent geometric constraints. For other real-world tasks, more examples may be needed.
I still had to apply at least one fix to about a third of the images, as some of them contained genuine ambiguities (including some cases where I did not know myself how to label).
Similarity search
How can we efficiently find the closest matching image in a set of thousands?
In the past, I successfully used locality-sensitive hashing (LSH) on another project, to study how movie trailers are assembled together. I used a hash-based image similarity approach. These methods compare images based on low-level visual characteristics and are sensitive to transformations such as scaling, rotation, and perspective changes. Similarity search is then performed by simply finding the image with the most hash collisions against the query image. This technique does not require implementing complex data structures and scales well, but only works for images that can be compared on a pixel-wise basis. Namely, it is not very robust to translation, scaling, rotation, and perspective changes.
A deep learning approach is the natural next step for this problem. While this could be reduced to a classification problem, such an approach would not scale well and would also be expensive to train (and I probably wouldn't have enough training data). Instead, retrieval methods avoid this by keeping the catalog separate from the model. Adding a new stamp therefore only requires computing and indexing its embedding, which is generally cheap since the model itself does not need to be retrained.
With the rise of large pretrained models, including large language models (LLMs), and specifically multimodal models, more effective solutions can be implemented. LLMs are fundamentally predictor models, which output the most likely continuation given an input. However, there is another class of models designed to "compress" an input into a fixed-size vector called an embedding.
Unlike a traditional hash, embeddings preserve similarity: visually similar inputs are mapped to nearby points in the embedding space. The cosine similarity is used to measure this.
I experimented a bit with generative adversarial networks (GANs), which were popular in the previous decade for generating images (before being largely replaced by stable diffusion), but which also had the ability to compute such embeddings.
Modern multimodal models can also be trained as embedding networks. This enables very interesting use cases, such as searching for documents using a natural language query. But they can also more simply help find similar documents. With multimodal models, these capabilities are not limited to text and can be extended to images as well. This is precisely what I used.
Qwen
I chose the model Qwen3-VL-Embedding-8B. Its embedding dimension, that is, the size of the vector, is 4,096. At the time of writing, it is already 6 months old, but having tested it on other projects, I considered it a safe choice. It is able to process several dozen stamp images per second on my GPU, without any particular optimization.
The model is fed both the stamps from the catalogue and the cropped detections from my collection (from the previous step). The embeddings for the catalogue stamps are then indexed using FAISS.
Retrieval-oriented embedding models are trained so that angular distance in the embedding space (that is, direction with respect to the origin) correlates with semantic similarity. Most of these tools therefore apply an L2 normalization step, scaling the magnitude to 1 while preserving the direction. Interestingly, after normalization, maximizing the dot product is exactly equivalent to minimizing the Euclidean distance.
In this project the catalogue is small enough (7k) that brute-force comparison would likely be sufficient. But such indexing enables scaling to much larger collections: for instance the number of distinct stamp designs across the world is likely on the order of one million.
On the above image the corresponding matching stamp systematically ranks first (bottom), even when my stamp image (top) is not perfectly aligned. One stamp on this page had a 90° misorientation but was still correctly matched (not shown here). This suggests that the model captures higher-level visual features, rather than relying purely on pixel-level similarity.
Overall, the only incorrectly labeled results were the following and accounted to less than 5% of all matches:
- Multiple similar variations exist for the same stamp (color, fee)
- Stamp not existing in the catalogue (though I was actually surprised by its exhaustiveness)
Results and future work
The above website also includes metadata about each stamp. We can, for instance, look at the distribution of years:
A lot of the stamps come from the 1980s, which is unsurprisingly when (what I believe) most of the collecting has taken place.
We can also look at the correlation between the page index and the stamp date. Because the stamps were supposed to have been sorted more or less in ascending order, we should observe a correlation:
The next step would be to make this pipeline reusable by others (including releasing the pre-trained model). I would also be interested in exporting this data into a small static web application for people to browse.