Building a Local LLM-as-Judge Pipeline for Image Dataset Curation

I run a travel app that needs a real photo for every place it recommends. Most of those photos are searched automatically, but they are often wrong. Here is the system I built to fix them. I’m building MonkeyEatingMango , a planner that writes day-by-day travel itineraries. It names real places, the kind you could pin on a map: Seville Cathedral with the Giralda beside it, the food stalls inside Mercado de San Telmo, the observation deck near the top of Menara KL. A place without a photo is not only bad UX for travel, it also wastes time on the critical path of latency. The goal of this exercise was to have a curated database of places with their photos and other metadata. The itinerary pass gives me the place name, and I attach this data to show on the frontend. I chose a wiki as an image source because it is manually curated and almost always of much better quality. The complete card looks like the image above. There are about 256,000 places in my database. Only about 9% of them come with a real photo from the open archives. For the rest, the photo is guessed: take the name, look it up in open data like Wikidata and Wikipedia, and grab whatever image that entity happens to be carrying. The guess is fast and free, and it runs live while a user waits for their itinerary. The bug that made me stop was a whole class of mistakes I started calling the celebrity match. Ozone, a bar at a Ritz-Carlton, was showing a photo of Margaret Thatcher. A café called Angelina showed Angelina Jolie. An Indian city, Alwar, carried the portrait of its sitting member of parliament. In every case, the resolver had taken a name, found the most famous Wikipedia or Wikidata entity that shared it, and served that entity's lead image. The name matched, and the picture was of a person. Nothing in the data looked broken: each one had a photo, a source, and not a single flag on it. Important ideas: Lambda Architecture, LLM as Judge, Local LLM, Local vision models, negative cache, algorithm Why it happens: the hot path is fast and blind The live photo lookup, the one that runs while a user is waiting, has no vision capability. It matches a name to an entity and takes that entity's lead image. When the place has a solid id it is usually fine. When it does not, the lookup falls back to "the first Wikipedia article that shares this name," with no check that the article is even about a place. That fallback is exactly how a cocktail bar ends up as a former prime minister. I cannot fix this in the live path the obvious way, by having a vision model look at every photo before serving it. The app runs on servers with no GPU. A hosted vision model on every request would pile latency onto the one moment a user is waiting on me. So the live path stays fast and approximate, on purpose. So I reached for lambda architecture: a fast layer that is allowed to be a little wrong, and a slow batch layer that is correct. The batch layer's results flow back to become the data the fast layer reads from. The hot path serves now and the cold path cleans up after it. When I started, I had only built the hot path. The cold path: one model, two jobs I bought a gaming desktop around 2021. It has NVIDIA GeForce RTX 3060 with 12GB vRAM, 32 GB RAM and AMD Ryzen™ 7 3700X × 16 core CPU. Absolutely the best purchase I made in this decade :smiley:. It came very handy in this experiment. The cold path runs offline on batch, and there I can afford eyes. I used gemma3:12b, a model I pull and run locally with one command, no API bill, thanks to ollama. I gave it two jobs with the same weights. First it reads a place name and classifies it: is this a clean string with place name thatI can photograph, a real venue wrapped inside an activity name? The pure activities and the bare areas get skipped, because there is no honest single photo for them. For everything that survives, the same model switches to vision and judges whether a candidate image is that place . The first candidate that passes wins. So it is one 12B model doing both the text classify and the vision judge. It runs one call at a time on the GPU, which makes it slow. I was fine with that. The world knowledge is the entire reason it works. It is the thing that knows a restaurant is not a prime minister. Why a 12B and not the small one I started with I tried a smaller model first, and I still use it for a narrower job. A 3B vision model, qwen2.5vl, handles the coarse question: is this even a place, or is it a portrait, a logo, a map, a close-up of one object. On that question the 3B turned out to be as accurate as the 7B I benchmarked it against. Both rejected every portrait in my test set, and the 3B runs almost three times faster. For a yes-or-no that needs to tell a building from a face, a bigger model make no difference . Identity is a different problem, and that is where the small model fails. "Is this a place" and "is this the right place" are not the same question. The case that made it click for me: a place listed as St Charles's Church in Vienna, which is the baroque Karlskirche. The small judge passed a photo for it. The photo was of St Stephen's Cathedral, a completely different Vienna church. Both are obviously churches, so the coarse judge had no reason to object. The 12B caught it, because it knows what Karlskirche looks like and what Stephansdom looks like. The small model tells you it is a church. The big one tells you it is the wrong church. | The question being asked | Model | Why that size | |----|----|----| | Is this a place at all, or a portrait / logo / map? | qwen2.5vl, 3B | a coarse visual call; the 3B matched the 7B and runs about 3x faster | | Is this the right place? | gemma3, 12B | needs world knowledge, since Karlskirche and Stephansdom are both churches | | Is this name even something I can photograph? | gemma3, 12B | a regex scored 0% here; it takes knowing what counts as a real venue | So I run both, each on the part it is good at, and I stopped trying to make one model do everything. The blind spot, and the cord I had about 17,000 places that had been handed a photo by the live resolver and then never once checked. That is where the wrong ones hide so the cold path's first job was to go back over all of them and re-judge every photo. Of the batches I have swept so far, about 96 in 300 of those auto-resolved photos checked out. That sounds reassuring until you remember the other few percent is hundreds of real places. The sweep does more than flag the bad ones. For each one it re-resolves the photo and judges the new candidate. If the new photo is good, it writes it. If there genuinely is no photo of the place anywhere, it clears the field to empty, because an empty card beats a wrong one. And if the re-resolved photo is still wrong, the line stops and waits for me. I modeled it is an andon cord taken from Japanese manufacturing. It’s a cord on a factory line that any worker can pull to halt everything the moment they see a defect, before it flows downstream. The re-resolve gets three tries first. The search path is non-deterministic, and I did not want one unlucky miss raising a false alarm. A few side benefits from this exercise: I added “ negative cache ” for the places we could not find any photos for. It helps us save time because hot path would also skip searching for them, and fallback to render without photographs. Whenever the andon cord was pulled, I analyzed what the issue was. Since the image search algorithm is same for both, fixing the issue actually made hot path also more resilient and accurate! Looking at the data more closely, opened more options. I added wiki excerpt and other such info also from the same search By the time I had been back over almost all of that blind spot, about 120 places had been quietly wearing a wrong face. Each one got re-resolved to the correct photo. A dozen or so more had no real photo of them anywhere, so I cleared those to empty and moved on. What the numbers actually say The recovery estimates before I touched anything: | Where the photos stood | | |----|----| | Places in the database | ~250,000 | | With a verified real photo | ~9% | | Never looked at, a cold bulk-import tail | ~85% | | Activity cards with no photo | 29% | | Headline-experience cards with no photo | 24% | Most of that 85% is a cold tail no itinerary ever surfaces. The number that matters is the bottom two rows: the places a user sees on a card, where roughly a quarter to a third had no real picture. When I first probed how many of those I could recover, a quick "does a photo plausibly exist somewhere" came back at 81%. it was a crazy overestimate. When I ran the same set through the real judge, with its strict identity check and a size and quality gate, correct recovery landed around 30%. The 81% had been counting near-misses: a photo of the right city, a photo of a same-named place in a different country, a photo that looks plausible right until the model that knows the building says no. The 30% is not as bad as it sounds. On the slice of places that genuinely have a findable photo, the full pipeline got it right 65 times in 100, up from 36 with the old string-matching approach. And it produced none of the city-level wrong-photos the loose version used to wave through. The rest is mostly a real ceiling, a backstreet food stall or a generic viewpoint that has no single correct photo in the open archive. For those, the empty card is the answer. The goal was never to fill every slot, just to fill as many slots as possible without filling them with the wrong thing. And when I broke the photoless places down by why they had no photo, the result was not what I expected either. The biggest single cause, by a wide margin, is not "no photo exists." It is "a photo exists and my first pass fumbled the lookup," usually on something as dumb as an accent in the name, a curly apostrophe, or a marketing suffix glued onto the real venue ("Seville Cathedral and La Giralda", "Menara KL Observation Deck"). Only about a fifth of the gap is a genuine no-source ceiling. That reframed the whole project for me: less computer vision, more boring string hygiene! When it stops, I do one of two things Either there really is no good source. A nature reserve in Bermuda called Paget Marsh has exactly two photos in the open archive, and both are close-ups of a cedar tree. The judge was right to reject them, and the search was right that nothing else exists. That is thin coverage, not a bug. I mark it as negative cache and move on. Or it is a real defect, and then I do not patch the one row. I go find the cause. The Alwar photo from the top of this post was one of these. The city shares its name with an electoral constituency. That constituency's lead image is a portrait of its sitting MP, so the blind resolver served the politician. The fix was to teach the resolver to refuse that whole class of entity outright. An electoral seat or a person is never the right answer to "find me a photo of this town," no matter how well the name matches. Another stop was the resolver matching "San Nicolás" to "San Telmo", because it treated the "San" prefix as a distinctive, matchable word. So I taught the matcher to ignore San, Santa, Saint and the rest. Every "San" place in the database got fixed at once, not just the one that had stopped the line. Then there was the Australian museum whose Wikipedia lead image is, for reasons known only to the editors, a photo of a van. The resolver picks one image blind, so it picked the van. The fix there was to stop picking blind: pull several candidates, shorten the query step by step, and judge each one until a real photo wins. The most useful bug was the judge being wrong Sometimes the judge rejected a photo that was correct. I am sure you did not expect this, so did I! A historic Danish theatre, relocated into an open-air museum, kept having its real photo thrown out. I looked at the image, because rule one is always look. It was obviously the theatre: white classical building, tiled roof, a portico. The model's own caption even described it correctly. The perception was fine, the decision it drew from that perception was the part that had gone wrong. The prompt had quietly drifted into reading "I cannot positively confirm this is the exact place" as "this is the wrong place." For a generic-looking building or park, and there are thousands of those, that is dangerous. A false reject does more than stop the line, it can swap a correct photo for a worse one, or wipe good coverage to empty. So I rewrote the judge to reject only on positive wrong-evidence : a person or a portrait, a clearly different place, a tight close-up of one object, a document, a logo. Anything that is a plausible view of the place is accepted, and "I am not sure" stopped being grounds to throw it out. I checked the new prompt against a small labeled set of the cases that had fooled the old one. Ran it twice to be sure. It’s hard to believe how much I have come to trust and fear overruling it :grinning: Then I bumped the judge's version number and let the sweep re-verify every doc it had already touched under the corrected judge. Because every place carries the version of the judge that last looked at it, fixing the judge re-runs over the whole history for free. That one felt good. Those version stamps live in the database, which had a second payoff I did not plan for: the sweep became crash-proof in a dumb but satisfying way. When my machine lost its network overnight and rebooted mid-run, I started it again. It skipped every place already stamped at the current judge version and carried on from where it stopped. Run faster The first version of the sweep was slow, because the GPU judge and the network fetches were taking turns. The judge has to run one at a time, but the fetches do not. So I let four places download their candidate images in parallel while a single judge call holds the GPU. That, plus the recovery work itself, moved the numbers I watch: | | Before | After | |----|----|----| | Resolved photos with an identity check | 0% | 99.8% | | Sweep throughput | ~0.21 places/sec | ~0.39 places/sec (+86%) | | Headline-experience photo coverage | ~half | ~three quarters | | Places that looked "never searched" | ~2,700 | 28 | That last row is my favorite. Those places were not un-searched, only searched-and-failed. Once I recorded "searched, nothing good exists" as its own state instead of a hopeful blank, the phantom backlog evaporated. A known dead end, written down once, is worth more than a blank you re-probe forever. A wrong photo is not a one-off to clean up by hand, it is a signal that some rule is wrong. The rule is the thing you fix, whether that is the matcher, the judge prompt, or the search step. You fix it once and the correction flows backwards over the whole dataset. The live path quietly starts reading better data. None of this is a smarter model. gemma3 is a commodity. I am not doing anything clever with it. What still gets me is how much of this one person can run in an evening, on a single consumer GPU, for the price of the electricity! If you want to see what the pipeline puts on the page, here is a finished trip it built, a 10-day Indonesia itinerary . Every place photo on it went through the path above, and so far, not one of them is a former prime minister. I hope so! \