Viralno is a Bosnian news aggregator I’m building. It scrapes RSS feeds from local news sites, and when several sites report the same story it merges them into one article with all the sources attached. Finding “the same story” is the whole trick, and my first attempt at it was wrong.

The first attempt: SimHash

SimHash turns a text into a 64-bit fingerprint. Similar texts get fingerprints that differ in only a few bits, so you compare two articles by counting the bits that differ (the Hamming distance). Mine hashed the title and body, and treated anything within 6 bits, published in the last 24 hours, as a duplicate.

It’s cheap, it runs locally, and it works well for what it was designed for: near-identical text. That’s not what news sites publish. Klix and Avaz will cover the same event with different headlines, different opening sentences and different quotes. To SimHash those are just two different documents.

It also failed the other way: an unrelated Klix story got merged into an Avaz story at a distance of 5. I weighted words by raw frequency, with no stopword list or IDF, so the most common Bosnian words dominated every fingerprint. Every article looked a bit like every other article.

The first test: what to embed

Before changing any code I wanted numbers. I labelled 82 articles from my dev database by hand into 14 groups of same-story articles, which gave 18 pairs that should be merged. Everything else should stay apart.

Then I embedded each article with gemini-embedding-2 at 768 dimensions and compared them with cosine similarity. The open question was what to feed the model, so I tried three inputs against SimHash:

  • the full article
  • the title only
  • the title plus the first ~600 characters of the body (the lede)
First test, share of 18 hand-labelled same-story pairs caught: SimHash 11% with 1 false merge, full-article embedding 72%, title-only embedding 89%, title plus 600-character lede embedding 100% with 0 false merges.first test · same-story pairs caught, out of 18simhash: 2 of 18 pairs (11%), 1 false mergesimhash64-bit, hamming ≤ 611%· 1 false mergeembedding of the full article: 72%full articleembedding72%embedding of the title only: 89%title onlyembedding89%embedding of title + first 600 characters: 18 of 18 (100%), 0 false mergestitle + ledeembedding, 600 chars100%

SimHash caught 2 of the 18 pairs, and it merged one pair of unrelated stories. Title and lede caught all 18, with no false merges.

The full article did worse than the shorter input. Long articles carry background paragraphs, quotes and boilerplate that many stories share, and that blurs the comparison. The title alone was too little: it missed 2 pairs where the headlines were worded very differently. The title says what the story is about and the first paragraph says what happened, and that’s enough to recognise a story.

The scores also had a clean gap. Every same-story pair scored at least 0.904 and every different pair at most 0.822. The two stories SimHash had merged scored 0.621, nowhere near. From that I’d have put the threshold in the middle, around 0.86. A first live run changed my mind: correct merges scored 0.908 and up, but one wrong match scored 0.885. That was two articles about the same football match day, one about the coach’s press conference, the other about the team’s training. I shipped with 0.90.

But 18 out of 18 says less than it sounds. With a sample that small, the honest statement is “recall is probably above 82%”. So once the new version had run in production for a while, I tested it properly.

The real test

I took every article viralno scraped over 10 days: 7,206 articles. A duplicate is a pair of articles, so the thing to test is pairs: every two articles from different sites published within 24 hours of each other. That’s 3.37 million pairs.

Both methods scored all of them. The embeddings were already stored in production, and for SimHash I ran the original code from git over the same titles and bodies. Same input for both, and it took about 40 seconds.

Here’s what SimHash sees across all 3.37 million pairs:

Histogram of SimHash Hamming distance over 3,371,845 article pairs. The median is 23 bits, random fingerprints would differ by 32, and only 277 pairs fall at 6 or less, where SimHash merges.distance 0: 3 pairsdistance 1: 3 pairsdistance 2: 13 pairsdistance 3: 19 pairsdistance 4: 36 pairsdistance 5: 58 pairsdistance 6: 145 pairsdistance 7: 429 pairsdistance 8: 1,067 pairsdistance 9: 2,427 pairsdistance 10: 5,085 pairsdistance 11: 9,893 pairsdistance 12: 17,817 pairsdistance 13: 29,785 pairsdistance 14: 46,551 pairsdistance 15: 68,699 pairsdistance 16: 96,096 pairsdistance 17: 127,525 pairsdistance 18: 163,091 pairsdistance 19: 196,587 pairsdistance 20: 227,535 pairsdistance 21: 251,818 pairsdistance 22: 268,115 pairsdistance 23: 274,413 pairsdistance 24: 269,070 pairsdistance 25: 253,850 pairsdistance 26: 231,289 pairsdistance 27: 200,194 pairsdistance 28: 168,075 pairsdistance 29: 135,370 pairsdistance 30: 103,483 pairsdistance 31: 76,551 pairsdistance 32: 54,165 pairsdistance 33: 36,474 pairsdistance 34: 23,751 pairsdistance 35: 14,507 pairsdistance 36: 8,543 pairsdistance 37: 4,567 pairsdistance 38: 2,555 pairsdistance 39: 1,167 pairsdistance 40: 585 pairsmedian 23random 32≤ 6 = merged277 of 3.37M pairs0610203040hamming distance between simhash fingerprints, bits of 64

Two unrelated fingerprints should differ by about 32 bits. Real articles have a median of 23: they all look somewhat alike, because they share the same common words. And only 277 pairs fall within the 6 bits where SimHash merges.

To know which pairs are really the same story, you need labels. Labelling 3.37 million pairs is impossible, and most of them are obviously unrelated anyway, so I sampled 250, with most of them taken where the methods disagree: pairs both methods merge, pairs only one of them merges, near misses with a cosine of 0.80–0.90, and a few random ones. Each sampled pair stands for its group, so the results are weighted back to all 3.37 million.

Two models labelled every pair: Claude Sonnet and GPT, each told that “same story” means the same specific event, not the same topic. I left Gemini out because the embeddings are Gemini’s. Sonnet and GPT agreed on 241 of the 250 pairs, and Claude Opus broke the 9 ties.

Two numbers matter. Precision: of the pairs a method merged, how many were really the same story. Recall: of the pairs that were the same story, how many it merged.

Precision: SimHash 41%, embeddings 97%. Recall: SimHash 1.6%, embeddings 54%.precisionmerges that were the same storysimhash precision: 41%simhash41%embeddings precision: 97%embeddings97%recallsame-story pairs caughtsimhash recall: 1.6%simhash1.6%embeddings recall: 54%embeddings54%

SimHash merged rarely, and when it did it was wrong more than half the time: 59 of the 60 sampled pairs only SimHash merged were different stories. One of them paired a politics story about Milorad Dodik with a football story about Amar Dedić. It caught 1.6% of the duplicates.

Embeddings were right 97% of the time and caught 54%. I ran the same test on just the last 48 hours (1,457 articles, 555 thousand pairs, 210 judged pairs) and got the same picture: 97% precision, 61% recall, and 40% precision for SimHash.

Where the threshold sits

Plotting the cosine score of every judged pair shows why recall isn’t higher:

Cosine similarity of 250 judged pairs. Same-story pairs range from 0.81 up; different-story pairs reach 0.90. The two groups overlap between about 0.80 and 0.90.threshold 0.90same story · 121 pairscosine 0.943cosine 0.972cosine 0.915cosine 0.936cosine 0.953cosine 0.819cosine 0.903cosine 0.970cosine 0.930cosine 0.983cosine 0.931cosine 0.948cosine 0.945cosine 0.933cosine 0.940cosine 0.984cosine 0.872cosine 0.989cosine 0.976cosine 0.975cosine 0.908cosine 0.828cosine 0.974cosine 0.978cosine 0.925cosine 0.886cosine 0.969cosine 0.961cosine 0.906cosine 0.931cosine 0.951cosine 0.909cosine 0.961cosine 0.910cosine 0.932cosine 0.919cosine 0.989cosine 0.982cosine 0.961cosine 0.946cosine 0.987cosine 0.837cosine 0.919cosine 0.971cosine 0.967cosine 0.926cosine 0.988cosine 0.983cosine 0.852cosine 0.970cosine 0.921cosine 0.949cosine 0.916cosine 0.927cosine 0.948cosine 0.991cosine 0.927cosine 0.966cosine 0.910cosine 0.955cosine 0.917cosine 0.952cosine 0.942cosine 0.947cosine 0.961cosine 0.934cosine 0.924cosine 0.945cosine 0.961cosine 0.955cosine 0.962cosine 0.980cosine 0.961cosine 0.977cosine 0.958cosine 0.842cosine 0.980cosine 0.935cosine 0.908cosine 0.965cosine 0.810cosine 0.947cosine 0.944cosine 0.945cosine 0.887cosine 0.960cosine 0.840cosine 0.913cosine 0.958cosine 0.956cosine 0.927cosine 0.964cosine 0.963cosine 0.940cosine 0.927cosine 0.956cosine 0.816cosine 0.951cosine 0.908cosine 0.881cosine 0.943cosine 0.956cosine 0.903cosine 0.985cosine 0.929cosine 0.982cosine 0.903cosine 0.985cosine 0.942cosine 0.968cosine 0.969cosine 0.973cosine 0.986cosine 0.984cosine 0.911cosine 0.936cosine 0.981cosine 0.962cosine 0.983cosine 0.871cosine 0.945different story · 129 pairscosine 0.807cosine 0.702cosine 0.711cosine 0.828cosine 0.664cosine 0.680cosine 0.647cosine 0.655cosine 0.691cosine 0.806cosine 0.631cosine 0.801cosine 0.887cosine 0.813cosine 0.802cosine 0.639cosine 0.711cosine 0.664cosine 0.860cosine 0.803cosine 0.812cosine 0.601cosine 0.657cosine 0.668cosine 0.679cosine 0.805cosine 0.613cosine 0.649cosine 0.651cosine 0.848cosine 0.605cosine 0.857cosine 0.903cosine 0.593cosine 0.626cosine 0.663cosine 0.644cosine 0.818cosine 0.801cosine 0.833cosine 0.641cosine 0.694cosine 0.904cosine 0.678cosine 0.863cosine 0.671cosine 0.865cosine 0.636cosine 0.680cosine 0.866cosine 0.635cosine 0.624cosine 0.543cosine 0.681cosine 0.620cosine 0.852cosine 0.817cosine 0.646cosine 0.835cosine 0.815cosine 0.838cosine 0.875cosine 0.635cosine 0.668cosine 0.686cosine 0.824cosine 0.823cosine 0.662cosine 0.638cosine 0.687cosine 0.854cosine 0.835cosine 0.801cosine 0.595cosine 0.631cosine 0.679cosine 0.891cosine 0.642cosine 0.658cosine 0.710cosine 0.616cosine 0.683cosine 0.658cosine 0.659cosine 0.695cosine 0.812cosine 0.681cosine 0.820cosine 0.813cosine 0.640cosine 0.663cosine 0.667cosine 0.670cosine 0.826cosine 0.598cosine 0.848cosine 0.812cosine 0.735cosine 0.612cosine 0.661cosine 0.657cosine 0.808cosine 0.831cosine 0.846cosine 0.587cosine 0.626cosine 0.645cosine 0.661cosine 0.633cosine 0.662cosine 0.832cosine 0.687cosine 0.821cosine 0.830cosine 0.640cosine 0.618cosine 0.680cosine 0.857cosine 0.812cosine 0.666cosine 0.805cosine 0.708cosine 0.688cosine 0.664cosine 0.807cosine 0.660cosine 0.862cosine 0.818cosine 0.6610.50.60.70.80.91.0cosine similarity of title + lede embeddings

Above 0.90 almost everything is the same story. Between 0.80 and 0.90 the two groups overlap, so any threshold in that range trades one kind of mistake for the other:

Precision and recall of embedding dedup for thresholds from 0.80 to 0.95. Lower thresholds catch more duplicates but merge more different stories. At 0.90, precision is 97% and recall 54%.0%50%100%0.90 (live)cosine ≥ 0.80: precision 35%cosine ≥ 0.81: precision 41%cosine ≥ 0.82: precision 46%cosine ≥ 0.83: precision 49%cosine ≥ 0.84: precision 56%cosine ≥ 0.85: precision 60%cosine ≥ 0.86: precision 71%cosine ≥ 0.87: precision 84%cosine ≥ 0.88: precision 87%cosine ≥ 0.89: precision 91%cosine ≥ 0.90: precision 97%cosine ≥ 0.91: precision 100%cosine ≥ 0.92: precision 100%cosine ≥ 0.93: precision 100%cosine ≥ 0.94: precision 100%cosine ≥ 0.95: precision 100%precisioncosine ≥ 0.80: recall 100%cosine ≥ 0.81: recall 100%cosine ≥ 0.82: recall 88%cosine ≥ 0.83: recall 85%cosine ≥ 0.84: recall 77%cosine ≥ 0.85: recall 73%cosine ≥ 0.86: recall 69%cosine ≥ 0.87: recall 69%cosine ≥ 0.88: recall 65%cosine ≥ 0.89: recall 54%cosine ≥ 0.90: recall 54%cosine ≥ 0.91: recall 48%cosine ≥ 0.92: recall 41%cosine ≥ 0.93: recall 34%cosine ≥ 0.94: recall 28%cosine ≥ 0.95: recall 19%recall97%54%0.800.850.900.95merge when cosine ≥ threshold

Lower the threshold to 0.87 and recall rises to 69%, but precision falls to 84%. I’m keeping 0.90 because the two mistakes don’t cost the same. A missed duplicate means two articles about one story, which readers barely notice. A false merge means an LLM rewrites two unrelated stories into one article, and everyone notices that.

What it misses

I read every duplicate the embeddings missed, and most of them aren’t really failures of the embedding. They’re follow-ups: one event covered from many angles. A femicide verdict produced the verdict itself, the crowd outside the court, the family’s lawyer, the father’s statement and a foundation’s reaction. A helicopter crash near Brčko produced the crash, the injured, the investigation and the defence minister’s visit. Even the judges couldn’t agree on some of these; most of the 9 ties were exactly this kind of pair.

Production also groups articles into clusters: if A matches B and B matches C, all three end up in one article. That catches more duplicates, 73% of them, but the same follow-ups make clusters drift. Over the 10 days, cluster precision was 73%. Over the quieter 48 hours it was 93%. Big breaking stories are where it goes wrong.

Reading the misses also turned up a scraper bug. Klix puts the first sentence of every article in a separate block, which my scraper skipped, so every Klix embedding was missing the sentence that says what happened. Fixing it and re-embedding the 251 Klix articles in the sample caught 6 more real duplicates and lost 2, but also merged 3 more follow-ups. For dedup it was roughly a wash. It still matters, because that sentence now reaches the LLM that rewrites the article.

How it runs

Each source article gets its embedding stored in Postgres with pgvector, in a vector(768) column. When a new article comes in, the worker embeds its title and lede and asks Postgres for the closest source from the last 48 hours:

SELECT s.article_id
FROM article_sources s
JOIN articles a ON a.id = s.article_id
WHERE s.embedding IS NOT NULL
  AND s.created_at >= NOW() - make_interval(hours => $3)
  AND a.status != 'rejected'
  AND 1 - (s.embedding <=> $1::vector) >= $2
ORDER BY s.embedding <=> $1::vector
LIMIT 1

<=> is pgvector’s cosine distance, so 1 - distance is the similarity. One embedding call takes about 0.57 seconds. If it fails, the ingest endpoint answers 503 and the scraper retries later, because storing the article without an embedding would let a duplicate slip through.

The caveats

The labels come from language models, not people. They agreed with each other on 96% of pairs, and the ones they split on were genuinely borderline, but they’re still models. The recall figure is the weakest number here: a single near-miss pair stands for about 270 others, so one label more or less moves it by several points. Read it as “a bit over half”, not as 54.0%. And none of the 20 random pairs turned out to be duplicates, so recall is measured against the duplicates the sample could find.

Precision is the solid number, and it’s the one that matters most.

SimHash answers “are these the same text?”. I needed “are these the same story?”. Embeddings answer that question, and most of what they still miss is a question I haven’t answered myself yet: is a follow-up the same story or a new one?