← Lab

Can Jev mask personal data better than an LLM?

Result

Jev scored 78.1% mean masking F2, versus 54.6% for GPT-6 Luna and 70.3% for GLiNER. That is 1.43× Luna’s score. Jev led on three of the four sources.

Personal-data masking score

  • Jev 1.1378.1%
  • GLiNER Multi PII70.3%
  • Mask every word56%
  • GPT-6 Luna54.6%
  • Regex41.5%
Mean character-level F2 across four sources. Higher is better. F2 weights missed personal data more heavily than unnecessary masking, so even masking every word scores 56.0%.

Privy reverses the ranking

  • GLiNER Multi PII58.6%
  • Jev 1.1351.1%
  • GPT-6 Luna41.5%
Jev’s precision on Privy was 24.4%: it hid many characters that should remain visible. A high average F2 is not enough to trust unattended redaction.

Methodology

Compare 400 short English sentences from PIIMB: 100 each from Ai4Privacy, Nemotron-PII, Gretel and Privy. Jev 1.13 makes one masking decision per word at a 0.5 threshold; GPT-6 Luna returns character spans; GLiNER Multi PII runs on CPU at a 0.3 threshold. Score the characters each method masks, then average F2 equally across the four sources. Count Luna’s 19 invalid outputs as misses. These short sentences are a first comparison, not a production document test.

PIIMB dataset (CC BY-NC 4.0)
← All findings