Vigil Field Notes

The Scary Text That Was Actually Fine

OTP. Bank. Login. Five minutes. Every word a scam filter fears, in the one message you are supposed to receive.

A one-time password is exactly what a scammer wants, so a security tool has every reason to be nervous about a text like this one, from our test suite:

Your OTP for login to HDFC Bank NetBanking is 482913. Valid for 5 minutes. Do not share this OTP with anyone.

It has the vocabulary of a scam and the intent of a bank doing its job. Telling those two apart is the entire problem.

What A Word Model Sees

Vigil’s fast first layer is a statistical language model. It looks at which words and fragments appear, and it is very good at that: on a held-out set of 4,378 SMS and email messages it had a 1.0% false positive rate.

But OTP theft and OTP delivery use the same words. In one recorded run on a version of this message, that model rated it 99.7% scam-like, naming “otp” and “bank” among its key terms. On a separate set of urgent but legitimate messages, it was wrong 18.5% of the time. Word statistics can’t read intent.

The Difference Is One Word

The rule engine isn’t looking for the word OTP. It looks for a request: a verb like share, send or give close to a word like OTP or code, and it checks that the verb isn’t preceded by not or never. “Share the OTP” is a request. “Do not share this OTP” is a warning.

The word model’s score can add at most 10 points out of 100. So a 99.7% alarm contributed 9 points, and nothing else in the message gave the risk engine a reason to go higher.

The disagreement stays visible. The report keeps a low-severity finding that the wording resembles known scams, so you can see what was weighed and why it didn’t decide the outcome. In the latest test suite run, this message scored 0/100, also Low Risk.

Why We Fine-Tuned A Model For This

An off-the-shelf small language model did far worse than simple word statistics here. On 169 held-out messages, including scam families never seen in training, the base model flagged 85.5% of legitimate messages. After fine-tuning, that fell to 9.7%, while it caught 93.3% of scams.

The full system flagged 11.3% of legitimate messages, and only 1.6% of the 62 legitimate messages were rated High Risk or worse. The rest of those false alarms stopped at Suspicious.

What This Doesn’t Prove

• Low Risk is not a guarantee. Vigil can’t verify who sent a text from its words alone, and a forged message can copy this wording exactly.
• The negation rule is narrow. We tested “Do not share this OTP with anyone except our officer”, and the rule does not match it. Catching that wording depends on the reasoning model, which is not always available and is not always right.
• If anyone contacts you and asks for the code, it is a scam, however legitimate the original text was.