Comment by joerebil

Giovanni here, I have been working for the last 10+ years in Cybersecurity, defense operations.

While building my cloud platform that's bootstrapping in Europe, I kept reading the access logs of my projects, and wondering over and over how to avoid placing a traditional rule-based WAF as a front door, in order to prevent a false-positive bloodbath.

Well aware of the positive security model (that traditional systems do with an OpenAPI schema), I kept thinking if I could push it further => having a language model to evaluate requests.

The first thing I did was taking Clef-flash from Cloudflare, and see if it showed positive signals to handle simple quizzes: given an HTTP request + build manifest, allow or block.

The results were good enough, so I decided to take a small language model (MiniCPM5-2B) and do a LoRA fine-tuning with a custom built dataset on real browser sessions, attack payloads from public collections and tools, and legitimately tricky/hard requests. It doesn’t generate anything, just one forward pass and the decision is the margin between the logits of just two tokens, block and allow against a fixed threshold, so there is no free-text output to hijack (though borderline attacks can still be nudged).

The results on websites (500+) never seen during training were incredibly good: - 98.1% of attacks blocked, 0.0045% of legitimate requests blocked, far better than OWASP CRS @ 87.4% : 11.9% and open-appsec @ 91.9% : 15.6% In many more benchmarks the results are pretty solid.

So I decided to test it on real production traffic, and also there, it scored brilliantly, only 2 false positive requests blocked out of 95k, p.s. those runs and benchmarks were done without providing any application manifest, just a raw HTTP request.

Of course, an inline WAF running at the edge has to be close to invisible in latency: on one H100 a minimal forward pass in plain PyTorch (bf16, replayed as one CUDA graph per 64-token length bucket) decides in 6 ms median. Kernel fusion + FP8 + smaller models, should eventually be able to push it even further down the millisecond.

Weights, code and benchmarks can be found here: https://huggingface.co/giorebil/Agumon-v1.0 If you want to try a live demo: https://eurobase.dev/research/agumon/demo

I'm already working on both a bigger and smaller model, I strongly believe that the "Next Generation" of security tools, will very likely rely on decision models playing a role, in collaboration with frontier models as offline-detection tools.

Thanks for reading, giorebil