Can open-source Laya replace Jev? 6/14 on SQL, then 3.9 hours of fine-tuning

Chinese narration · 720p · captions track (zh) · MP4

Article: Fine-tuning Laya took its SQL safety score from 7/14 to 14/14

When you build AI into a workflow, there are plenty of moments where you don't want a long explanation. You just want it to pick one option: should this inquiry go to sales or to support, and is this SQL statement safe to run in production?

Models built for that kind of call are known as decision models, and Jev and Laya are two of them. Instead of generating text, they take a short description of the current state plus a few typed questions (multiple choice, score, or yes/no) and return a probability for each option. Both accept exactly the same request format, so you can send the identical payload to each and compare the answers directly.

Jev is a proprietary service from TypeSafe that you call over an API; you can't see inside it. Laya, from Convai Innovations, is released under Apache 2.0. It is built on ModernBERT, a bidirectional encoder, has about 421 million parameters, and runs on a Mac CPU. Because the weights are public, you can keep training it on data from your own use case. That is fine-tuning.

For this experiment I picked SQL, the language used to query and change databases. The question: if I teach Laya examples of SQL judgments, how much better does it get, and how does it compare with Jev?

What the models are asked to judge

Reviewing SQL involves two separate questions: is it safe to run, and does it produce the result you actually wanted?

A DELETE with the wrong condition wipes out rows that should have stayed. That is a safety problem. A report query that double-counts revenue changes no data at all, yet it returns the wrong number. That is a correctness problem: the logic doesn't match the intent.

To see whether the models can tell the difference, I wrote 14 PostgreSQL statements by hand. Some are correct and ready to ship; the others each hide one bug. The bugs are a JOIN that double-counts a SUM, an = NULL condition that is never true, a DELETE whose OR removes two extra years of orders, an UPDATE with no WHERE clause, an accidental CROSS JOIN, a CREATE INDEX without CONCURRENTLY that locks an 80-million-row table, a BETWEEN that includes one extra day, and a leftover injection fragment that makes the condition always true.

Without fine-tuning, Laya got the safety question right on 7 of 14 statements and the correctness question on 6 of 14. Jev got safety right on all 14 and correctness on 13. The Laya used here is typed-decisions, the best-scoring of its three public checkpoints.

Out of the box, the gap was clear. So would fine-tuning on SQL examples change the picture?

Teaching it good and bad examples

Laya's official training data covers workflows like customer service and invoicing, and contains no SQL, so I had to build my own. I took a public SQL dataset on Hugging Face (gretelai's synthetic_text_to_sql), used each original statement as a correct example, and broke one part of it by rule to make an incorrect one.

For example, deleting the WHERE clause that limits an UPDATE turns a query meant to change specific orders into one that changes every order. Other rules flip AND and OR, shift a comparison operator, or drop a column from GROUP BY. Since every mutation follows a fixed rule, the labels for each example were generated by rule too.

That produced 2,924 examples. 2,684 went into training and 240 were held out for evaluation, never shown to the model during training.

Training used Laya's official training loop, which is set up for two T4 GPUs; I ported it to run on my M4 Mac. Two passes over the data (two epochs) took about 3.9 hours. No rented compute was needed.

Safety improved, but more training wasn't better

After training, I ran the same 14 statements again.

After one epoch, Laya got every safety question right, matching Jev on these 14 statements.

After the second epoch it got one wrong: it rated the statement with the injection fragment as safe. Training loss was already very low going into the second epoch, which suggests the model was starting to memorize the training set, i.e. overfitting. More training time did not automatically mean better results.

A perfect score on 14 statements also doesn't mean it can judge the safety of any SQL. Whether the improvement carries over to other queries still needs testing.

Spotting "is this SQL correct?" is still a gap

Separately from safety, I checked whether each statement does what it was meant to do.

Here Laya got 8 of 14 after one epoch and 10 after two, still short of Jev's 13.

One of the misses was the BETWEEN query with the wrong date range, and Laya called it "correct" with high confidence (0.94). The = NULL, CROSS JOIN, and missing-CONCURRENTLY statements went the same way: all confidently rated correct. So the fallback of "only ask a human when the model is unsure" doesn't work here.

On the other hand, it learned the mutations it was explicitly taught very quickly. It gave the double-counting JOIN only a 0.11 probability of being correct, and the injection statement 0.04. On the 240 held-out examples, safety accuracy reached 1.000. The trouble is that bug types it never saw remain invisible, and even with similar bugs in training it can miss them on a different statement. Memorizing training examples and catching problems in new situations are still two different things.

Fine-tuning helped. It still isn't enough to hand SQL review over to the model entirely.

Where Jev is valuable: trying it out cheaply

What does this mean for a company? What follows is my proposal based on this experiment, not something measured in real operations.

With Jev, you don't have to train a model or set up servers to run one before you can try it. You hand it cases from your own work and see how often its decisions match.

A company that wants to automate routing of customer inquiries, for example, could have Jev guess the responsible department for past inquiries and compare the results with how people actually classified them.

The value is that you skip the burden of building infrastructure to run a model and can focus on whether it helps the business at all. Classification and routing are also the uses envisioned in Jev's official introduction.

Before actually relying on it, though, you need to check accuracy on your own cases. Whether the data is allowed to leave for an external service is also part of the decision.

Where Laya is valuable: running in-house and growing it to fit

Laya can run in your own environment, so inputs never have to be sent to an outside service. The public model and fine-tuning instructions are on the official model card.

For the same inquiry-routing task, if your company has its own categories and plenty of past examples with the right answers, those could become training data.

The value then is being able to improve the model against your own criteria. Keeping data in-house is another reason to choose it.

In exchange, preparing training data, running training, and checking accuracy all become your job, and the hardware to run it costs money. Even at high volume, whether it ends up cheaper than an external service depends on including those operating costs in the comparison.

How I would split the work here

If you first want to find out whether a decision model helps at all, Jev is the candidate, since it needs little training or deployment work. If you have your own training data and the people to run it, and want processing to stay in-house, growing Laya is worth considering.

For SQL review specifically, the two could be combined: let Laya handle the categories that have been evaluated thoroughly, and use Jev or a human reviewer for the step that checks what a query actually means.

Deciding "no review needed" from Laya's probability alone looks hard, though, because it has been confidently wrong. High-impact operations such as updates and deletes should be routed to review by a separate rule, and SQL that can't leave the company should be checked internally.

What ultimately needs testing is whether review effort goes down while missed bugs stay within an acceptable range. The 14-for-14 result is a first step toward finding that kind of use.

Data and scripts: DDnim/jev-vs-laya, including per-model scores and training settings. A Japanese video and full write-up are on the /ja/ page.