Comment by rstuart4133

If they didn't have a very comprehensive test suite at the end of the exercise, then based on my experience someone is kidding themselves.

A friend had a tiny accounting system - maybe 15k loc, written in a combination of Python 2.7 / bash / LaTeX. I migrated it to Python 3.13 or whatever by hand. I did try using an LLM for that, but it did things like delete chunks of functionality so I had to check every bloody line it changed. Translating Python 2 to 3 is a fairly mechanical process, so traditional tooling worked better. Then I thought I'd explore LLMs by getting one to translate it to Rust.

The original had no tests or doco. As others have noted, the first attempt superficially looked identical to the original near immediately. You think "Wow". But in my case, it was riddled with bugs. When I asked it to fix bugs I noticed, it introduced as many bugs as it fixed.

I eventually figured out the limited context meant it kept forgetting all the lessons it learned when it fixed things, so it reintroduced old bugs and misunderstandings fixed earlier. (Like I said, this was a learning exercise.) Which is how I came to the same realisation a lot of the industry has - the solution to the context problem is you have to define the application via tests, then tell it to make the tests pass.

The tests rapidly became huge and unwieldy, so part of the exercise became DRY'ing the test code out by getting the LLM to build an integration test harness. The harness captured UI responses, files written, emails and printer output; froze time; fed in known input; and compared the outputs with results captured from the original program.

That cured the bug injection problem, but there were still bugs in all sorts of unexpected places. By unexpected I mean places I would be surprised to find them in human-written code. Human devs run the program in the same way human users do. They use the backspace key and the browser back button to navigate as they test their work. AIs don't. They will use APIs to inject test data unless you explicitly tell them to do it differently. The size of the tests exploded to be far, far larger than the code. Rather than coding I spent my time testing, telling the LLM to write tests to catch the errors I found, then writing standing prompts to try and prevent similar errors. It was meant to be a vibe-coded project, but in the end I got worried about code paths I suspected existed, so I looked at the code anyway to see if they were written the wrong way. Sometimes they would have blown up, say when the SMTP server was offline. To be fair, there was no need to look - I could have just asked the LLM. How a programmer without decades of experience would know what to ask is a mystery.

There is no doubt I got a working Rust program faster than if I had written it myself. Several times faster - but not orders of magnitude. If it was a program I had written, so I was very familiar with it, and it had excellent doco, and a high-coverage test suite, it probably would have been orders of magnitude faster. I think Bun's Rust migration might be an example of that - it's a best-case scenario. There were lots of early announcements, but it took over 6 weeks before they were confident enough to ship it in Claude Code.

But an "I migrated a program with hundreds of thousands of lines of code to a new language and ended up with production-ready code in a few days" - sorry, pics or it didn't happen.