Comment by eru
> But... the frame is still being rendered behind a hop of latency?Not necessarily, why?
The generalised idea that I see is: 'what can you do with beefy processing in the cloud (but behind a latency hop) and a very small amount of processing power locally?'
A VR example: you do most of the rendering on the server, but you do all the correction for small head movements locally. The server has to send a bit more (and not just as a flat image), but you get over the nausea.
Something like this has been state of the art for ages. And I only say 'something like this' because I don't remember the details.
As a generalised solution if you have crazy amounts of bandwidth to burn: the server is a latency hop behind, so it could send all possible futures that correspond to the inputs that might have happened during that latency hop, and the client picks what to display.
As written, this is crazy. But if instead of all possible futures you have a probability model of human input, and you send the most likely futures, you can probably make it work. Especially if you don't insist on sending raw video. (And even with raw video, you can save a lot of bandwidth by using a special codec that can compress multiple video streams simultaneously by taking advantage of inter-stream redundancies.)
As an example: with this system in the original Super Mario Bros your probability model would most of the time say that the user is most likely to just keep doing what they were doing (running right, or holding the jump button etc). Crucially, in the long run Mario will behave as if he jumped exactly when you pressed the jump button, but you might see a short visual glitch where Mario seems to be doing something wrong or weird, but it'll correct itself very quickly.
Similar to how mosh's predictions work.
If you add a bit of extra smarts locally, you can even add some cheap interpolation between possible futures to try to cover some that the server hasn't sent.
Mostly, what the local player and what the other players are doing will have differ between 'futures', but eg you'll mostly likely to be able to re-use the complex lighting rendering for the fully realistic leafy trees and blades of grass you see in the background between nearly all your possible 'futures'.