26 Visual Intelligence Translation

[rctneil]

Hi,

Anyone able to test out the updated visual intelligence with this scenario:

Right now i'm organsing a trip to China and a lot of the Chinese apps, Didi, WeChat, AliPay, they do ahve English modes but i'm finding that not everything has been translated. A lot of the time, message popups, or window titles (a lot of the time it's text that is non-user selectable).

If you screenshotted those apps, can VI translate that non-user selectable text?

 

[HouseLannister]

Yes.

 

[rctneil]

Yes.

That's awesome!

Any chance you could do a screen recording and post it?

 

[HouseLannister]

 

[rctneil]

View attachment 2521963

Incredible thankyou! Shame it won't be out in time for my trip next month but glad it's finally coming!

 

[BigBlur]

Incredible thankyou! Shame it won't be out in time for my trip next month but glad it's finally coming!

You can do something similar in iOS 18 too. Take a screenshot, go to the Photos app, select the text, and translate it. Or open the photo/screenshot in the Translate app to view the translation overlaid on top of it.

 

[rctneil]

You absolutely can yes. I've been using the Google Translate app. It works but the new method is just quicker and more convenient.

With current methods, anyone know of a way when you take a screenshot and click share if any transtion app offers an extension or action like "Open in xyz translator", or "translate image"? That would speed up current methods.

 

[HouseLannister]

Gemini Live is also quite good for on-screen translation now that it added screen sharing on iOS.

 

[rctneil]

How would you use that for it? I’ve just installed the Gemini app but it doesn’t show in the share sheet

 

[HouseLannister]

just launch Gemini, press the Live button, then press the screen share button. Then go into whatever app or website has foreign text and ask your phone something along the lines of “what does this mean?”

 

[rctneil]

But i'm talking about stuff onscreen on the phone itself, not via the camera.

 

[HouseLannister]

Sorry, second button over. Starts a broadcast of your screen.

 

[rctneil]

Ahh! Gotcha, That's pretty cool actually!

Even if it did just tell me that "the big blue button in the centre of the screen says "Login"" - when the button was clearly bright red!