AI and smartphones are tearing down the vacation language barrier

Flying to a country where you don't speak the language can be daunting — especially if you are traveling alone and even the alphabet isn't a romanized one with which much of Europe and the U.S. are familiar.

Traveling to China, Japan and Korea can mean tackling traffic signs, menus and other pieces of written information where not a single character is recognizable, save for the occasional number.

Read More:

• 6 devices to pack for a quick weekend away
• Why the Amazon Echo Dot is a great travel accessory
• AR directions for Google Maps gets its debut

Thankfully, the smartphone in your pocket and artificial intelligence in the cloud can come to the rescue, via a number of different translation apps.

These work in several ways. Some ask you to type out what you'd like translating — perhaps not helpful when tackling Chinese — while others are audio-based, acting as a virtual interpreter on behalf of you and whoever you are trying to communicate with.

But the focus of this article will be on using the camera to translate written material, like restaurant menus.

First up, a small disclaimer. I'm writing this article in Hong Kong, which may technically be part of China, but due to being under British rule from 1841 to 1997, English is very widely spoken here. All road signs are written in English and Cantonese (a Chinese dialect), and this duplication is seen almost everywhere, including on many restaurant menus and right across the MTR underground rail network.

But this is not a universal rule, and if you venture just slightly off the main tourist trail or away from the city center, English can start to disappear - especially in the window menus of smaller restaurants, and handwritten signs on shop fronts.

Google Translate - iOS and Android

I began with Google Translate, which is free and available for iOS and Android. The app lets you pick from the languages to translate from and to, or you can leave it to work things out automatically — handy if you're not sure of the language you are looking at. Instead of letting you pick between Cantonese and Mandarin, the app simply has Chinese as an option, then works out the rest.

I opened the app, tapping on the camera option, then pointed my iPhone X at a menu in the window of a local restaurant. The live feature —where the printed writing is instantly translated with augmented reality before your eyes — can serve up a real 'wow' moment when it gets it right. The characters shift from Chinese to English so quickly (while maintaining their color, font and background) that it can take a moment for your brain to catch up with what's just happened.

But the app sometimes struggles to make its mind up, with the English translation often flickering between different words as your hand moves slightly.

This is likely down to the Chinese characters looking slightly different as the camera's view of them shifts with the movement of your hand. But, unless you really just need a single word translating, you'll struggle to get a coherent sentence this way.

Instead, the best option with Google Translate is to take a photo within the app. This uploads it to Google's AI cloud for analysis, which takes just a second. You can now draw your finger over the pieces of text you want translating, which appear in your chosen language as you swipe. This second, cloud-powered translation requires a network connection over Wi-Fi or cellular — which, given you're abroad, might not be possible in every scenario.

If you do have a data connection, this option works much more accurately than the live view. I was able to quickly see the names and descriptions of items on a Chinese menu, and get a feel for what the dishes were made of.

Every translation you perform with Google Translate can be viewed as text in the language of your choice, and read allowed by the Google Assistant in either language; all of these searches are also saved in the app for future reference.

Some results were perfectly usable when you're a tourist looking for lunch, like "Korea wild mushroom dish", and "Authentic fried rice cake". These simple translations, paired with the photos of food which invariable appear on such menus, makes the app genuinely useful.

Other translations came out rather strangely, but still made enough sense to help me understand what's going on, such as: "Sauce, but in fact the protagonist is the middle of the fried pork."

Handwritten Chinese

Next, I wanted to stretch Google Translate's legs a little and point it at some handwritten Chinese with no English anywhere to be seen.

I found a sign on the door of a cafe, photographed it with the app, and worked through each line, swiping with my finger and letting the cloud AI translate. At this point, a friendly passerby stopped to help me, explaining that the sign was looking for a waiter to fill a vacancy.

Sure enough, Google spotted "Recruitment" at the top of the sign, followed by "Full-time or part-time". I was seriously impressed, but Google soon stumbled, inexplicably translating "waiter" as "endangered stage name"

If Google isn't for you, there are a lot of other translation apps to pick from. But their feature sets and costs vary. Where the Google option is free for all features, the popular iTranslate costs $3 per month if you want to use the camera feature (after a seven-day trial, which you have to remember to cancel to avoid being charged).

Waygo - iOS and Android

Next, I tried out Waygo, an app specifically made to translate between Chinese, English, Japanese and Korean. The app can work entirely offline (useful if you have no data connection), but this also means it isn't employing Google's trick of asking cloud AI for assistance; all the processing is being done on your phone.

The free version of the app gives you 10 translations per day. But the app is so eager to translate it will get to work as soon as you open the camera, meaning any busy restaurant window will probably use up your 10 translations before you realize what's happened. At least this was my experience.

The translations I intended to use worked well, and offered a little more information than Google. Descriptions of food felt more complete and flowed better, such as "Platter of boiled pork cuts with fresh assorted vegetables [and] black bean sauce."

Even when the translation came out less than fluent, the understandable information made up for the lack of grammatical prowess, such as when a dish was described as "comparatively hot and sour a bit."

My only real criticism here is that Waygo doesn't automatically save the photos you take, instead storing the Chinese (plus a romanized version) and English texts. The app tries to supplement this with images found online, but I didn't find this particularly helpful.

Microsoft Translator - iOS and Android

Finally, I tried the free Microsoft Translator. This lets you speak or type into the app, hold a conversation which it will actively interpret, and take photos of text. The app worked quite well, presenting me with very similar translations to Google - including that unexpected "protagonist" in the fried pork.

Microsoft Translator did a neater job at laying translations over the original text, where Google tends to take a more haphazard approach, with various font sizes sometimes making it difficult to read.

Conclusion

The complexity of Chinese characters surely makes this a tough test for even the most advanced artificial intelligence, and my own lack of any Chinese knowledge means I was relying fully on the apps. If I were pointing my phone at French or Spanish, I'd have a good chance of joining the dots when the apps' understanding of grammar let them down. Sometimes these apps picked out a few nouns, but failed to string them together into a meaningful sentence.

That said, the apps often worked well enough. They didn't produce fluent and beautifully flowing prose, but for the hungry tourist in need of lunch, they did their job.

I perhaps wouldn't rely on them if I had a food allergy, or was specifically in search of vegetarian or vegan food, but they gave me enough confidence to visit a local restaurant, 6,000 miles from home, and place an order. And, best of all, I didn't have to resort to a fast-food burger chain.