Unicode strings: Difference between revisions

It is a very capable and [https://www.youtube.com/watch?v=MijmeoH9LT4 remarquable] tool, but also quite complex compared to older single- and double-byte character encodings.

<langsyntaxhighlight lang="algol68">#!/usr/local/bin/a68g --script #

)</langsyntaxhighlight>

=={{header|BBC BASIC}}==

<langsyntaxhighlight lang="bbcbasic"> VDU 23,22,640;512;8,16,16,128+8 : REM Select UTF-8 mode

= LEFT$(B$)</langsyntaxhighlight>

C is not the most unicode friendly language, to put it mildly. Generally using unicode in C requires dealing with locales, manage data types carefully, and checking various aspects of your compiler. Directly embedding unicode strings in your C source might be a bad idea, too; it's safer to use their hex values. Here's a short example of doing the simplest string handling: print it.<langsyntaxhighlight Clang="c">#include <stdio.h>

/* Set the locale to alert C's multibyte output routines */

if (!setlocale(LC_CTYPE, "")) {

fprintf(stderr, "Locale failure, check your env vars\n");

return 1;

/* C99 compilers should understand these */

printf("%lc\n", 0x2708); /* ✈ */

printf("%ls\n", poker); /* ♥♦♣♠ */

printf("%ls\n", four_two); /* 四十二 */

/* oh well */

printf("airplane\n");

printf("club diamond club spade\n");

printf("for ty two\n");

return 0;

}</langsyntaxhighlight>

<langsyntaxhighlight lang="lisp">

#var 四十二строка := "♥♦♣♠Привет". w; // UTF8UTF16 string

console writeLine:строка.writeLine(四十二);

Elixir has exceptionally good Unicode support in Strings. Its String module is fully compliant with the Unicode Standard, version 6.3.0. Internally, Strings are encoded in UTF-8. As source files are also typically Unicode encoded, String literals can be either written directly or via escape sequences. However, non-ASCII Unicode identifiers (variables, functions, ...) are not allowed.

<langsyntaxhighlight lang="go"> var i int

}</langsyntaxhighlight>

<langsyntaxhighlight lang="go"> w := "voilà"

<langsyntaxhighlight lang="j"> '♥♦♣♠'

♥♦♣♠</langsyntaxhighlight>

<langsyntaxhighlight lang="j"> #'♥♦♣♠'

12</langsyntaxhighlight>

<langsyntaxhighlight lang="j"> 7 u:'♥♦♣♠'

4</langsyntaxhighlight>

<langsyntaxhighlight lang="j"> '♥♦♣♠' -: 7 u:'♥♦♣♠'

0</langsyntaxhighlight>

<langsyntaxhighlight lang="j"> 'abcd'-:7 u:'abcd'

1</langsyntaxhighlight>

<langsyntaxhighlight lang="j"> '♥♦♣♠' -:&(7&u:) 7 u:'♥♦♣♠'

1</langsyntaxhighlight>

Here, we see that even when comparing non-ascii characters, we can coerce both arguments to be utf-8 or utf-16 andor inutf-32 either caseand the resulting literal strings would match. (8 u: string produces a utf-8 result.)

How broad/deep does the language support Unicode? What encodings (e.g. UTF-8, UTF-16, etc) can be used? Normalization?

<langsyntaxhighlight Julialang="julia">julia> 四十二 = "voilà";

voilà</langsyntaxhighlight>

<langsyntaxhighlight Julialang="julia">julia>println("\u2708")

✈</langsyntaxhighlight>

<langsyntaxhighlight Lassolang="lasso">local(unicode = '♥♦♣♠')

#unicode -> get (4) -> integer</langsyntaxhighlight>

<langsyntaxhighlight lang="lisp">

</langsyntaxhighlight>

<langsyntaxhighlight lang="lisp">

<langsyntaxhighlight lang="lisp">

In recent versions (since v11.5) of Lingo's only implementation "Director" UTF-8 is the default encoding for both scripts and strings. Therefor unicodeUnicode string literals can be specified directly in the code, and also variable names support Unicode. To represent/deal with string data in other encodings, you have to use the ByteArray data type. Various ByteArray as well as FileIO methods support an optional 'charSet' parameter that allows to transcode data to/from UTF-8 on the fly. The supported 'charSet' strings can be displayed like this:

<syntaxhighlight lang ="lingo">put _system.getInstalledCharSets()

<langsyntaxhighlight lang="locobasic">10 CLS:DEFINT a-z

220 DATA &20,&C3,&A4,&C3,&B6,&C3,&BC,&20,&56,&69,&65,&6C,&65,&20,&47,&72,&C3,&BC,&C3,&9F,&65,&21,&00</langsyntaxhighlight>

=={{header|Mathematica Lua}}==

In Perl, "Unicode" means "UTF-8". If you want to include utf8 characters in your source file, unless you have set <code>PERL_UNICODE</code> environment correctly, you should do<syntaxhighlight lang Perl="perl">use utf8;</langsyntaxhighlight> or you risk the parser treating the file as raw bytes.

Inside the script, utf8 characters can be used both as identifiers and literal strings, and built-in string functions will respect it:<langsyntaxhighlight Perllang="perl">$四十二 = "voilà";

print "$四十二"; # voilà

print uc($四十二); # VOILÀ</langsyntaxhighlight>

or you can specify unicode characters by name or ordinal:<langsyntaxhighlight Perllang="perl">use charnames qw(greek);

print scalar reverse("$x $y"); # ✈ Σ σ</langsyntaxhighlight>

Regular expressions also have support for unicode based on properties, for example, finding characters that's normally written from right to left:<langsyntaxhighlight Perllang="perl">print "Say עִבְרִית" =~ /(\p{BidiClass:R})/g; # עברית</langsyntaxhighlight>

When it comes to IO, one should specify whether a file is to be opened in utf8 or raw byte mode:<langsyntaxhighlight Perllang="perl">open IN, "<:utf8", "file_utf";

open OUT, ">:raw", "file_byte";</langsyntaxhighlight>

<langsyntaxhighlight PicoLisplang="picolisp">(in '(iconv "-f" "ISO-8859-15" "file.txt") (line))</langsyntaxhighlight>

<langsyntaxhighlight Pythonlang="python">#!/usr/bin/env python

print(ord(u[-1]))</langsyntaxhighlight>

Ruby has hardly any specific support for Unicode; however since it focuses on encodings (exactly 100 encodings are supported in Ruby 2.1.0). itIt includes pretty much all known Unicode Transformation Formats, including UTF-8 which is the default encoding since 2.1.0 .

Most Unicode support is to be found in the Regexp engine, for instance /\p{Sc}/ matches everything from the Symbol: Currency category; \p{} matches a character’s Unicode script, like /\p{Linear_B}/.

<langsyntaxhighlight lang="ruby">str = "你好"

str.include?("好") # => true</langsyntaxhighlight>

<langsyntaxhighlight lang="ruby">def Σ(array)

<langsyntaxhighlight lang="ruby">

<langsyntaxhighlight lang="scala">object UTF8 extends App {

}</langsyntaxhighlight>

$ \u0024

a \u0061

b \u0062

c \u0063

d \u0064

e \u0065

¢ \u00A2

£ \u00A3

¤ \u00A4

¥ \u00A5

© \u00A9

Ç \u00C7

ß \u00DF

ç \u00E7

IJ \u0132

ij \u0133

Ł \u0141

ł \u0142

ʒ \u0292

λ \u03BB

π \u03C0

• \u2022

₠ \u20A0

₡ \u20A1

₢ \u20A2

₣ \u20A3

₤ \u20A4

₥ \u20A5

₦ \u20A6

₧ \u20A7

₨ \u20A8

₩ \u20A9

₪ \u20AA

₫ \u20AB

€ \u20AC

₭ \u20AD

₮ \u20AE

₯ \u20AF

₰ \u20B0

₱ \u20B1

₲ \u20B2

₳ \u20B3

₴ \u20B4

₵ \u20B5

₵ \u20B5

← \u2190

→ \u2192

⇒ \u21D2

∙ \u2219

⌘ \u2318

☺ \u263A

☻ \u263B

ア \u30A2

字 \u5B57

文 \u6587

? \uD869

? \uDEA5

 \uF8FF

<langsyntaxhighlight lang="ruby"># International class; name and street

}</langsyntaxhighlight>

<langsyntaxhighlight TXRlang="txr">@{TITLE /[あ-ん一-耙]+/} (@ROMAJI/@ENGLISH)