UTF-8 encode and decode: Difference between revisions

As described in [[UTF-8]] and in [[wp:UTF-8|Wikipedia]], UTF-8 is a popular encoding of (multi-byte) [[Unicode]] code-points into eight-bit octets.

 

 

 

Demonstrate the functionality of your encoder and decoder on the following five characters:

 

Provided below is a reference implementation in Common Lisp.

 

=={{header|8th}}==

hex \ so bytes print nicely

 

 

bye

Output:<pre>

A 41

[e2,82,ac] €

[f0,9d,84,9e] 𝄞

</pre>

 

{{works with|Ada|Ada|2012}}

 

with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;

with Ada.Integer_Text_IO;

end;

end loop;

{{out}}

<pre>Character Unicode UTF-8 encoding (hex)

𝄞 U+1D11E F0 9D 84 9E

</pre>

 

=={{header|C}}==

#include <stdio.h>

#include <stdlib.h>

return 0;

}

Output

Character Unicode UTF-8 encoding (hex)

----------------------------------------

𝄞 U+1d11e f0 9d 84 9e

 

 

=={{header|Common Lisp}}==

Helper functions

 

(defun ascii-byte-p (octet)

"Return t if octet is a single-byte 7-bit ASCII char.

when (= (nth i templates) (logand (nth i bitmasks) octet))

return i)))

 

Encoder

 

(defun unicode-to-utf-8 (int)

"Take a unicode code point, return a list of one to four UTF-8 encoded bytes (octets)."

;; return the list of UTF-8 encoded bytes.

byte-list))

 

Decoder

 

(defun utf-8-to-unicode (byte-list)

"Take a list of one to four utf-8 encoded bytes (octets), return a code point."

(error "calculated number of bytes doesnt match the length of the byte list")))

(error "first byte in the list isnt a lead byte"))))))

 

The test

 

(defun test-utf-8 ()

"Return t if the chosen unicode points are encoded and decoded correctly."

;; return t if all are t

(every #'= unicodes-orig unicodes-test)))

 

Test output

 

CL-USER> (test-utf-8)

character A, code point: 41, utf-8: 41

character 𝄞, code point: 1D11E, utf-8: F0 9D 84 9E

T

 

=={{header|D}}==

import std.stdio;

 

writefln("%s %7X [%(%X, %)]", c, unicode, bytes);

}

 

{{out}}

 

=={{header|Elena}}==

import extensions;

string printAsUTF8Array()

{

}

string printAsUTF32()

{

}

}

"𝄞".printAsString().printAsUTF8Array().printAsUTF32();

console.printLine();

{{out}}

<pre>

€ E2 82 AC U+20AC

𝄞 F0 9D 84 9E U+1D11E</pre>

 

=={{header|F_Sharp|F#}}==

// Unicode character point to UTF8. Nigel Galloway: March 19th., 2018

let fN g = match List.findIndex (fun n->n>g) [0x80;0x800;0x10000;0x110000] with

|2->[0xe0+(g&&&0xf000>>>12);0x80+(g&&&0xfc0>>>6);0x80+(g&&&0x3f)]

|_->[0xf0+(g&&&0x1c0000>>>18);0x80+(g&&&0x3f000>>>12);0x80+(g&&&0xfc0>>>6);0x80+(g&&&0x3f)]

{{out}}

<pre>

{{works with|gforth|0.7.9_20191121}}

{{works with|lxf|1.6-982-823}}

over + swap ?do

i c@ 3 .r loop ;

\ can also be written as

\ 'A' test 'ö' test 'Ж' test '€' test '𝄞' test

{{out}}

<pre>

If you also want to see the implementation of <code>xc!+</code> and <code>xc@+</code>, here it is (<code>u8!+</code> is the UTF-8 implementation of <code>xc!+</code>, and likewise for <code>u8@+</code>):

 

 

$80 Constant max-single-byte

REPEAT $7F xor 2* or r>

BEGIN over $80 u>= WHILE tuck c! 1+ REPEAT nip ;

 

=={{header|Go}}==

===Implementation===

This implementation is missing all checks for invalid data and so is not production-ready, but illustrates the basic UTF-8 encoding scheme.

 

import (

rune(b[3]&mbMask)

}

{{out}}

<pre>

 

===Library/language===

 

import (

fmt.Printf("%-7c U+%04X\t%-12X\t%c\n", codepoint, codepoint, encoded, decoded)

}

{{out}}

<pre>

 

Alternately:

 

import (

fmt.Printf("%-7c U+%04X\t%-12X\t%c\n", codepoint, codepoint, encoded, decoded)

}

{{out}}

<pre>

=={{header|Groovy}}==

{{trans|Java}}

 

class UTF8EncodeDecode {

}

}

{{out}}

<pre>Char Name Unicode UTF-8 encoded Decoded

Example makes use of [http://hackage.haskell.org/package/bytestring <tt>bytestring</tt>] and [http://hackage.haskell.org/package/text <tt>text</tt>] packages:

 

import qualified Data.ByteString as ByteString (pack, unpack)

(printf "U+%04X" codepoint :: String)

(intercalate " " (map (printf "%02X") values))

{{out}}

<pre>

=={{header|J}}==

'''Solution:'''

ucp=: 9&u: NB. converts to unicode from UTF-8 or unicode codepoint integer

 

'''Examples:'''

AöЖ€𝄞

ucp 65 246 1046 8364 119070

1d11e

utf8@dfh ucp_hex utf8 65 246 1046 8364 119070

 

=={{header|Java}}==

{{works with|Java|7+}}

import java.util.Formatter;

 

}

}

{{out}}

<pre>

=={{header|JavaScript}}==

An implementation in ECMAScript 2015 (ES6):

/***************************************************************************\

|* Pure UTF-8 handling without detailed error reporting functionality. *|

?( m&0x07)<<18|( n&0x3f)<<12|( o&0x3f)<<6|( p&0x3f)<<0

:(()=>{throw'Invalid UTF-8 encoding!'})()

The testing inputs:

const

str=

:[ [ a,b,c]]

,inputs=zip3(str,cps,cus);

The testing code:

console.log(`\

${'Character'.padEnd(16)}\

${`[${[...utf8encode(cp)].map(n=>n.toString(0x10))}]`.padEnd(16)}\

${utf8decode(cu).toString(0x10).padStart(8,'U+000000')}`)

and finally, the output from the test:

<pre>

</pre>Note that the misalign there on the last line is caused by the string length of astral characters being 2 so the padding functions break.

 

 

 

 

{{out}}

 

=={{header|Kotlin}}==

 

fun utf8Encode(codePoint: Int) = String(intArrayOf(codePoint), 0, 1).toByteArray(Charsets.UTF_8)

System.out.printf("%-${n}s %c\n", s, decoded)

}

 

{{out}}

 

=={{header|langur}}==

 

for .cp in "AöЖ€𝄞" {

val .utf8 = s2b cp2s .cp

val .cpstr = b2s .utf8

 

{{out}}

- byteArray.toHexString (intStart, intLen): returns hex string representation of byte array (e.g. for printing)<br />

- byteArray.readRawString (intLen, [strCharSet="UTF-8"]): reads a fixed number of bytes as a string

put "Character Unicode (int) UTF-8 (hex) Decoded"

repeat with c in chars

ba = bytearray(c)

put col(c, 12) & col(charToNum(c), 16) & col(ba.toHexString(1, ba.length), 14) & ba.readRawString(ba.length)

Helper function for table formatting

str = string(val)

repeat with i = str.length+1 to len

end repeat

return str

{{out}}

<pre>

=={{header|Lua}}==

{{works with|Lua|5.3}}

-- Accept an integer representing a codepoint.

-- Return the values of the individual octets.

end

end

{{out}}

<pre>

 

=={{header|M2000 Interpreter}}==

Module EncodeDecodeUTF8 {

a$=string$("Hello" as UTF8enc)

}

EncodeDecodeUTF8

{{out}}

<pre>

</pre >

 

{{out}}

<pre>{65, 195, 182, 208, 150, 226, 130, 172}

{65, 246, 1046, 8364}

</pre>

 

=={{header|Perl}}==

use strict;

use warnings;

} split //, $utf8;

print "\n";

 

{{out}}

 

=={{header|Phix}}==

Standard autoinclude, see the manual and/or builtins/utfconv.e

As requested in the task description:

 

 

{{out}}

<pre>

 

=={{header|Processing}}==

 

Integer[] code_points = {0x0041, 0x00F6, 0x0416, 0x20AC, 0x1D11E};

tel_1 += 30; tel_2 = 50;

}

 

 

The encoding and decoding procedure are kept simple and designed to work with an array of 5 elements for input/output of the UTF-8 encoding for a single code point at a time. It was decided not to use a more elaborate example that would have been able to operate on a buffer to encode/decode more than one code point at a time.

 

 

Procedure UTF8_encode(x, Array encoded_codepoint.a(1)) ;x is codepoint to encode, the array will contain output

Print(#CRLF$ + #CRLF$ + "Press ENTER to exit"): Input()

CloseConsole()

Sample output:

<pre> Unicode UTF-8 Decoded

 

=={{header|Python}}==

#!/usr/bin/env python3

from unicodedata import name

chars = ['A', 'ö', 'Ж', '€', '𝄞']

for char in chars:

{{out}}

<pre>Character Name Unicode UTF-8 encoding (hex)

 

=={{header|Racket}}==

 

(define char-map

(map (curryr number->string 16) bites)

(bytes->string/utf-8 (list->bytes bites))

{{out}}

<pre>#\A A (41) A LATIN-CAPITAL-LETTER-A

{{works with|Rakudo|2017.02}}

Pretty much all built in to the language.

 

for < A ö Ж € 𝄞 😜 👨‍👩‍👧‍👦> -> $char {

printf " %-5s | %-43s | %6s | %-7s | %12s |%4s\n", $char, $char.uninames.join(','), $char.ords.join(' '),

('U+' X~ $char.ords».base(16)).join(' '), $char.encode('UTF8').list».base(16).Str, $char.encode('UTF8').decode;

{{out}}

<pre>Character| Name | Ordinal| Unicode | UTF-8 encoded | decoded

=={{header|Ruby}}==

 

character_arr = ["A","ö","Ж","€","𝄞"]

for c in character_arr do

puts ""

end

{{out}}

<pre>

Code-Units: F0 9D 84 9E

 

</pre>

 

=={{header|Scala}}==

=== Imperative solution===

 

val codePoints = Seq(0x0041, 0x00F6, 0x0416, 0x20AC, 0x1D11E)

printf(s"%-${w}c %-36s %-7s %-${16 - w}s%c%n",

codePoint, Character.getName(codePoint), leftAlignedHex, s, utf8Decode(bytes))

=== Functional solution===

 

object UTF8EncodeAndDecode extends App {

 

println(s"\nSuccessfully completed without errors. [total ${scala.compat.Platform.currentTime - executionStart} ms]")

=== Composable and testable solution===

 

object UTF8EncodeAndDecode extends TheMeat with App {

 

}

 

=={{header|Seed7}}==

include "unicode.s7i";

include "console.s7i";

writeln("-------------------------------------------------");

for ch range "AöЖ€𝄞" do

writeln(ch rpad 11 <& "U+" <& ord(ch) radix 16 lpad0 4 rpad 7 <&

end for;

 

{{out}}

 

=={{header|Sidef}}==

code.chr.encode('UTF-8').bytes.map{.chr}

}

assert_eq(n, decoded.ord)

say "#{decoded} -> #{encoded}"

{{out}}

<pre>

=={{header|Swift}}==

In Swift there's a difference between UnicodeScalar, which is a single unicode code point, and Character which may consist out of multiple UnicodeScalars, usually because of combining characters.

 

func encode(_ scalar: UnicodeScalar) -> Data {

print("character: \(decoded), code point: U+\(String(scalar.value, radix: 16)), \tutf-8: \(formattedBytes)")

}

{{out}}

<pre>

=={{header|Tcl}}==

Note: Tcl can handle Unicodes only up to U+FFFD, i.e. the Basic Multilingual Plane (BMP, 16 bits wide). Therefore, the fifth test fails as expected.

set u [format %c $int]

set bytes {}

lappend res [encoder $test] -> [decoder [encoder $test]]

puts $res

<pre>

0x0041 41 -> A

While perhaps not as readable as the above, this version handles beyond-BMP codepoints by manually composing the utf-8 byte sequences and emitting raw bytes to the console. <tt>encoding convertto utf-8</tt> command still does the heavy lifting where it can.

 

scan $codepoint %llx cp

if {$cp < 0x10000} {

set utf8 [utf8 $codepoint]

puts "[format U+%04s $codepoint]\t$utf8\t[hexchars $utf8]"

 

{{out}}<pre>U+0041 A 41

U+1D11E 𝄞 f0 9d 84 9e

</pre>

 

=={{header|VBA}}==

Dim y() As Byte

Dim r As Long

Debug.Print String$(8 - Len(s), " "); s

Next cpi

A 41 41 41

ö F6 C3 B6 F6

€ 20AC E2 82 AC 20AC

? 1D11E F0 9D 84 9E 1D11E

</pre>

 

=={{header|zkl}}==

foreach utf,unicode_int in (T( T("\U41;",0x41), T("\Uf6;",0xf6),

T("\U416;",0x416), T("\U20AC;",0x20ac), T("\U1D11E;",0x1d11e))){

 

println("%s %s %9s %x".fmt(char,char2,"U+%x".fmt(unicode_int),utf_int));

Int.len() --> number of bytes in int. This could be hard coded because UTF-8

has a max of 6 bytes and (0x41).toBigEndian(6) --> 0x41,0,0,0,0,0 which is