Tokenize a string with escaping: Difference between revisions

{{trans|Python}}

 

[String] result

V token = ‘’

R result

 

 

{{out}}

=={{header|8080 Assembly}}==

 

jmp demo

;;; Routine to split a 0-terminated string

pfx: db '> $' ; Prefix to make the output more obvious

nl: db 13,10,'$'

 

{{out}}

 

=={{header|Action!}}==

 

TYPE Tokens=[

Tokenize("one^|uno||three^^^^|four^^^|^cuatro|",'|,'^,t)

PrintTokens(t)

{{out}}

[https://gitlab.com/amarok8bit/action-rosetta-code/-/raw/master/images/Tokenize_a_string_with_escaping.png Screenshot from Atari 8-bit computer]

 

=={{header|Ada}}==

with Ada.Containers.Indefinite_Vectors;

with Ada.Strings.Unbounded;

begin

Put_Vector (Split ("one^|uno||three^^^^|four^^^|^cuatro|"));

{{out}}

<pre>'one|uno'

 

=={{header|ALGOL 68}}==

# returns s parsed according to delimiter and escape #

PROC parse with escapes = ( STRING s, CHAR delimiter, escape )[]STRING:

[]STRING tokens = parse with escapes( "one^|uno||three^^^^|four^^^|^cuatro|", "|", "^" );

FOR t pos FROM LWB tokens TO UPB tokens DO print( ( "[", tokens[ t pos ], "]", newline ) ) OD

{{out}}

<pre>

 

{{Trans|JavaScript}}

 

-- tokenize :: String -> Character -> Character -> [String]

g

end if

{{Out}}

<pre>1: one|uno

 

=={{header|Arturo}}==

escaping: 0

 

 

str: "one^|uno||three^^^^|four^^^|^cuatro|"

 

{{out}}

 

=={{header|AutoHotkey}}==

for i,v in x:=StrSplit(StrReplace(StrReplace(StrReplace(s,e e,Chr(0xFFFE)),e d,Chr(0xFFFF)),e),d)

x[i]:=StrReplace(StrReplace(v,Chr(0xFFFE),e),Chr(0xFFFF),d)

return x

for i, v in Tokenize(str, "|", "^")

output .= i " : " v "`n"

{{out}}

<pre>1 : one|uno

 

=={{header|BBC BASIC}}==

PROC_tokenize("one^|uno||three^^^^|four^^^|^cuatro|", "|", "^")

END

NEXT

PRINT

{{out}}

<pre> 1 one|uno

4 four^|cuatro

5 </pre>

 

=={{header|C}}==

{{works with|C}}

 

#include <stdio.h>

 

 

return list;

 

{{Out}}

 

=={{header|C sharp}}==

using System.Text;

using System.Collections.Generic;

return result;

}

{{out}}

<pre>

 

=={{header|C++}}==

#include <stdexcept>

#include <string>

 

return 0;

{{out}}

<pre>one^|uno||three^^^^|four^^^|^cuatro|

 

=={{header|CLU}}==

escape: bool := false

part: array[char] := array[char]$[]

stream$putl(po, "\"" || part || "\"")

end

{{out}}

<pre>"one|uno"

 

=={{header|COBOL}}==

identification division.

program-id. 'tokenizewithescaping'.

.

end program 'tokenizewithescaping'.

 

{{out}}

 

=={{header|Common Lisp}}==

(flet ((make-string-buffer ()

(make-array 0 :element-type 'character :adjustable t :fill-pointer t)))

(defun main ()

(dolist (token (split "one^|uno||three^^^^|four^^^|^cuatro|" #\| #\^))

{{out}}

<pre>'one|uno'

=={{header|D}}==

{{trans|Java}}

 

void main() {

output.put(token.data.idup);

return output.data;

 

{{out}}

{{trans|C#}}

 

var buffer = []

var escaping = false

for c in this {

if escaping {

escaping = false

} else if c == escape {

escaping = true

} else if c == separator {

} else {

}

}

}

}

str

}

let testcase = "one^|uno||three^^^^|four^^^|^cuatro|";

print(": \(token)")

 

{{out}}

=={{header|Elena}}==

{{trans|C#}}

import extensions'routines;

import system'collections;

bool escaping := false;

{

if (escaping)

{

escaping := false

}

else

{

}

};

public program()

{

{{out}}

<pre>

 

=={{header|F_Sharp|F#}}==

open System.Text.RegularExpressions

 

|> Seq.map (unescape esc)

|> Seq.iter (fun s -> printfn "'%s'" s)

{{out}}

<pre>'one|uno'

This example uses Factor's <code>parser-combinators</code> vocabulary, which is modeled after Haskell's parser combinators. Page <tt>51</tt> of [https://bluishcoder.co.nz/factor-articles.pdf this pdf] contains a useful introduction to this vocabulary.

{{works with|Factor|0.99 2019-10-06}}

parser-combinators prettyprint sequences strings ;

 

 

"one^|uno||three^^^^|four^^^|^cuatro|"

{{out}}

<pre>

 

=={{header|Forth}}==

variable #src

variable offset

page

cr ." #### start ####" cr tokenize cr ." #### End ####" cr

{{output}}

<pre>

First Fortran (1958) offered no facilities for inspecting or manipulating text, until Fortran IV when the <code>A</code> format code was introduced whereby text could be read or written from numeric variables. The difficulties and incompatibilities between different computers were eased with F77 that offered CHARACTER*n variables, though they are not quite strings that have a varying length. F95 introduces the ability to define a compound entity such as a string and F2003 standardised a version of strings whereby with each assignment to such a variable, it would be re-allocated with the required amount of storage. Otherwise, one proceeds with CHARACTER variables and an associated variable containing its current length as with <code>TOKEN</code> and <code>L</code>. However, when passed to subroutines (or functions) as a parameter, a CHARACTER variable is supplied along with a secret additional parameter giving the size of the variable, and this is stringlike, so long as there is no need to change the length. Thus, the length of parameter TEXT to subroutine SPLIT can be found via LEN(TEXT).

 

CHARACTER*(*) TEXT !To be scanned.

CHARACTER*(1) SEP !The only separator for tokens.

CALL SPLIT("one^|uno||three^^^^|four^^^|^cuatro|","|","^")

 

 

The output has the text of the tokens marked >thus<

=={{header|FreeBASIC}}==

{{trans|Ring}}

Dim As Integer campo = 1

Dim As Boolean escapando = false

 

tokenize("one^|uno||three^^^^|four^^^|^cuatro|", "|", "^")

{{out}}

<pre>

 

=={{header|Go}}==

 

import (

fmt.Printf("Tokens: %q\n", tokens)

}

{{out}}

<pre>

=== Deterministic Finite Automaton ===

 

splitEsc sep esc = reverse . map reverse . snd . foldl process (0, [[]])

where process (st, r:rs) ch

| st == 0 && ch == sep = (0, []:r:rs)

| st == 1 && sep == esc && ch /= sep = (0, [ch]:r:rs)

 

{{out}}

Constant in space (~ O(k), where k -- is token length), as fast as DFA-based solution.

 

import Conduit

 

Just ch | notEsc && ch == esc -> go False b

| notEsc && ch == sep -> yield b >> go True []

 

This new conduit could be used in a pipeline as follows:

 

yieldMany "one^|uno||three^^^^|four^^^|^cuatro|"

.| splitEscC '|' '^'

 

<pre>λ> main

===Alternative===

This is essentially equivalent to the first (DFA) example, but, though possibly less elegant than the guard idiom, appears to be fractionally faster with larger (eg 180k) test strings.

 

------------------ TOKENIZE WITH ESCAPING ----------------

'|'

'^'

{{Out}}

<pre>"one|uno"

 

From the python example:

tokenize1=: tokenize =: '^|'&$: :(4 : 0)

'ESC SEP' =. x

RESULT =. RESULT , < TOKEN

)

<pre>

tokenize 'one^|uno||three^^^^|four^^^|^cuatro|'

Here's a somewhat more efficient approach (over 100 times faster on a 100k textual example):

 

'^|' tokenize2 y NB. task default escape and separator

:

T=. (#y){. 1,}.S NB. token beginnings

(T<;.1 K)#&.>T<;.1 y

 

Example use:

 

┌───────┬┬───────┬────────────┬┐

│one|uno││three^^│four^|cuatro││

 

 

splitTokens =: ((<,'|')&= <;._1 ])@:((<,'|'),])

removeExtra =: (}.^:(1<#)) L:0

 

tokenize t

 

$tokenize t

 

Relative efficiencies:

 

(%"1 <./) timespacex every 'tokenize1 txt';'tokenize2 txt';'tokenize3 txt'

132.856 1

1 7.73534

 

So tokenize2 is the fastest, while tokenize1 uses the least amount of memory. Also, tokenize1 is the slowest and tokenize3 uses the most memory. (First column is relative time used, second is relative space used, rows correspond to implementations.)

{{trans|Go}}

{{works with|Java|7}}

 

public class TokenizeStringWithEscaping {

return tokens;

}

 

<pre>[one|uno, , three^^, four^|cuatro, ]</pre>

===ES5===

====Iterative====

for (var a=[], t='', i=0, e=s.length; i<e; i+=1) {

var c = s.charAt(i)

var s = 'one^|uno||three^^^^|four^^^|^cuatro|'

document.write(s, '<br>')

{{out}}

<pre>one^|uno||three^^^^|four^^^|^cuatro|

 

====Functional====

'use strict';

 

.join('\n');

 

{{Out}}

<pre>one|uno

 

{{Trans|Haskell}} (Single fold version)

 

// tokenize :: String -> Character -> Character -> [String]

.map(show)

.join('\n');

 

{{Out}}

Defining the function as a composition of generics from a parser combinator library:

 

'use strict';

 

// MAIN ---

return main();

{{Out}}

<pre>[

=={{header|jq}}==

{{works with| jq|1.5}}

def tokenize(separator; escape):

 

| map( if type == "string" then split(escape) else . end)

| flatten

 

'''Example:'''

 

{{out}}

[

"one|uno",

"four^|cuatro",

""

 

=={{header|Julia}}==

{{trans|Kotlin}}

 

SPE = "\ufffe"

SPF = "\uffff"

end

 

 

{{out}}

 

=={{header|Kotlin}}==

 

const val SPE = "\ufffe" // unused unicode char in Specials block

val items = tokenize(str, sep, esc)

for (item in items) println(if (item.isEmpty()) "(empty)" else item)

 

{{out}}

 

=={{header|Lingo}}==

 

on tokenize (str, sep, esc)

end repeat

return str

 

sep = "|"

esc = "^"

put tokenize(str, sep, esc)

 

=={{header|Lua}}==

local strList, word, escaped, ch = {}, "", false

for pos = 1, #str do

for k, v in pairs(tokenise(testStr, testSep, testEsc)) do

print(k, v)

{{out}}

<pre>1 one|uno

 

=={{header|Mathematica}} / {{header|Wolfram Language}}==

Tokenize[str_String, escape_String : "^", sep_String : "|"] :=

Module[{results = {}, token = "", state = 0, a},

results

]

{{out}}

<pre>{"one|uno", "", "three^^", "four^|cuatro", ""}</pre>

 

=={{header|Nim}}==

 

proc tokenize(s: Stream, sep: static[char] = '|', esc: static[char] = '^'): seq[string] =

for i, s in tokenize(newStringStream "one^|uno||three^^^^|four^^^|^cuatro|"):

echo i, ":", s

{{out}}

<pre>0:one|uno

=={{header|OCaml}}==

 

let len = String.length s in

let buf = Buffer.create 16 in

end

in

 

Example:

 

=={{header|Perl}}==

The built-in <code>split</code> function can be used with a regex that matches the delimiter ''(although [http://perldoc.perl.org/perlre.html#Special-Backtracking-Control-Verbs advanced backtracking control verbs] are needed to skip escaped delimiters)'':

 

my ($string, $sep, $esc) = (shift, quotemeta shift, quotemeta shift);

my @fields = split /$esc . (*SKIP)(*FAIL) | $sep/sx, $string, -1;

return map { s/$esc(.)/$1/gsr } @fields;

 

A more traditional approach is to parse the input string step by step ''(using a repeatedly-matching regex of the form [http://perldoc.perl.org/perlretut.html#Global-matching <code>/\G.../g</code>])'', and throw away the separators ''(which can be done implicitly using [http://perldoc.perl.org/perlre.html#%28?%3C=pattern%29-\K \K])'':

 

 

In both cases, stripping the escape characters happens as a separate step.

Testing:

 

 

{{out}}

 

=={{header|Phix}}==

<span style="color: #008080;">function</span> <span style="color: #000000;">tokenize</span><span style="color: #0000FF;">(</span><span style="color: #004080;">string</span> <span style="color: #000000;">s</span><span style="color: #0000FF;">,</span> <span style="color: #004080;">integer</span> <span style="color: #000000;">sep</span><span style="color: #0000FF;">,</span> <span style="color: #004080;">integer</span> <span style="color: #000000;">esc</span><span style="color: #0000FF;">)</span>

<span style="color: #004080;">sequence</span> <span style="color: #000000;">ret</span> <span style="color: #0000FF;">=</span> <span style="color: #0000FF;">{}</span>

<span style="color: #0000FF;">?</span><span style="color: #000000;">tokenize</span><span style="color: #0000FF;">(</span><span style="color: #008000;">"one^|uno||three^^^^|four^^^|^cuatro|"</span><span style="color: #0000FF;">,</span><span style="color: #008000;">'|'</span><span style="color: #0000FF;">,</span><span style="color: #008000;">'^'</span><span style="color: #0000FF;">)</span>

{{Out}}

<pre>

 

=={{header|PicoLisp}}==

(split

(make

((= C Sep) (link 0))

(T (link C)) ) ) ) )

Test:

Output:

<pre>1: one|uno

 

=={{header|PowerShell}}==

function Split-String ([string]$String, [char]$Separator, [char]$Escape)

{

if ($String[-1] -eq $Separator) {[String]::Empty}

}

Split-String "one^|uno||three^^^^|four^^^|^cuatro|" -Separator "|" -Escape "^" | ForEach-Object `

-Begin {$n = 0} `

-Process {$n+= 1; "{0}: {1}" -f $n, $_}

{{Out}}

<pre>

=={{header|Python}}==

===Procedural===

'''

Issue python -m doctest thisfile.py to run the doctests.

state = 0

result.append(token)

 

===Functional===

{{Works with|Python|3}}

 

from functools import reduce

# MAIN ---

if __name__ == '__main__':

{{Out}}

<pre>['one|uno', '', 'three^^', 'four^|cuatro', '']</pre>

 

===Regex-based===

 

The python <code>re</code> library has a handy class <code>Scanner</code> which is intended precisely for this use-case.

 

The following code also illustrates an important feature of Python ‒ nested functions with closures.

 

 

STRING = 'one^|uno||three^^^^|four^^^|^cuatro|'

 

if __name__ == '__main__':

 

Output is the same as in the functional Python version above.

 

=={{header|Racket}}==

(require racket/match)

 

(report-input-output "|")

(report-input-output "^")

 

{{out}}

(formerly Perl 6)

 

return $string.match(/([ <!before $sep | $esc> . | $esc . ]*)+ % $sep/)\

.[0].map(*.subst: /$esc )> ./, '', :g);

}

 

 

{{out}}

=={{header|REXX}}==

===IF/THEN logic===

str = 'one^|uno||three^^^^|four^^^|^cuatro|' /*the character string to be tokenized.*/

esc = '^' /* " escape character to be used. */

exit /*stick a fork in it, we're all done. */

/*──────────────────────────────────────────────────────────────────────────────────────*/

'''output'''

<pre>

===SELECT logic===

This REXX version also shows a scale in the output.

str = 'one^|uno||three^^^^|four^^^|^cuatro|' /*the character string to be tokenized.*/

esc = '^' /* " escape character to be used. */

exit /*stick a fork in it, we're all done. */

/*──────────────────────────────────────────────────────────────────────────────────────*/

'''output'''

<pre>

 

=={{header|Ring}}==

tokenize("one^|uno||three^^^^|four^^^|^cuatro|", "|", "^")

 

next

see nl

Output:

<pre>

{{trans|Perl}}

 

def tokenize(string, sep, esc)

sep = Regexp.escape(sep)

p tokenize('one^|uno||three^^^^|four^^^|^cuatro|', '|', '^')

 

 

=={{header|Rust}}==

const ESCAPE: char = '^';

const STRING: &str = "one^|uno||three^^^^|four^^^|^cuatro|";

fn main() {

println!("{:#?}", tokenize(STRING));

{{out}}

<pre>

===Old fashioned Imperative===

Imperative with removed (ugly) mutable variables.

 

val (markerSpE,markerSpF) = ("\ufffe" , "\uffff")

 

tokenize(str, "|", "^").foreach(it => println(if (it.isEmpty) "<empty token>" else it))

 

===Idiomatic===

====Functional with Tail recursion====

 

object TokenizeStringWithEscaping1 extends App {

println(

f"[length:${it.length}%3d] ${if (it.isEmpty) "<empty token>" else it}"))

 

{{Out}}See it in running in your browser by [https://scalafiddle.io/sf/EsIjPQg/0 ScalaFiddle (JavaScript)] or by [https://scastie.scala-lang.org/O3DgMmuOSCS5DD6zQXK7MA Scastie (JVM)].

=={{header|Sidef}}==

{{trans|Perl}}

var fields = string.split(

Regex(esc.escape + '.(*SKIP)(*FAIL)|' + sep.escape, 's'), -1

tokenize("one^|uno||three^^^^|four^^^|^cuatro|", '|', '^').each { |str|

say str.dump

{{out}}

<pre>

 

=={{header|Simula}}==

SIMSET

BEGIN

 

END.

{{out}}

<pre>

PART4: 'FOUR^|CUATRO'

PART5: ''

</pre>

 

{{trans|Rust}}

 

func tokenize(separator: Character, escape: Character) -> [String] {

var token = ""

}

 

 

{{out}}

=={{header|Tcl}}==

Putting a coroutine in a TclOO object following the "generator pattern" gives a nice structure:

constructor {s} {

puts [coroutine Next my Iter $s]

}

 

 

{{out}}

=={{header|TMG}}==

Unix TMG:

char(esc) *

str: smark

ch: 0;

sep: 0;

 

Input:

 

=={{header|VBA}}==

Dim ret As New Collection

Dim this As String

Next i

Debug.Print Join(outstring, ", ")

<pre>one|uno, , three^^, four^|cuatro, </pre>

 

=={{header|Wren}}==

{{trans|Kotlin}}

var SPF = "\uffff" // ditto

 

var esc = "^"

var items = tokenize.call(str, sep, esc)

 

{{out}}

=={{header|zkl}}==

Two simplifying assumptions (since their behavior is undefined): A string ending with an un-escaped escape is an error and 0xff is not an allowed character in the string.

sink:=Sink(String);

foreach c in (str){

}

sink.close().split("\xff");

Or, if you prefer brevity:

sink:=Sink(String);

foreach c in (str){ sink.write( (c==esc and __cWalker.next()) or (c==sep and "\xff") or c ) }

sink.close().split("\xff");

{{out}}

<pre>L("one|uno","","three^^","four^|cuatro","")</pre>