Tokenize a string with escaping: Difference between revisions

(Print the output list in any format you like, as long as it is it easy to see what the fields are.)

 

<br><br>

 

 

{{Trans|JavaScript}}

 

-- tokenize :: String -> Character -> Character -> [String]

script charParse

set blnEscChar to ((not blnEsc) and (x = chrEsc))

else

end if

end |λ|

end script

 

 

on run

script numberedLine

 

 

 

-- foldl :: (a -> b -> a) -> a -> [b] -> a

end tell

end foldl

 

-- Lift 2nd class handler function into 1st class script wrapper

end mReturn

 

set my text item delimiters to dlm

return xs

end splitOn

 

-- cond :: Bool -> a -> a -> a

g

end if

{{Out}}

<pre>1: one|uno

3: three^^

4: four^|cuatro

=={{header|Arturo}}==

 

 

prints chr

escaping: 0

print ""

 

{{out}}

 

three^^

 

{{out}}

 

=={{header|BBC BASIC}}==

PROC_tokenize("one^|uno||three^^^^|four^^^|^cuatro|", "|", "^")

END

NEXT

PRINT

{{out}}

<pre> 1 one|uno

4 four^|cuatro

5 </pre>

 

=={{header|C}}==

{{works with|C}}

 

#include <stdio.h>

 

 

return list;

 

{{Out}}

5.

</pre>

 

=={{header|C sharp}}==

using System.Text;

using System.Collections.Generic;

return result;

}

{{out}}

<pre>

: four^|cuatro

: </pre>

 

=={{header|COBOL}}==

identification division.

program-id. 'tokenizewithescaping'.

.

end program 'tokenizewithescaping'.

 

{{out}}

 

=={{header|Common Lisp}}==

(flet ((make-string-buffer ()

(make-array 0 :element-type 'character :adjustable t :fill-pointer t)))

(defun main ()

(dolist (token (split "one^|uno||three^^^^|four^^^|^cuatro|" #\| #\^))

{{out}}

<pre>'one|uno'

=={{header|D}}==

{{trans|Java}}

 

void main() {

output.put(token.data.idup);

return output.data;

 

{{out}}

{{trans|C#}}

 

var buffer = []

var escaping = false

for c in this {

if escaping {

escaping = false

} else if c == escape {

escaping = true

} else if c == separator {

} else {

}

}

}

}

str

}

print(": \(token)")

 

{{out}}

=={{header|Elena}}==

{{trans|C#}}

import extensions'routines;

import system'collections;

bool escaping := false;

{

if (escaping)

{

escaping := false

}

else

{

}

};

public program()

{

{{out}}

<pre>

 

=={{header|F_Sharp|F#}}==

open System.Text.RegularExpressions

 

|> Seq.map (unescape esc)

|> Seq.iter (fun s -> printfn "'%s'" s)

{{out}}

<pre>'one|uno'

This example uses Factor's <code>parser-combinators</code> vocabulary, which is modeled after Haskell's parser combinators. Page <tt>51</tt> of [https://bluishcoder.co.nz/factor-articles.pdf this pdf] contains a useful introduction to this vocabulary.

{{works with|Factor|0.99 2019-10-06}}

parser-combinators prettyprint sequences strings ;

 

 

"one^|uno||three^^^^|four^^^|^cuatro|"

{{out}}

<pre>

{ "one|uno" "" "three^^" "four^|cuatro" "" }

</pre>

 

=={{header|Fortran}}==

First Fortran (1958) offered no facilities for inspecting or manipulating text, until Fortran IV when the <code>A</code> format code was introduced whereby text could be read or written from numeric variables. The difficulties and incompatibilities between different computers were eased with F77 that offered CHARACTER*n variables, though they are not quite strings that have a varying length. F95 introduces the ability to define a compound entity such as a string and F2003 standardised a version of strings whereby with each assignment to such a variable, it would be re-allocated with the required amount of storage. Otherwise, one proceeds with CHARACTER variables and an associated variable containing its current length as with <code>TOKEN</code> and <code>L</code>. However, when passed to subroutines (or functions) as a parameter, a CHARACTER variable is supplied along with a secret additional parameter giving the size of the variable, and this is stringlike, so long as there is no need to change the length. Thus, the length of parameter TEXT to subroutine SPLIT can be found via LEN(TEXT).

 

CHARACTER*(*) TEXT !To be scanned.

CHARACTER*(1) SEP !The only separator for tokens.

CALL SPLIT("one^|uno||three^^^^|four^^^|^cuatro|","|","^")

 

 

The output has the text of the tokens marked >thus<

 

In this example the DO-loop relentlessly steps through the text, and in general this would not be convenient. Normally, token identification proceeds within a much larger context where one would not discard the token immediately after it is isolated, and rather than copying the text hither and thither, one might prefer to identify it in-place, say with variables <code>L1</code> and <code>L2</code> identifying the start and end positions within the working area. In such a case there would no longer be a need for a variable <code>TOKEN</code> and the angst of deciding on a suitable maximum size. This would also make it easier in any error messages to show context and provenance. However, the bizarre miscegnation of "escape" sequences (especially confusing within text ''literals''), means that the source text does not necessarily constitute the text of the token.

 

=={{header|Go}}==

 

import (

fmt.Printf("Tokens: %q\n", tokens)

}

{{out}}

<pre>

=== Deterministic Finite Automaton ===

 

splitEsc sep esc = reverse . map reverse . snd . foldl process (0, [[]])

where process (st, r:rs) ch

| st == 0 && ch == sep = (0, []:r:rs)

| st == 1 && sep == esc && ch /= sep = (0, [ch]:r:rs)

 

{{out}}

Constant in space (~ O(k), where k -- is token length), as fast as DFA-based solution.

 

import Conduit

 

Just ch | notEsc && ch == esc -> go False b

| notEsc && ch == sep -> yield b >> go True []

 

This new conduit could be used in a pipeline as follows:

 

yieldMany "one^|uno||three^^^^|four^^^|^cuatro|"

.| splitEscC '|' '^'

 

<pre>λ> main

===Alternative===

This is essentially equivalent to the first (DFA) example, but, though possibly less elegant than the guard idiom, appears to be fractionally faster with larger (eg 180k) test strings.

 

tokenize :: Char -> Char -> String -> [String]

where

(token, list, _) =

foldr

let literal = not aEsc

isEsc = literal && (x == esc)

([], [], False)

(reverse str)

 

main :: IO ()

{{Out}}

<pre>"one|uno"

 

From the python example:

tokenize1=: tokenize =: '^|'&$: :(4 : 0)

'ESC SEP' =. x

RESULT =. RESULT , < TOKEN

)

<pre>

tokenize 'one^|uno||three^^^^|four^^^|^cuatro|'

Here's a somewhat more efficient approach (over 100 times faster on a 100k textual example):

 

'^|' tokenize2 y NB. task default escape and separator

:

T=. (#y){. 1,}.S NB. token beginnings

(T<;.1 K)#&.>T<;.1 y

 

Example use:

 

┌───────┬┬───────┬────────────┬┐

│one|uno││three^^│four^|cuatro││

 

 

splitTokens =: ((<,'|')&= <;._1 ])@:((<,'|'),])

removeExtra =: (}.^:(1<#)) L:0

 

tokenize t

 

$tokenize t

 

Relative efficiencies:

 

(%"1 <./) timespacex every 'tokenize1 txt';'tokenize2 txt';'tokenize3 txt'

132.856 1

1 7.73534

 

So tokenize2 is the fastest, while tokenize1 uses the least amount of memory. Also, tokenize1 is the slowest and tokenize3 uses the most memory. (First column is relative time used, second is relative space used, rows correspond to implementations.)

{{trans|Go}}

{{works with|Java|7}}

 

public class TokenizeStringWithEscaping {

return tokens;

}

 

<pre>[one|uno, , three^^, four^|cuatro, ]</pre>

===ES5===

====Iterative====

for (var a=[], t='', i=0, e=s.length; i<e; i+=1) {

var c = s.charAt(i)

var s = 'one^|uno||three^^^^|four^^^|^cuatro|'

document.write(s, '<br>')

{{out}}

<pre>one^|uno||three^^^^|four^^^|^cuatro|

 

====Functional====

'use strict';

 

.join('\n');

 

{{Out}}

<pre>one|uno

 

===ES6===

{{Trans|Haskell}} (Single fold version)

 

// tokenize :: String -> Character -> Character -> [String]

.map(show)

.join('\n');

 

{{Out}}

"four^|cuatro"

""</pre>

 

=={{header|jq}}==

{{works with| jq|1.5}}

def tokenize(separator; escape):

 

| map( if type == "string" then split(escape) else . end)

| flatten

 

'''Example:'''

 

{{out}}

[

"one|uno",

"four^|cuatro",

""

 

=={{header|Julia}}==

{{trans|Kotlin}}

 

SPE = "\ufffe"

SPF = "\uffff"

end

 

 

{{out}}

 

=={{header|Kotlin}}==

 

const val SPE = "\ufffe" // unused unicode char in Specials block

val items = tokenize(str, sep, esc)

for (item in items) println(if (item.isEmpty()) "(empty)" else item)

 

{{out}}

 

=={{header|Lingo}}==

 

on tokenize (str, sep, esc)

end repeat

return str

 

sep = "|"

esc = "^"

put tokenize(str, sep, esc)

 

=={{header|Lua}}==

local strList, word, escaped, ch = {}, "", false

for pos = 1, #str do

for k, v in pairs(tokenise(testStr, testSep, testEsc)) do

print(k, v)

{{out}}

<pre>1 one|uno

4 four^|cuatro

5</pre>

 

=={{header|Nim}}==

 

var buff = ""

while not s.atEnd():

case c

of sep:

buff.add s.readChar

else:

result.add buff

 

echo i, ":", s

{{out}}

<pre>0:one|uno

=={{header|OCaml}}==

 

let len = String.length s in

let buf = Buffer.create 16 in

end

in

 

Example:

 

=={{header|Perl}}==

The built-in <code>split</code> function can be used with a regex that matches the delimiter ''(although [http://perldoc.perl.org/perlre.html#Special-Backtracking-Control-Verbs advanced backtracking control verbs] are needed to skip escaped delimiters)'':

 

my ($string, $sep, $esc) = (shift, quotemeta shift, quotemeta shift);

my @fields = split /$esc . (*SKIP)(*FAIL) | $sep/sx, $string, -1;

return map { s/$esc(.)/$1/gsr } @fields;

 

A more traditional approach is to parse the input string step by step ''(using a repeatedly-matching regex of the form [http://perldoc.perl.org/perlretut.html#Global-matching <code>/\G.../g</code>])'', and throw away the separators ''(which can be done implicitly using [http://perldoc.perl.org/perlre.html#%28?%3C=pattern%29-\K \K])'':

 

 

In both cases, stripping the escape characters happens as a separate step.

Testing:

 

 

{{out}}

''

</pre>

 

=={{header|Phix}}==

{{Out}}

<pre>

 

=={{header|PicoLisp}}==

(split

(make

((= C Sep) (link 0))

(T (link C)) ) ) ) )

Test:

Output:

<pre>1: one|uno

 

=={{header|PowerShell}}==

function Split-String ([string]$String, [char]$Separator, [char]$Escape)

{

if ($String[-1] -eq $Separator) {[String]::Empty}

}

Split-String "one^|uno||three^^^^|four^^^|^cuatro|" -Separator "|" -Escape "^" | ForEach-Object `

-Begin {$n = 0} `

-Process {$n+= 1; "{0}: {1}" -f $n, $_}

{{Out}}

<pre>

=={{header|Python}}==

===Procedural===

'''

Issue python -m doctest thisfile.py to run the doctests.

token = ''

state = 0

if state == 0:

if c == escape:

state = 0

result.append(token)

 

===Functional===

{{Works with|Python|3}}

 

from functools import reduce

# MAIN ---

if __name__ == '__main__':

{{Out}}

<pre>['one|uno', '', 'three^^', 'four^|cuatro', '']</pre>

 

=={{header|Racket}}==

(require racket/match)

 

(report-input-output "|")

(report-input-output "^")

 

{{out}}

Input: "."

Output: (".")</pre>

 

=={{header|REXX}}==

===IF/THEN logic===

str = 'one^|uno||three^^^^|four^^^|^cuatro|' /*the character string to be tokenized.*/

esc = '^' /* " escape character to be used. */

exit /*stick a fork in it, we're all done. */

/*──────────────────────────────────────────────────────────────────────────────────────*/

'''output'''

<pre>

===SELECT logic===

This REXX version also shows a scale in the output.

str = 'one^|uno||three^^^^|four^^^|^cuatro|' /*the character string to be tokenized.*/

esc = '^' /* " escape character to be used. */

exit /*stick a fork in it, we're all done. */

/*──────────────────────────────────────────────────────────────────────────────────────*/

'''output'''

<pre>

 

=={{header|Ring}}==

tokenize("one^|uno||three^^^^|four^^^|^cuatro|", "|", "^")

 

next

see nl

Output:

<pre>

{{trans|Perl}}

 

def tokenize(string, sep, esc)

sep = Regexp.escape(sep)

p tokenize('one^|uno||three^^^^|four^^^|^cuatro|', '|', '^')

 

 

=={{header|Rust}}==

const ESCAPE: char = '^';

const STRING: &str = "one^|uno||three^^^^|four^^^|^cuatro|";

fn main() {

println!("{:#?}", tokenize(STRING));

{{out}}

<pre>

===Old fashioned Imperative===

Imperative with removed (ugly) mutable variables.

 

val (markerSpE,markerSpF) = ("\ufffe" , "\uffff")

 

tokenize(str, "|", "^").foreach(it => println(if (it.isEmpty) "<empty token>" else it))

 

===Idiomatic===

====Functional with Tail recursion====

 

object TokenizeStringWithEscaping1 extends App {

println(

f"[length:${it.length}%3d] ${if (it.isEmpty) "<empty token>" else it}"))

 

{{Out}}See it in running in your browser by [https://scalafiddle.io/sf/EsIjPQg/0 ScalaFiddle (JavaScript)] or by [https://scastie.scala-lang.org/O3DgMmuOSCS5DD6zQXK7MA Scastie (JVM)].

=={{header|Sidef}}==

{{trans|Perl}}

var fields = string.split(

Regex(esc.escape + '.(*SKIP)(*FAIL)|' + sep.escape, 's'), -1

)

}

 

tokenize("one^|uno||three^^^^|four^^^|^cuatro|", '|', '^').each { |str|

say str.dump

<pre>

""

""

</pre>

 

=={{header|Tcl}}==

Putting a coroutine in a TclOO object following the "generator pattern" gives a nice structure:

constructor {s} {

puts [coroutine Next my Iter $s]

}

 

 

{{out}}

<pre>one|uno {} three^^ four^|cuatro {}</pre>

 

=={{header|VBA}}==

Dim ret As New Collection

Dim this As String

Next i

Debug.Print Join(outstring, ", ")

<pre>one|uno, , three^^, four^|cuatro, </pre>

 

=={{header|zkl}}==

Two simplifying assumptions (since their behavior is undefined): A string ending with an un-escaped escape is an error and 0xff is not an allowed character in the string.

sink:=Sink(String);

foreach c in (str){

}

sink.close().split("\xff");

Or, if you prefer brevity:

sink:=Sink(String);

foreach c in (str){ sink.write( (c==esc and __cWalker.next()) or (c==sep and "\xff") or c ) }

sink.close().split("\xff");

{{out}}

<pre>L("one|uno","","three^^","four^|cuatro","")</pre>