XXXX redacted: Difference between revisions

Content added Content deleted

Inline

Revision as of 03:24, 30 March 2020

You've been given a contract from a three letter abbreviation government agency. They want a program to automatically redact sensitive information from documents to be released to the public. They want fine control over what gets redacted though.

Given a piece of free-form, possibly Unicode text, (assume text only, no markup or formatting codes) they want to be able to redact: whole words, (case sensitive or insensitive) or partial words, (case sensitive or insensitive). Further, they want the option to "overkill" redact a partial word. Overkill redact means if the word contains the redact target, even if is only part of the word, redact the entire word.

For our purposes, a "word" here, means: a character group, separated by white-space and possibly punctuation; not necessarily strictly alphabetic characters. To "redact" a word or partial word, replace each character in the redaction target with a capital letter 'X'. There should be the same number of graphemes in the final redacted word as there were in the non-redacted word.

Task

Write a procedure to "redact" a given piece of text. Your procedure should take the text (or a link to it), the redaction target (or a link to it) and the redaction options. It need not be a single routine, as long as there is some way to programmatically select which operation will be performed. It may be invoked from a command line or as an internal routine, but it should be separately invokable, not just a hard coded block.

The given strings are enclosed in square brackets to denote them. The brackets should not be counted as part of the strings.

Using the test string: [Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.]

Show the redacted sentence for each of the redaction target strings [Tom] & [tom] using the following options:

Whole word
Whole word, Case insensitive
Partial word
Partial word, Case insensitive
Partial word, Overkill
Partial word, Case insensitive, Overkill

Note that some combinations don't, or at least, shouldn't really differ from less specific combination. E.G. "Whole word, Overkill" should be theoretically be exactly the same as "Whole word".

Extra kudos for not including adjoining punctuation during "Overkill" redaction.

Extra kudos if the redaction target can contain non-letter characters.

The demo strings use the abbreviations w/p for whole/partial word, i/s for case insensitive/sensitive, n/o for normal/overkill. You are not required to use those, or any abbreviation. They are just for display, though may be useful to show what operation you are intending to perform.

Ideal expected output (adjoining punctuation untouched):

   Redact 'Tom':
   [w|s|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
   [w|i|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
   [p|s|n] XXX? XXXs bottom tomato is in his stomach while playing the "XXX-tom" brand tom-toms. That's so tom.
   [p|i|n] XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
   [p|s|o] XXX? XXXX bottom tomato is in his stomach while playing the "XXXXXXX" brand tom-toms. That's so tom.
   [p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.

   Redact 'tom':
   [w|s|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
   [w|i|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
   [p|s|n] Tom? Toms botXXX XXXato is in his sXXXach while playing the "Tom-XXX" brand XXX-XXXs. That's so XXX.
   [p|i|n] XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
   [p|s|o] Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
   [p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.

Stretch

Complex Unicode: Using the test string: [🧑 👨 🧔 👨‍👩‍👦] and the redaction strings: [👨] and [👨‍👩‍👦]

Show the redacted strings when using the option "Whole word" (Case sensitivity shouldn't matter.) A single grapheme should be replaced by a single 'X'.

                  🧑 👨 🧔 👨‍👩‍👦
   Redact '👨' [w] 🧑 X 🧔 👨‍👩‍👦
   Redact '👨‍👩‍👦' [w] 🧑 👨 🧔 X

Go

Just the basic part of the task for now and, even there, I've had to resort to some rather crude hacks to deal with words containing hyphens and apostrophes as Go's regular expression engine (which is based on RE2) is not as sophisticated as those found in some other languages. <lang go>package main

import (

   "fmt"
   "regexp"
   "strings"
   "unicode/utf8"

)

func redact(text, word, opts string) {

   // temporarily replace hyphen with underscore so former not treated as a word boundary
   text = strings.ReplaceAll(text, "-", "_")
   // similarly replace apostrophe with '9' (yeah I know!) 
   text = strings.ReplaceAll(text, "'", "9")
   var exp string
   switch opts {
   case "[w|s|n]":
       exp = `\b` + word + `\b`
   case "[w|i|n]":
       exp = `(?i)\b` + word + `\b`
   case "[p|s|n]":
       exp = word
   case "[p|i|n]":
       exp = `(?i)` + word
   case "[p|s|o]", "[w|s|o]":
       exp = `\b\w*` + word + `\w*\b`
   case "[p|i|o]", "[w|i|o]":
       exp = `(?i)\b\w*` + word + `\w*\b`
   }
   rgx := regexp.MustCompile(exp)
   if opts[5] == 'n' {
       repl := strings.Repeat("X", utf8.RuneCountInString(word))
       text = rgx.ReplaceAllLiteralString(text, repl)
   } else {
       rf := func(match string) string {
           return strings.Repeat("X", utf8.RuneCountInString(match))
       }
       text = rgx.ReplaceAllStringFunc(text, rf)
   }
   // replace any remaining underscores with hyphens and '9's with apostrophes
   text = strings.ReplaceAll(text, "_", "-")
   text = strings.ReplaceAll(text, "9", "'")
   fmt.Printf("%s %s\n\n", opts, text)

}

func main() {

   text := `Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.

'Tis very tomish, don't you think?`

   allOpts := []string{"[w|s|n]", "[w|i|n]", "[p|s|n]", "[p|i|n]", "[p|s|o]", "[p|i|o]"}
   for _, word := range []string{"Tom", "tom", "t"} {
       fmt.Printf("Redact '%s':\n", word)
       for _, opts := range allOpts {
           redact(text, word, opts)
       }
       fmt.Println()
   }

}</lang>

Output:

Redact 'Tom':
[w|s|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[w|i|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?

[p|s|n] XXX? XXXs bottom tomato is in his stomach while playing the "XXX-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[p|i|n] XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?

[p|s|o] XXX? XXXX bottom tomato is in his stomach while playing the "XXXXXXX" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXX, don't you think?


Redact 'tom':
[w|s|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?

[w|i|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?

[p|s|n] Tom? Toms botXXX XXXato is in his sXXXach while playing the "Tom-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?

[p|i|n] XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?

[p|s|o] Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXX, don't you think?

[p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXX, don't you think?


Redact 't':
[w|s|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[w|i|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[p|s|n] Tom? Toms boXXom XomaXo is in his sXomach while playing Xhe "Tom-Xom" brand Xom-Xoms. ThaX's so Xom.
'Tis very Xomish, don'X you Xhink?

[p|i|n] Xom? Xoms boXXom XomaXo is in his sXomach while playing Xhe "Xom-Xom" brand Xom-Xoms. XhaX's so Xom.
'Xis very Xomish, don'X you Xhink?

[p|s|o] Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing XXX "XXXXXXX" brand XXXXXXXX. XXXXXX so XXX.
'Tis very XXXXXX, XXXXX you XXXXX?

[p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing XXX "XXXXXXX" brand XXXXXXXX. XXXXXX so XXX.
XXXX very XXXXXX, XXXXX you XXXXX?

Phix

Written on the assumption that overkill implies partial (see talk page). <lang Phix>enum WHOLE,PARTIAL,OVERKILL,INSENSITIVE constant spunc = " \r\n\t.?\"" -- spaces and punctuation

function redact(string text, word, integer options)

   sequence t_utf32 = utf8_to_utf32(text),
            l_utf32 = t_utf32,
            w_utf32 = utf8_to_utf32(word)
   string opt = "[?|s]"
   if options>INSENSITIVE then
       options -= INSENSITIVE
       opt[4] = 'i'
       l_utf32 = lower(t_utf32)
       w_utf32 = lower(w_utf32)
   end if
   opt[2] = "wpo"[options]
   integer idx = 1
   while true do
       idx = match(w_utf32,l_utf32,idx)
       if idx=0 then exit end if
       integer edx = idx+length(w_utf32)-1
       if options=WHOLE then
           if (idx=1 or find(l_utf32[idx-1],spunc))
           and (edx=length(l_utf32) or find(l_utf32[edx+1],spunc)) then
               t_utf32[idx..edx] = 'X'
           end if
       elsif options=PARTIAL
          or options=OVERKILL then
           if options=OVERKILL then
               while idx>1 and not find(l_utf32[idx-1],spunc) do idx -= 1 end while
               while edx<length(l_utf32) and not find(l_utf32[edx+1],spunc) do edx += 1 end while
           end if
           t_utf32[idx..edx] = 'X'
       end if
       idx = edx+1
   end while

--hmmm...

   text = iff(string(t_utf32)?t_utf32:utf32_to_utf8(t_utf32))
   return {opt,text}

end function

constant test = ` Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom. 'Tis very tomish, don't you think?`,

        tests = {"Tom","tom","t"}

for t=1 to length(tests) do

   printf(1,"Redact %s:\n",{tests[t]})
   for o=WHOLE to OVERKILL do
       printf(1,"%s:%s\n",redact(test,tests[t],o))
       printf(1,"%s:%s\n",redact(test,tests[t],o+INSENSITIVE))
   end for

end for constant ut = "🧑 👨 🧔 👨‍👩‍👦", fmt = """

%s

Redact 👨 %s %s Redact 👨‍👩‍👦 %s %s """ printf(1,fmt,{ut}&redact(ut,"👨",WHOLE)&redact(ut,"👨‍👩‍👦",WHOLE))</lang>

Output:

If you scroll down you'll see the windows console makes a complete mockery of those unicode characters (it should work unaltered on linux...) and it has replaced those right-hand combined characters with five X, but hey, at least it compiled cleanly and didn't crash!

Redact Tom:
[w|s]:XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?
[w|i]:XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?
[p|s]:XXX? XXXs bottom tomato is in his stomach while playing the "XXX-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?
[p|i]:XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?
[o|s]:XXX? XXXX bottom tomato is in his stomach while playing the "XXXXXXX" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?
[o|i]:XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXXX don't you think?
Redact tom:
[w|s]:Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?
[w|i]:XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?
[p|s]:Tom? Toms botXXX XXXato is in his sXXXach while playing the "Tom-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?
[p|i]:XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?
[o|s]:Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXXX don't you think?
[o|i]:XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXXX don't you think?
Redact t:
[w|s]:Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?
[w|i]:Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?
[p|s]:Tom? Toms boXXom XomaXo is in his sXomach while playing Xhe "Tom-Xom" brand Xom-Xoms. ThaX's so Xom.
'Tis very Xomish, don'X you Xhink?
[p|i]:Xom? Xoms boXXom XomaXo is in his sXomach while playing Xhe "Xom-Xom" brand Xom-Xoms. XhaX's so Xom.
'Xis very Xomish, don'X you Xhink?
[o|s]:Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing XXX "XXXXXXX" brand XXXXXXXX. XXXXXX so XXX.
'Tis very XXXXXXX XXXXX you XXXXX?
[o|i]:XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing XXX "XXXXXXX" brand XXXXXXXX. XXXXXX so XXX.
XXXX very XXXXXXX XXXXX you XXXXX?
               ƒºæ ƒæ¿ ƒºö ƒæ¿ÔÇìƒæ®ÔÇìƒæª
Redact ƒæ¿ [w|s] ƒºæ X ƒºö ƒæ¿ÔÇìƒæ®ÔÇìƒæª
Redact ƒæ¿ÔÇìƒæ®ÔÇìƒæª [w|s] ƒºæ ƒæ¿ ƒºö XXXXX

Raku

Works with: Rakudo version 2020.02

<lang perl6>sub redact ( Str $str, Str $redact, :i(:$insensitive) = False, :p(:$partial) = False, :o(:$overkill) = False ) {

   my $rx =
       $insensitive ??
           $partial ??
          $overkill ?? rx/:i <?after ^ | <:Po> | \s > (<[\w<:!Po>-]>*? [\w*\']? $redact [\w*\'\w+]? \S*?) <?before $ | <:Po> | \s > / #'
                    !! rx/:i ($redact) /
                    !! rx/:i <?after ^ | [\s<:Po>] | \s > ($redact) <?before $ | <:Po> | \s > /
                    !!
           $partial ??
          $overkill ?? rx/ <?after ^ | <:Po> | \s > (<[\w<:!Po>-]>*? [\w*\']? $redact [\w*\'\w+]? \S*?) <?before $ | <:Po> | \s > / #'
                    !! rx/ ($redact) /
                    !! rx/ <?after ^ | [\s<:Po>] | \s > ($redact) <?before $ | <:Po> | \s > /
   ;
   $str.subst( $rx, {'X' x $0.chars}, :g )

}

my %*SUB-MAIN-OPTS = :named-anywhere;

Operate on a given file with the given parameters

multi MAIN (

   Str $file,    #= File name with path
   Str $target,  #= Redact target text string
   :$i = False,  #= Case insensitive flag
   :$p = False,  #= Partial words flag
   :$o = False   #= Overkill flag
 ) { put $file.IO.slurp.&redact( $target, :i($i), :p($p), :o($o) ) }

Operate on the internal strings / parameters

multi MAIN () {

TESTING

   my $test = q:to/END/;
       Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
       'Tis very tomish, don't you think?
       END
       #'

   for 'Tom', 'tom', 't' -> $redact {
       say "\nRedact '$redact':";
       for '[w|s|n]', $redact, {},
           '[w|i|n]', $redact, {:i},
           '[p|s|n]', $redact, {:p},
           '[p|i|n]', $redact, {:p, :i},
           '[p|s|o]', $redact, {:p, :o},
           '[p|i|o]', $redact, {:p, :i, :o}
       -> $option, $str, %opts { printf "%s %s\n", $option, $test.&redact($str, |%opts) }
   }

   my $emoji = '🧑 👨 🧔 👨‍👩‍👦';
   printf "%20s %s\n", , $emoji;
   printf "%20s %s\n", "Redact '👨' [w]", $emoji.&redact('👨');
   printf "%20s %s\n", "Redact '👨‍👩‍👦' [w]", $emoji.&redact('👨‍👩‍👦');

   # Even more complicated Unicode

   $emoji = 'Argentina🧑🇦🇹  France👨🇫🇷  Germany🧔🇩🇪  Netherlands👨‍👩‍👦🇳🇱';
   printf "\n%20s %s\n", , $emoji;
   printf "%20s %s\n", "Redact '👨' [p]", $emoji.&redact('👨', :p);
   printf "%20s %s\n", "Redact '👨‍👩‍👦' [p]", $emoji.&redact('👨‍👩‍👦', :p);
   printf "%20s %s\n", "Redact '👨' [p|o]", $emoji.&redact('👨', :p, :o);
   printf "%20s %s\n", "Redact '👨‍👩‍👦' [p|o]", $emoji.&redact('👨‍👩‍👦', :p, :o);

}</lang>

Output:

Redact 'Tom':
[w|s|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[w|i|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?

[p|s|n] XXX? XXXs bottom tomato is in his stomach while playing the "XXX-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[p|i|n] XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?

[p|s|o] XXX? XXXX bottom tomato is in his stomach while playing the "XXXXXXX" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXX, don't you think?


Redact 'tom':
[w|s|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?

[w|i|n] XXX? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so XXX.
'Tis very tomish, don't you think?

[p|s|n] Tom? Toms botXXX XXXato is in his sXXXach while playing the "Tom-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?

[p|i|n] XXX? XXXs botXXX XXXato is in his sXXXach while playing the "XXX-XXX" brand XXX-XXXs. That's so XXX.
'Tis very XXXish, don't you think?

[p|s|o] Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXX, don't you think?

[p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing the "XXXXXXX" brand XXXXXXXX. That's so XXX.
'Tis very XXXXXX, don't you think?


Redact 't':
[w|s|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[w|i|n] Tom? Toms bottom tomato is in his stomach while playing the "Tom-tom" brand tom-toms. That's so tom.
'Tis very tomish, don't you think?

[p|s|n] Tom? Toms boXXom XomaXo is in his sXomach while playing Xhe "Tom-Xom" brand Xom-Xoms. ThaX's so Xom.
'Tis very Xomish, don'X you Xhink?

[p|i|n] Xom? Xoms boXXom XomaXo is in his sXomach while playing Xhe "Xom-Xom" brand Xom-Xoms. XhaX's so Xom.
'Xis very Xomish, don'X you Xhink?

[p|s|o] Tom? Toms XXXXXX XXXXXX is in his XXXXXXX while playing XXX "XXXXXXX" brand XXXXXXXX. XXXXXX so XXX.
'Tis very XXXXXX, XXXXX you XXXXX?

[p|i|o] XXX? XXXX XXXXXX XXXXXX is in his XXXXXXX while playing XXX "XXXXXXX" brand XXXXXXXX. XXXXXX so XXX.
XXXX very XXXXXX, XXXXX you XXXXX?

                     🧑 👨 🧔 👨‍👩‍👦
      Redact '👨' [w] 🧑 X 🧔 👨‍👩‍👦
      Redact '👨‍👩‍👦' [w] 🧑 👨 🧔 X

                     Argentina🧑🇦🇹  France👨🇫🇷  Germany🧔🇩🇪  Netherlands👨‍👩‍👦🇳🇱
      Redact '👨' [p] Argentina🧑🇦🇹  FranceX🇫🇷  Germany🧔🇩🇪  Netherlands👨‍👩‍👦🇳🇱
      Redact '👨‍👩‍👦' [p] Argentina🧑🇦🇹  France👨🇫🇷  Germany🧔🇩🇪  NetherlandsX🇳🇱
    Redact '👨' [p|o] Argentina🧑🇦🇹  XXXXXXXX  Germany🧔🇩🇪  Netherlands👨‍👩‍👦🇳🇱
    Redact '👨‍👩‍👦' [p|o] Argentina🧑🇦🇹  France👨🇫🇷  Germany🧔🇩🇪  XXXXXXXXXXXXX