remove_duplicate_lines does not seem to validate a good output

[RouxRC]

Hi, cool game!

Although I admit now that I've looked at the challenges.yaml file that my solution is quite big compared to a simple awk, I do not understand why my solution won't get validated for https://cmdchallenge.com/#/remove_duplicate_lines

cat faces.txt | while read l; do if ! grep "^$l$" .tmp > /dev/null 2> /dev/null; then echo "$l" >> .tmp; echo "$l"; fi; done

It does return a proper originally ordered output of the 24 unique faces though!

(◕‿◕)
(^̮^)
ʘ‿ʘ
ಠ_ಠ
ಠ⌣ಠ
ಠ‿ಠ
(ʘ‿ʘ)
(ಠ_ಠ)
¯_(ツ)_/¯
(ಠ⌣ಠ
ಠಠ⌣ಠ)
(ಠ‿ಠ)
٩◔̯◔۶
ヽ༼ຈل͜ຈ༽ﾉ
♥‿♥
◔̯◔
⊙﹏⊙
(¬_¬)
(；一_一)
(͡° ͜ʖ ͡°)
(° ͜ʖ °)
¯(°_o)/¯
( ﾟヮﾟ)
(︺︹︺)

[gripedthumbtacks]

yeah this one either unless i am dumb?

# You have a new challenge!
# Print the file faces.txt, but only print the first instance of each
# duplicate line, even if the duplicates don't appear next to each other.
# Note that order matters so don't sort the lines before removing duplicates.
# 
bash(0)> cat faces.txt | while read line; do md5="$(echo "${line}" | md5sum | cut -d ' ' -f 1)"; test -e "/tmp/zzz${md5}"; if [[ $? -eq 1 ]]; then echo "${line}"; touc
h "/tmp/zzz${md5}"; else touch "/tmp/zzz${md5}"; fi; done
(◕‿◕)
(^̮^)
ʘ‿ʘ
ಠ_ಠ
ಠ⌣ಠ
ಠ‿ಠ
(ʘ‿ʘ)
(ಠ_ಠ)
¯_(ツ)_/¯
(ಠ⌣ಠ
ಠಠ⌣ಠ)
(ಠ‿ಠ)
٩◔̯◔۶
ヽ༼ຈل͜ຈ༽ﾉ
♥‿♥
◔̯◔
⊙﹏⊙
(¬_¬)
(；一_一)
(͡° ͜ʖ ͡°)
(° ͜ʖ °)
¯(°_o)/¯
( ﾟヮﾟ)
(︺︹︺)
bash(0)>

[gripedthumbtacks]

This one line might be the issue?

"ಠ_ಠ"

[gripedthumbtacks]

what say you? proof by raw bytes?

$ rm zzz*; cat faces.txt | while read line; do md5="$(echo "${line}" | md5sum | cut -d ' ' -f 1)"; test -e "/tmp/zzz${md5}"; if [[ $? -eq 1 ]]; then echo "${line}"; touch "/tmp/zzz${md5}"; else touch "/tmp/zzz${md5}"; fi; done | while read unique_line; do echo "${unique_line}" | xxd -p; done       
28e29795e280bfe29795290a
285eccae5e290a
ca98e280bfca980a
e0b2a05fe0b2a00a
e0b2a0e28ca3e0b2a00a
e0b2a0e280bfe0b2a00a
28ca98e280bfca98290a
28e0b2a05fe0b2a0290a
c2af5f28e38384295f2fc2af0a
28e0b2a0e28ca3e0b2a00a
e0b2a0e0b2a0e28ca3e0b2a0290a
28e0b2a0e280bfe0b2a0290a
d9a9e29794ccafe29794dbb60a
e383bde0bcbce0ba88d984cd9ce0ba88e0bcbdefbe890a
e299a5e280bfe299a50a
e29794ccafe297940a
e28a99efb98fe28a990a
28c2ac5fc2ac290a
28efbc9be4b8805fe4b880290a
28cda1c2b020cd9cca9620cda1c2b0290a
28c2b020cd9cca9620c2b0290a
c2af28c2b05f6f292fc2af0a
2820efbe9fe383aeefbe9f290a
28efb8baefb8b9efb8ba290a

[gripedthumbtacks]

24 unique lines. i think the character encoding (utf8?) might be causing this to fail?

[gripedthumbtacks]

what do you think @RouxRC ?

[gripedthumbtacks]

i think our solutions may be correct, just a flawed verification check

[FirefighterBlu3]

awk 'seen[$0]++' faces.txt|awk '!seen[$0]++' should work but doesn't