Questions tagged [uniq]
The uniq tag has no summary.
178 questions
0 votes
2 answers
96 views
pipe to uniq from a variable not showing the desired output
I have a pipeline using array jobs and need to change the number of inputs for some steps. I thought about testing uniq since the only part changing in my folders are the last four characters (the hap ...
- 387
1 vote
1 answer
130 views
How can I find duplicate lines among files?
I have a software module which contains some files with same pattern. <whitespaces>private static final long serialVersionUID = \dL; How can I find files with the same value? $ grep -R ...
- 564
1 vote
1 answer
209 views
Why is sorted uniq -c command showing duplicates
I am trying to count how many times I use a certain version of a library on my computer. For some reason, uniq -c is outputing duplicates, despite sorting it, and despite the sort order seeming in ...
- 171
-1 votes
2 answers
242 views
how to de-duplicate block (timestamp+command) from bash history?
I'm working with bash_history file containing blocks with the following format: #unixtimestamp\ncommand\n here's sample of the bash_history file: #1713308636 cat > ./initramfs/init << "...
- 33
0 votes
3 answers
104 views
Does a command exist that lists all the directories where a word appears in a file or directory name?
When I don't remember where a file or a folder is, I sometime use the locate command (that finds more occurrences, allow more candidates than a find, in my mind. But maybe I'm mistaking). But then ...
- 2,499
11 votes
1 answer
1k views
Use uniq to filter adjacent lines in pipeline
I'm trying to monitor theme changes using this command: dbus-monitor --session "interface='org.freedesktop.portal.Settings', member=SettingChanged" | grep -o "uint32 ." Output ...
- 235
1 vote
5 answers
541 views
Find and delete partially duplicate lines
https://www.domain.com/files/G5SPNDOF/AAA-1080p.mp4.html https://www.domain2.com/dl/G5SPNDOF/JHCGTS/AAA-1080p.mp4.html https://www.domain.com/files/ZQWL80BG/AAA-1080p.mp4.html https://www.domain.com/...
0 votes
0 answers
29 views
numeric sort with unique option does not show 0! [duplicate]
I have a file with many redundant numbers in each row. Imagine something like the below: echo "10 9 5 6 4 cell 3 2 0 7 0 1" > test When I use sort -un test I get the following output: ...
- 131
0 votes
2 answers
354 views
tar processing files multiple times with find -newer
I'm trying to use tar(1) to create an archive of files newer than a specific file (fileA). However, when I use find(1) to obtain the list of files to pass to tar, some files are listed multiple times: ...
- 1,659
4 votes
1 answer
11k views
Difference between sort -u and uniq -u
I always have been using sort -u to get rid of duplicates until now. But I am having a real doubt about a list generated by a software tool. The question is: is the output of sort -u |wc the same as ...
- 248
0 votes
3 answers
107 views
de-duplicate list but group parts of it
I am compiling some access rules from failed logins and after some piping I arrived at this: cat <<INPUT | sort -k 3,3 --unique Deny from 13.42.98.142 # demo Deny from 13.42.98.142 # test Deny ...
- 523
14 votes
2 answers
4k views
Who killed my sort? or How to efficient count distinct values from a csv column
I'm doing some processing trying to get how many different lines in a file containing 160,353,104 lines. Here is my pipeline and stderr output. $ tail -n+2 2022_place_canvas_history.csv | cut -d, -f2 |...
- 213
3 votes
5 answers
911 views
find duplicate 1st field and concat its values in single line
I have a file that has entries in key: value format like the below: cat data.txt name: 'tom' tom_age: '31' status_tom_mar: 'yes' school: 'anne' fd_year_anne: '1987' name: 'hmz' hmz_age: '21' ...
- 527
-1 votes
1 answer
132 views
Skip line from console if equal than line before, adding count (in realtime)
Using uniq it is possible to filter out sequential duplicate lines. while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq becomes: 1 2 1 Is there a way to have duplicated sequential ...
- 101
3 votes
2 answers
873 views
Output lines with the same md5 sum
I have a script like this find path -type f -exec md5sum {} +' It has this conclusion /tmp ❯ find $pwd -type f -exec md5sum {} + \a7c8252355166214d1f6cd47db917226 ./guess.bash ...
- 51