Questions tagged [large-files]
The large-files tag has no summary.
92 questions
2 votes
1 answer
312 views
Use GNU parallel with very long lines
I have a very large SQL dumpfile (30GB) that I need to edit (do some find/replace) before loading back into the database. Besides having a large size, the file also contains very long lines. Except ...
- 23
16 votes
5 answers
13k views
How do I read the last lines of a huge log file?
I have a log of 55GB in size. I tried: cat logfile.log | tail But this approach takes a lot of time. Is there any way to read huge files faster or any other approach?
- 162
2 votes
4 answers
6k views
EXT4 for very large (>1GB) files : increase block size, use block clusters, or both?
I'd like to format a 12 TB HDD (not SSD) with EXT4, in order to store large video files (each file being at least 1 GiB in size). I am working with an x86-64 (a.k.a. x64 or amd64) processor. There's ...
- 2,019
1 vote
1 answer
647 views
Transferring very large dataset from cluster to a storage server
We have to move a set of very large data (in petabytes) from HPC cluster to a storage server. We have a high capacity communication link between the devices. However, the bottleneck seems to be a fast ...
- 113
6 votes
4 answers
1k views
Delete huge directory that causes all commands to hang
How do I delete this large directory? stat session/ File: ‘session/’ Size: 321540096 Blocks: 628040 IO Block: 4096 directory Device: 903h/2307d Inode: 11149319 Links: 2 Access: ...
- 200
9 votes
2 answers
3k views
Is rsync --append able to resume an interrupted copy process without reading all the copied data?
I need to copy one very large file (3TB) on the same machine from one external drive to another. This might take (because of low bandwidth) many days. So I want to be prepared when I have to interrupt ...
- 659
3 votes
1 answer
9k views
Is cp faster than rsync during the first run?
I am currently using rsync to copy a 73GB file from a Samsung portable SSD T7 to an HPC cluster. rsync -avh path/to/dataset [email protected]:/path/to/dest The following applies: My local machine (...
- 135
2 votes
5 answers
2k views
How to compare huge files with progress information
In a Unix command line context I would like to compare two truly huge files (around 1TB each), preferable with a progress indicator. I have tried diff and cmp, and they both crashed the system (macOS ...
- 659
3 votes
0 answers
2k views
How to use less in a 'safe' way on files with huge lines?
I have a relatively large file, a minified json file which is about 4 Gigs. The file isn't huge...but a lot of programs choke on it because it's a single-lined file. I have noticed less works ok on it ...
- 359
1 vote
5 answers
590 views
Pipe skip 99 lines out of every 100
I have a bash commands pipeline that produces a ton of logging text output. But mostly it repeats the previous line except for the timestamp and some minor flags, the main output data changes only ...
- 121
2 votes
3 answers
2k views
Remove duplicated lines containing specific text
I have a big text file containing a million lines. I would like to find identical lines that match my specific text and leave the first occurrence intact. Any ideas? So the algorithm should roughly be ...
- 21
2 votes
1 answer
2k views
Split large file into chunks and delete original?
I'm having trouble with a large text file (30GB) I would like to create smaller files from it (5GB lets say) But sadly I don't have no more storage (only ~10G is available). This line: split -b 5g &...
0 votes
1 answer
622 views
How do you keep unison from syncing large files?
I am using unison file syncing software and I am aware of the config that ignores syncing files with specific regex or name. But is there a way to block syncing of large files e.g. larger than 10 MB? ...
- 28.6k
1 vote
0 answers
123 views
Grep something that is between two known strings
I have a large potentially zipped log file and I can identify which line number some text I'm interested in is on using: find . -name "*" -exec zgrep -C 1 -n -i -H TextToFind {} \; But in a second ...
- 11
1 vote
0 answers
208 views
I want a shared folder between Arch (host) and Android
I have a music folder of nearly 10 GB, of completely offline music. I want to keep it on my phone, so i can take it on the go. The simplest answer is a file transfer between my phone and PC, however ...
- 11