Timeline for Why is appending different streams to a file safe?
Current License: CC BY-SA 4.0
18 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 7, 2023 at 16:44 | comment | added | Sweet Shell O'Mine | Thank you very much! | |
| Feb 7, 2023 at 16:44 | vote | accept | Sweet Shell O'Mine | ||
| Feb 7, 2023 at 16:05 | comment | added | ilkkachu | @SweetShellO'Mine, no, the order of writes is clear. It's just with two separate redirections, it writes them to the same place so that the second overwrites the first. But if it writes the error message first, that one goes first and gets overwritten, not the other way around. If you have multiple processes running at the same time, they might get to do the writes in whatever order, essentially randomly. | |
| Feb 7, 2023 at 15:51 | comment | added | Sweet Shell O'Mine | @ikkachu Thank you. I'm not trying to be unnecessarily nitpicky, I'm really not understanding this. You said that "If there's only one process, there's no question of ordering". But in ls file.txt another_file.txt >out_err.txt 2>out_err.txt, there's only one process and a question of ordering (presumably, because the content of out_err.txt is what it is). | |
| Feb 7, 2023 at 15:30 | comment | added | Barmar | Beside kernel atomicity of write() there's also the issue of stdio buffering. Unless writing to a terminal, stdout is fully buffered while stderr is line-buffered. | |
| Feb 7, 2023 at 6:18 | comment | added | ilkkachu | @SweetShellO'Mine, a process is a running instance of a program, you have one there, that ls. The file descriptors / output streams / whatever aren't processes, they don't have code that runs. | |
| Feb 7, 2023 at 5:51 | comment | added | ilkkachu | Also I haven't really seen a hard specification, the Linux man page doesn't seem to say, but I haven't looked further. If you have a link, I'd be happy to see one. Anyway, I do suspect what you're saying is essentially right, that writes up to some sensible length are atomic. If they weren't, anything appending to some sort of a log file from two processes would be effed. | |
| Feb 7, 2023 at 5:49 | comment | added | ilkkachu | Section 2.9.7 Thread Interactions with Regular File Operations however says "All of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2017 when they operate on regular files or symbolic links: ... write(), writev()" so there appears to be an intent of atomicity, to some length? Though then again I found some article saying that's not the case for existing systems wrt. simultaneous writes and reads. Or maybe it only means threads within a single process? | |
| Feb 7, 2023 at 5:48 | comment | added | ilkkachu | @gntskn, it would be interesting to know how existing systems work in practice. The specification of write() goes to some lengths in describing how writes to a pipe or FIFO are atomic up to a certain length. That reads like an exception, but it doesn't really say there what the underlying rule is: if it's supposed to be not atomic at all for non-pipes, or atomic all the way. | |
| Feb 7, 2023 at 5:16 | comment | added | gntskn | Minor nitpick: not all writes are atomic, the kernel only guarantees atomicity up to a certain size (I forget which constant it is off the top of my head). For many writes it won’t matter, but for large writes you will still see races | |
| Feb 7, 2023 at 2:43 | comment | added | Jasen | more info about file position can be found in the lseek(2) man page | |
| Feb 7, 2023 at 2:41 | comment | added | Jasen | I'm fairly sure that write to files are guaranteed atomic up-to the kernel buffer size (8K) writes to ttys are diferent, I'm not sure about writes to sockets, but I think pipes are also atomic to 8k. | |
| Feb 7, 2023 at 0:56 | comment | added | Sweet Shell O'Mine | Thanks, ikkachu. Is there only one process when using >> and more than one when using > in my example? | |
| Feb 6, 2023 at 20:23 | comment | added | ilkkachu | (Technically, write() can return after writing just some but not all of the given data, but in practice I'd be surprised if that happens when writing to a regular file. If it does happen, it's the job of the program to try again with the rest of the data, and anyway, with just one process involved, there still wouldn't be interleaving. Unless the program goes out of its way to do that on purpose.) | |
| Feb 6, 2023 at 20:20 | comment | added | ilkkachu | @SweetShellO'Mine, the output streams don't do anything them, it's the process making the system calls that causes the output. If ls calls write(1, "foo", 3), those three bytes get written to stdout as one unit. If there's only one process, there's no question of ordering. If there was more than one, scheduling might after how their system calls are interleaved, but the data written in one call should still go as one atomic unit. | |
| Feb 6, 2023 at 20:10 | history | edited | ilkkachu | CC BY-SA 4.0 | added 438 characters in body |
| Feb 6, 2023 at 14:27 | comment | added | Sweet Shell O'Mine | Thank you, this is helpful. Can you elaborate on how the reading and writing positions work? In my ignorant eyes, it is conceivable that while appending writes to the end of the file, both streams may be being writting portions of data to the end of the file one at a time. For example, in the example I now added to the question, stdout may write fi to the end of the file, then stderr may write ls: to the end of the file (resulting in the file starting with fils:) and so on and so forth. | |
| Feb 6, 2023 at 14:17 | history | answered | ilkkachu | CC BY-SA 4.0 |