1

I would like to find the parent process for a large amount of small child processes performing small IO operations.

For example, consider the following python script:

import os import time import subprocess def high_tps_reader(file_path, num_reads_per_second, block_size): while True: for _ in range(num_reads_per_second): subprocess.run([ "dd", f"if={file_path}", "of=/dev/null", f"bs={block_size}", "count=1" ], stderr=subprocess.DEVNULL) if __name__ == __name__: file_path = "/tmp/testfile.txt" num_reads_per_second = 10000 block_size = "4K" # Read 4KB at a time if not os.path.exists(file_path): with open(file_path, "wb") as f: f.write(b"a" * 1024 * 1024) # Start the high TPS reader high_tps_reader(file_path, num_reads_per_second, block_size)

Saving it under /tmp/high_tps.py and running:

$ python3 /tmp/high_tps.py &

The script will run forever and spawn a very large number of small dd child processes, who themselves perform IO operations. The combined IO is therefore very large for the parent process.

However, I cannot seem to find a way to see that using standard tools like pidstat or iotop as they only report individual processes.

The following does not work:

# pidstat -d 1 5 # iotop -ao

Also, since the dd child processes almost instantly die, it is impossible for me to see them in the output for the above commands.

Improve this question
2
  • 1
    I am not sure if this concept of "combined I/O of the parent process" exists. Are you sure it does? I mean, everything is a child of PID 1, right? As far as my (very limited) knowledge goes, I/O is only linked to the specific process and isn't relevant to the parents. Commented Aug 24, 2024 at 18:28
  • @terdon it does, in the context of cgroups, where one parent process "shares" its IO accounting with all its children, by means of inheritance of the cgroup io controllers. But for that to be helpful here, the original process would need to start its own cgroup. Generally not a bad idea to give different services on a server or a desktop machine their own systemd scope, but if this parent process is just one of many in a cgroup, then that won't help. Commented Aug 24, 2024 at 18:55

1 Answer 1

Reset to default
0

Since per-process bookkeeping IO load is a thing on Linux, what you want really would just amount to building a process tree with each node carrying the individual IO rate of the respective process, and then doing a leaf-to-bottom sum of the individual IO rates (sum over the children, not the node itself).

Finally, you'd then walk the tree to find the "heaviest" nodes.

Note that these nodes will be all at the root – in the end, all processes are children of process 1, so that must have the highest cumulative sum.

Building that tree from userland is probably good enough in most cases. All you need to do is read all /proc/*/stat, go for the 1. field (pid) and 4. field (ppid) and use that to build a doubly-linked tree structure (in Python, since you seem to know python). This is written from the depths of my heart, no testing whatsoever happened. You find a bug, you keep it.

#!/usr/bin/env python3 # # This is licensed under European Public License 1.2 # https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12 from os import walk from dataclasses import dataclass @dataclass class process: pid: int ppid: int io_rate: int children_io: int children: set[int] unprocessed_children: set[int] nodes = dict() for dirpath, dirnames, filenames in walk("/proc"): # ignore the files, we only care about the directories for dir in dirnames: if not dir.isnumeric(): continue stat = open(f"/proc/{dir}/stat", "r", encoding="ascii").read() stat_fields = stat.split(" ") io_fields = dict() with open(f"/proc/{dir}/io", "r", encoding="ascii") as io: for line in io: key, val = line.split(":") if val.isnumeric(): io_fields[key] = int(val) node = process( pid=int(stat_fields[0]), ppid=int(stat_fields[3]), io_rate=io_fields.get("read_bytes", 0) + io_fields.get("write_bytes", 0), unprocessed_children=set(), children=set(), children_io=0, ) nodes[node.pid] = process # Fill in children for node in nodes: parent = nodes.get(node.ppid, None) if not parent: # parent has already died continue parent.children.add(node.pid) parent.unprocessed_children.add(node.pid) while True: # Find leaf nodes leafs = set(node for node in nodes if not node.unprocessed_children) if not leafs: # done! break for leaf in leafs: leaf.seen = True parent = nodes[leaf.ppid] parent.children_io += leaf.io_rate # if you also want to count the IO of children's children: parent.children_io += leaf.children_io parent.unprocessed_children.remove(leaf.pid) # print result level = 0 print(nodes) def print_node(level: int, node: process): print(f"{' '*level} {node.pid:5d}: {node.children_io}") def recurse(level: int, node: process): print_node(level, node) level += 1 for child in node.children: recurse(level, child) recurse(nodes[1]) # Start at init
Improve this answer
2
  • Is there a way I can get that information using any existing CLI tool? The dstat command is able to figure it out, but it is unfortunately no longer maintained Commented Aug 24, 2024 at 19:15
  • @emandret see my answer Commented Aug 24, 2024 at 19:55

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.