I'm reading about binary formats, the ELF format for example, so suppose i have two binary files, one compiled as an ELF file and another as a COFF(Or another binary format), how the kernel handles this? i mean, when you execute the program, how linux knows how to handle each different format?? Has the kernel some interface which selects according the header of the binary, the correct code to handle each kind of binary??
1 Answer
As you said, the kernel detects the type of binary based on the header.
Different binary formats are registered using register_binfmt(). Take a look at the fs/binfmt_* files for the different implementations.
This is done by exec_binprm() - basically the meat of the execve syscall - (in fs/exec.c). It calls search_binary_handler(), which searches the registered format handlersto find one willing to handle the file.
CONFIG_BINFMT_ELFflag in.config. There is also so-called 'misc' format - special feature in kernel that allows binding any format (via extension or magic number) to user-space interpreters (like java virtual machine, or wine).