Manipulating files in Unix

Usually, reading and writing files is done best using the standard C library functions. However, in various occasions we need a more low-level to the files. For example, we cannot check file permissions or file size using the standard C library. Also, you will see that Unix treats various devices in a similar manner to using files, and using the same functions you can read from a file, from a network connection and so on. Thus, it is useful to learn this generic interface.

The Little File Descriptor That Could

The basic system object used to manipulate files is called a file descriptor. This is an integer number that is used by the various I/O system calls to access a memory area containing data about the open file. This memory area has a similar role to the FILE structure in the standard C library I/O functions, and thus the pointer returned from fopen() has a role similar to a file descriptor.

Each process has its own file descriptors table, with each entry pointing to a an entry in a system file descriptor table. This allows several processes to share file descriptors, by having a table entry pointing to the same entry in the system file descriptors table. You will encounter this phenomena, and how it can be used, when learning about multi-process programming.

The value of the file descriptor is a non-negative integer. Usually, three file descriptors are automatically opened by the shell that started the process. File descriptor '0' is used for the standard input of the process. File descriptor '1' is used for the standard output of the process, and file descriptor '2' is used for the standard error of the process. Normally the standard input gets input from the keyboard, while standard output and standard error write data to the terminal from which the process was started.

Opening And Closing File Descriptors

Opening files using the system call interface is done using the open() system call. Similar to fopen(), it accepts two parameters. One containing the path to the file to open, the other contains the mode in which to open the file. The mode may be any of the following:

O_RDONLY: Open the file in read-only mode.
O_WRONLY: Open the file in write-only mode.
O_RDWR: Open the file for both reading and writing.

In addition, any of the following flags may be OR-ed with the mode flag:

O_CREAT: If the file does not exist already - create it.
O_EXCL: If used together with O_CREAT, the call will fail if the file already exists.
O_TRUNC: If the file already exists, truncate it (i.e. erase its contents).
O_APPEND: Open the file in append mode. Any data written to the file is appended at the end of the file.
O_NONBLOCK (or O_NDELAY): If any operation on the file is supposed to cause the calling process block, the system call instead will fail, and errno be set to EAGAIN. This requires caution on the part of the programmer, to handle these situations properly.
O_SYNC: Open the file in synchronous mode. Any write operation to the file will block until the data is written to disk. This is useful in critical files (such as database files) that must always remain in a consistent state, even if the system crashes in the middle of a file operation.

Unlike the fopen() function, open() accepts one more (optional) parameter, which defines the access permissions that will be given to the file, in case of file creation. This parameter is a combination of any of the following flags:

S_IRWXU: Owner of the file has read, write and execute permissions to the file.
S_IRUSR: Owner of the file has read permission to the file.
S_IWUSR: Owner of the file has write permission to the file.
S_IXUSR: Owner of the file has execute permission to the file.
S_IRWXG: Group of the file has read,write and execute permissions to the file.
S_IRGRP: Group of the file has read permission to the file.
S_IWGRP: Group of the file has write permission to the file.
S_IXGRP: Group of the file has execute permission to the file.
S_IRWXO: Other users have read,write and execute permissions to the file.
S_IROTH: Other users have read permission to the file.
S_IWOTH: Other users have write permission to the file.
S_IXOTH: Other users have execute permission to the file.

Here are a few examples of using open():


/* these hold file descriptors returned from open(). */
int fd_read;
int fd_write;
int fd_readwrite;
int fd_append;

/* Open the file /etc/passwd in read-only mode. */
fd_read = open("/etc/passwd", O_RDONLY);
if (fd_read < 0) {
    perror("open");
    exit(1);
}

/* Open the file run.log (in the current directory) in write-only mode. */
/* and truncate it, if it has any contents.                             */
fd_write = open("run.log", O_WRONLY | O_TRUNC);
if (fd_write < 0) {
    perror("open");
    exit(1);
}

/* Open the file /var/data/food.db in read-write mode. */
fd_readwrite = open("/var/data/food.db", O_RDWR);
if (fd_readwrite < 0) {
    perror("open");
    exit(1);
}

/* Open the file /var/log/messages in append mode. */
fd_append = open("/var/log/messages", O_WRONLY | O_APPEND);
if (fd_append < 0) {
    perror("open");
    exit(1);
}

Once we are done working with a file, we need to close it, using the close() system call, as follows:


if (close(fd) == -1) {
    perror("close");
    exit(1);
}

This will cause the file to be closed. Note that no buffering is normally associated with files opened with open(), so no buffer flushing is required.

Note: If a file that is currently open by a Unix process is being erased (using the Unix "rm" command, for example), the file is not really removed from the disk. Only when the process (or all processes) holding the file open, the file is physically removed from the disk. Until then it is just removed from its directory, not from the disk.

Reading From A File Descriptor

Once we got a file descriptor to an open file (that was opened in read mode), we may read data from the file using the read() system call. This call takes three parameters: the file descriptor to read from, a buffer to read data into, and the number of characters to read into the buffer. The buffer must be large enough to contain the data. Here is how to use this call. We assume 'fd' contains a file descriptor returned from a previous call to open().


/* return value from the read() call. */
size_t rc;
/* buffer to read data into.          */
char buf[20];

/* read 20 bytes from the file.       */
rc = read(fd, buf, 20);
if (rc == 0) {
    printf("End of file encountered\n");
}
else if (rc < 0) {
    perror("read");
    exit(1);
}
else {
    printf("read in '%d' bytes\n", rc);
}

As you can see, read() does not always read the number of bytes we asked it to read. This could be due to a signal interrupting it in the middle, or the end of the file was encountered. In such a case, read() returns the number of bytes it actually read.

Writing Into A File Descriptor

Just like we used read() to read from the file, we use the write() system call, to write data to the file. The write operations is done in the location of the current read/write pointer of the given file, much like the various standard C library output functions did. write() gets the same parameters as read() does, and just like read(), might write only part of the data to the given file, if interrupted in the middle, or for other reasons. In such a case it will return the number of bytes actually written to the file. Here is a usage example:


/* return value from the write() call. */
size_t rc;

/* write the given string to the file. */
rc = write(fd, "hello world\n", strlen("hello world\n"));
if (rc < 0) {
    perror("write");
    exit(1);
}
else {
    printf("wrote in '%d' bytes\n", rc);
}

As you can see, there is never an end-of-file case with a write operation. If we write past the current end of the file, the file will be enlarged to contain the new data.

Sometimes, writing out the data is not enough. We want to be sure the file on the physical disk gets updated immediately (note that even thought the system calls do not buffer writes, the operating system still buffers write operations using its disk cache). In such cases, we may use the fsync() system call. It ensures that any write operations for the given file descriptor that are kept in the system's disk cache, are actually written to disk, when the fsync() system call returns to the caller. Here is how to use it:


#include <unistd.h>    /* declaration of fsync() */
.
.
if (fsync(fd) == -1) {
    perror("fsync");
}

Note that fsync() updates both the file's contents, and its book-keeping data (such as last modification time). If we only need to assure that the file's contents is written to disk, and don't care about the last update time, we can use fdatasync() instead. This is more efficient, as it will issue one fewer disk write operation. In applications that need to synchronize data often, this small saving is important.

Seeking In An Open File

Just like we used the fseek() function to move the read/write pointer of the file stream, we can use the lseek() system call to move the read/write pointer for a file descriptor. Assuming you understood the fseek() examples above, here are a few similar examples using lseek(). We assume that 'fd_read' is an integer variable containing a file descriptor to a previously opened file, in read only mode. 'fd_readwrite' is a similar file descriptor, but for a file opened in read/write mode.


/* this variable is used for storing locations returned by       */
/* lseek().                                                      */
off_t location;

/* move the read/write pointer of the file to position '40'      */
/* in the file. Note that the first position in the file is '0', */
/* not '1'.                                                      */
location = lseek(fd_read, 39L, SEEK_START);

/* move the read/write pointer of the file stream 67 characters  */
/* forward from its given location.                              */
location = lseek(fd_read, 67L, SEEK_SET);
printf("read/write pointer location: %ld\n", location);

/* remember the current read/write pointer's position, move it   */
/* to location '664' in the file, write the string "hello world",*/
/* and move the pointer back to the previous location.           */
location = lseek(fd_readwrite, 0L, SEEK_SET);
if (location == -1) {
    perror("lseek");
    exit(0);
}
if (lseek(fd_readwrite, 663L, SEEK_SET) == -1) {
    perror("lseek(fd_readwrite, 663L, SEEK_SET)");
    exit(0);
}
rc = write(fd_readwrite, "hello world\n", strlen("hello world\n"));
if (lseek(fd_readwrite, location, SEEK_SET) == -1) {
    perror("lseek(fd_readwrite, location, SEEK_SET)");
    exit(0);
}

Note that lseek() might not always work for a file descriptor (e.g. if this file descriptor represents the standard input, surely we cannot have random-access to it). You will encounter other similar cases when you deal with network programming and inter-process communications, in the future.

Acknowledgement: This notes have been taken from the LUPG tutorial and slightly modified by Mirela Damian.