Append a record to the end of a text file
Many systems offer the ability to open a file for writing, such that any data written will be appended to the end of the file. Further, the file operations will always adjust the position pointer to guarantee the end of the file, even in a multitasking environment.
This feature is most useful in the case of log files, where many jobs may be appending to the log file at the same time, or where care must be taken to avoid concurrently overwriting the same record from another job.
Task: Given a two record sample for a mythical "passwd" file:
- Write these records out in the typical system format.
- Ideally these records will have named fields of various types.
- Close the file, then reopen the file for append.
- Append a new record to the file and close the file again.
- Take appropriate care to avoid concurrently overwrites from another job.
- Open the file and demonstrate the new record has indeed written to the end.
account | password | UID | GID | fullname,office,extension,homephone,email | directory | shell |
---|---|---|---|---|---|---|
string | string | int | int | struct(string,string,string,string,string) | string | string |
jsmith | x | 1001 | 1000 | Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org | /home/jsmith | /bin/bash |
jdoe | x | 1002 | 1000 | Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org | /home/jsmith | /bin/bash |
account | password | UID | GID | fullname,office,extension,homephone,email | directory | shell |
---|---|---|---|---|---|---|
string | string | int | int | struct(string,string,string,string,string) | string | string |
xyz | x | 1003 | 1000 | X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org | /home/xyz | /bin/bash |
Resulting file format: should mimic Linux's /etc/passwd file format with particular attention to the "," separator used in the GECOS field. But if the specific language has a particular or unique format of storing records in text file, then this format should be named and demonstrated with an additional example.
Expected output:
Appended record: xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/bash
Finally: Provide a summary of the language's "append record" capabilities in a table. eg.
Data Representation | IO Library |
Append Possible |
Automatic Append |
Multi-tasking Safe | |
---|---|---|---|---|---|
In core | On disk | ||||
C struct | CSV text file | glibc/stdio | ☑ | ☑ | ☑ (Not all, eg NFS) |
Alternatively: If the language's appends can not guarantee its writes will always append, then note this restriction in the table. If possible, provide an actual code example (possibly using file/record locking) to guarantee correct concurrent appends.
C
Data Representation | IO Library |
Append Possible |
Automatic Append |
Multi-tasking Safe | |
---|---|---|---|---|---|
In core | On disk | ||||
C struct | CSV text file | glibc/stdio | ☑ | ☑ | ☑ (Not all, eg NFS) |
Note: Not all File Systems support atomic appends. In particular NFS does not. It is not known if there is a standard OS independent way of detecting if atomic appends are available. However most Unix & Linux File Systems do support atomic appends, especially for log files.
From a C "struct" to CSV File <lang C>#include <stdio.h>
- include <string.h>
/* note that UID & GID are of type "int" */ typedef char *STRING; typedef struct{STRING fullname, office, extension, homephone, email; } gecos_t; typedef struct{STRING account, password; int uid, gid; gecos_t gecos; STRING directory, shell; } passwd_t;
- define GECOS_FMT "%s,%s,%s,%s,%s"
- define PASSWD_FMT "%s:%s:%d:%d:"GECOS_FMT":%s:%s"
passwd_t passwd_list[]={
{"jsmith", "x", 1001, 1000, /* UID and GID are type int */ {"Joe Smith", "Room 1007", "(234)555-8917", "(234)555-0077", "jsmith@rosettacode.org"}, "/home/jsmith", "/bin/bash"}, {"jdoe", "x", 1002, 1000, {"Jane Doe", "Room 1004", "(234)555-8914", "(234)555-0044", "jdoe@rosettacode.org"}, "/home/jdoe", "/bin/bash"}
};
main(){ /****************************
- Create a passwd text file *
- /
FILE *passwd_text=fopen("passwd.txt", "w"); int rec_num; for(rec_num=0; rec_num < sizeof passwd_list/sizeof(passwd_t); rec_num++) fprintf(passwd_text, PASSWD_FMT"\n", passwd_list[rec_num]); fclose(passwd_text);
/********************************
- Load text ready for appending *
- /
passwd_text=fopen("passwd.txt", "a+"); char passwd_buf[BUFSIZ]; /* warning: fixed length */ passwd_t new_rec = {"xyz", "x", 1003, 1000, /* UID and GID are type int */ {"X Yz", "Room 1003", "(234)555-8913", "(234)555-0033", "xyz@rosettacode.org"}, "/home/xyz", "/bin/bash"}; sprintf(passwd_buf, PASSWD_FMT"\n", new_rec);
/* An atomic append without a file lock,
Note: wont work on some file systems, eg NFS */ write(fileno(passwd_text), passwd_buf, strlen(passwd_buf)); close(passwd_text);
/***********************************************
- Finally reopen and check record was appended *
- /
passwd_text=fopen("passwd.txt", "r"); while(!feof(passwd_text)) fscanf(passwd_text, "%[^\n]\n", passwd_buf, "\n"); if(strstr(passwd_buf, "xyz")) printf("Appended record: %s\n", passwd_buf);
}</lang> Output:
Appended record: xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/bash
Go
<lang go>package main
import (
"bytes" "fmt" "io" "io/ioutil" "os"
)
type pw struct {
account, password string uid, gid uint gecos directory, shell string
}
type gecos struct {
fullname, office, extension, homephone, email string
}
func (p *pw) encode(w io.Writer) (int, error) {
return fmt.Fprintf(w, "%s:%s:%d:%d:%s,%s,%s,%s,%s:%s:%s\n", p.account, p.password, p.uid, p.gid, p.fullname, p.office, p.extension, p.homephone, p.email, p.directory, p.shell)
}
// data cut and pasted from task description var p2 = []pw{
{"jsmith", "x", 1001, 1000, gecos{"Joe Smith", "Room 1007", "(234)555-8917", "(234)555-0077", "jsmith@rosettacode.org"}, "/home/jsmith", "/bin/bash"}, {"jdoe", "x", 1002, 1000, gecos{"Jane Doe", "Room 1004", "(234)555-8914", "(234)555-0044", "jdoe@rosettacode.org"}, "/home/jsmith", "/bin/bash"},
}
var pa = pw{"xyz", "x", 1003, 1000, gecos{"X Yz", "Room 1003",
"(234)555-8913", "(234)555-0033", "xyz@rosettacode.org"}, "/home/xyz", "/bin/bash"}
var expected = "xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913," +
"(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/bash"
const pfn = "mythical"
func main() {
writeTwo() appendOneMore() checkResult()
}
func writeTwo() {
// overwrites any existing file f, err := os.Create(pfn) if err != nil { fmt.Println(err) return } defer func() { if cErr := f.Close(); cErr != nil && err == nil { fmt.Println(cErr) } }() for _, p := range p2 { if _, err = p.encode(f); err != nil { fmt.Println(err) return } }
}
func appendOneMore() {
// file must already exist f, err := os.OpenFile(pfn, os.O_RDWR|os.O_APPEND, 0) if err != nil { fmt.Println(err) return } if _, err := pa.encode(f); err != nil { fmt.Println(err) } if cErr := f.Close(); cErr != nil && err == nil { fmt.Println(cErr) }
}
func checkResult() {
// reads entire file then closes it b, err := ioutil.ReadFile(pfn) if err != nil { fmt.Println(err) return } if string(bytes.Split(b, []byte{'\n'})[2]) == expected { fmt.Println("append okay") } else { fmt.Println("it didn't work") }
}</lang>
- Output:
append okay
Data Representation | IO Library |
Append Possible |
Automatic Append |
Multi-tasking Safe | |
---|---|---|---|---|---|
In core | On disk | ||||
Byte slice. See note 1. | Abstracted. See note 2. | Go standard library. See note 3. | Yes | No. See note 4. | Should be, but see note 5. |
Notes:
- Data arguments for basic IO functions are type []byte. All data being written ultimately gets encoded to []byte. Some higher level functions accept strings and do the conversion as a convenience. The fmt.Fprintf function used in this task encodes basic types directly to an io.Writer. Some packages such as json and xml have marshallers to encode more complex data structures. These features are common to all writes and are not specific to append.
- The device data representation is abstracted by the type os.File and its associated functions and methods. These methods include seeking by linear file position so a typical data representation is mappable to a linear sequence of bytes. That is about all that can be concluded from the capabilites of the os.File type. This linear nature is not specific to append.
- Go's os package provides high-level services which work reasonably the same across all supported operating systems. Currently the list is darwin, freebsd, linux, netbsd, openbsd, windows. It does this with a common OS-independent interface. Operations which cannot be offered accross all operating systems are not in this package. Additional low-level and operating system specific functions are provided in the syscall package. This organization is not specific to append.
- The file position after opening a file is 0 unless os.O_APPEND is specified in the os.OpenFile function.
- The Go documentation says nothing about file operations by multiple concurrent processes or threads. As the operations are expected to work the same across operating systems however, they presumably do what you would expect. It seems os.FileOpen with os.O_RDWR for example, gives exclusive file access (on local file systems anyway) whether os.O_APPEND is specified or not.
J
<lang j>require'strings ~system/packages/misc/xenos.ijs' record=: [:|: <@deb;._1@(':',]);._2@do bind '0 :0'
passfields=: <;._1':username:password:gid:uid:gecos:home:shell'
passrec=: LF,~ [: }.@;@ , ':';"0 (passfields i. {. ) { a:,~ {:
R1=: passrec record
username: jsmith password: x gid: 1001 uid: 1000 gecos: Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org home: /home/jsmith shell: /bin/bash
)
R2=: passrec record
username: jdoe password: x gid: 1002 uid: 1000 gecos: Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org home: /home/jdoe shell: /bin/bash
)
R3=: passrec record
username: xyz password: x gid: 1003 uid: 1000 gecos: X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org home: /home/xyz shell: /bin/bash
)
passwd=: <'/tmp/passwd.txt' NB. file needs to be writable on implementation machine
(R1,R2) fwrite passwd R3 fappend passwd
assert 1 e. R3 E. fread passwd </lang>
Data Representation | |
---|---|
In core | On disk |
array | literal |
Note that no file locking is needed if this is implemented under windows (since all file writes are atomic across processes -- only one process can have a file open at one time, by default). Note that file locking would be needed under Linux (or unix), but it's not clear which file locking mechanism should be used.
Java
<lang java>import java.io.BufferedReader; import java.io.FileNotFoundException; import java.io.FileReader; import java.io.FileWriter; import java.io.IOException; import java.io.PrintWriter;
public class Append{ public static void main(String[] args){ try{ System.out.println("Pre-loading file..."); //the "true" here is for autoflush PrintWriter out = new PrintWriter(new FileWriter("passwd"), true); out.println("user1 password1"); out.println("user2 password2"); out.close(); showFile("passwd");
System.out.println("\nAppending new line to file..."); //the first "true" is for append, the second is for autoflush out = new PrintWriter(new FileWriter("passwd", true), true); out.println("user3 password3"); out.close(); showFile("passwd"); }catch(IOException e){ System.err.println("Error in writing file: " + e.getMessage()); } }
private static void showFile(String filename){ try{ BufferedReader in = new BufferedReader(new FileReader(filename)); while(in.ready()){ System.out.println(in.readLine()); } }catch(FileNotFoundException e){ System.err.println("File " + filename + " not found."); }catch(IOException e){ System.err.println("Error in reading file: " + e.getMessage()); } } }</lang> Output:
Pre-loading file... user1 password1 user2 password2 Appending new line to file... user1 password1 user2 password2 user3 password3
MATLAB / Octave
<lang Matlab> DS{1}.account='jsmith';
DS{1}.password='x'; DS{1}.UID=1001; DS{1}.GID=1000; DS{1}.fullname='Joe Smith'; DS{1}.office='Room 1007'; DS{1}.extension='(234)555-8917'; DS{1}.homephone='(234)555-0077'; DS{1}.email='jsmith@rosettacode.org'; DS{1}.directory='/home/jsmith'; DS{1}.shell='/bin/bash';
DS{2}.account='jdoe'; DS{2}.password='x'; DS{2}.UID=1002; DS{2}.GID=1000; DS{2}.fullname='Jane Doe'; DS{2}.office='Room 1004'; DS{2}.extension='(234)555-8914'; DS{2}.homephone='(234)555-0044'; DS{2}.email='jdoe@rosettacode.org'; DS{2}.directory='/home/jdoe'; DS{2}.shell='/bin/bash';
function WriteRecord(fid, rec) fprintf(fid,"%s:%s:%i:%i:%s,%s,%s,%s,%s:%s%s\n", rec.account, rec.password, rec.UID, rec.GID, rec.fullname, rec.office, rec.extension, rec.homephone, rec.email, rec.directory, rec.shell); return; end
%% write fid = fopen('passwd.txt','w'); WriteRecord(fid,DS{1}); WriteRecord(fid,DS{2}); fclose(fid);
new.account='xyz'; new.password='x'; new.UID=1003; new.GID=1000; new.fullname='X Yz'; new.office='Room 1003'; new.extension='(234)555-8913'; new.homephone='(234)555-0033'; new.email='xyz@rosettacode.org'; new.directory='/home/xyz'; new.shell='/bin/bash';
%% append fid = fopen('passwd.txt','a+'); WriteRecord(fid,new); fclose(fid);
% read password file fid = fopen('passwd.txt','r'); while ~feof(fid)
printf('%s\n',fgetl(fid));
end; fclose(fid); </lang>
Output:
jsmith:x:1001:1000:Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org:/home/jsmith/bin/bash jdoe:x:1002:1000:Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org:/home/jdoe/bin/bash xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz/bin/bash
Python
Data Representation | IO Library |
Append Possible |
Automatic Append |
Multi-tasking Safe | |
---|---|---|---|---|---|
In core | On disk | ||||
dict | CSV text file | builtin | ☑ | ☑ | ☒ |
instance | CSV text file | builtin | To do. |
From a "dict" to a CSV File <lang python>#############################
- Create a passwd text file
- note that UID & gid are of type "text"
passwd_list=[
dict(account='jsmith', password='x', UID=1001, GID=1000, # UID and GID are type int GECOS=dict(fullname='Joe Smith', office='Room 1007', extension='(234)555-8917', homephone='(234)555-0077', email='jsmith@rosettacode.org'), directory='/home/jsmith', shell='/bin/bash'), dict(account='jdoe', password='x', UID=1002, GID=1000, GECOS=dict(fullname='Jane Doe', office='Room 1004', extension='(234)555-8914', homephone='(234)555-0044', email='jdoe@rosettacode.org'), directory='/home/jdoe', shell='/bin/bash')
]
passwd_fields="account password UID GID GECOS directory shell".split() GECOS_fields="fullname office extension homephone email".split()
def passwd_text_repr(passwd_rec):
- convert individual fields to string type
passwd_rec["GECOS"]=",".join([ passwd_rec["GECOS"][field] for field in GECOS_fields]) for field in passwd_rec: # convert "int" fields if not isinstance(passwd_rec[field], str): passwd_rec[field]=`passwd_rec[field]` return ":".join([ passwd_rec[field] for field in passwd_fields ])
passwd_text=open("passwd.txt","w") for passwd_rec in passwd_list:
print >> passwd_text,passwd_text_repr(passwd_rec)
passwd_text.close()
- Load text ready for appending
passwd_text=open("passwd.txt","a+") new_rec=dict(account='xyz', password='x', UID=1003, GID=1000,
GECOS=dict(fullname='X Yz', office='Room 1003', extension='(234)555-8913', homephone='(234)555-0033', email='xyz@rosettacode.org'), directory='/home/xyz', shell='/bin/bash')
print >> passwd_text, passwd_text_repr(new_rec) passwd_text.close()
- Finally reopen and check record was appended
passwd_list=list(open("passwd.txt","r")) if "xyz" in passwd_list[-1]:
print "Appended record:",passwd_list[-1][:-1]</lang>
Output:
Appended record: xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/bash
Ruby
Data Representation | IO Library |
Append Possible |
Automatic Append |
Multi-tasking Safe | |
---|---|---|---|---|---|
In core | On disk | ||||
objects (subclass of Struct builtin) | text file | builtin | ☑ | ☑ | ☒ |
<lang ruby>Gecos = Struct.new :fullname, :office, :extension, :homephone, :email class Gecos
def to_s "%s,%s,%s,%s,%s" % [fullname, office, extension, homephone, email] end
end
Passwd = Struct.new(:account, :password, :uid, :gid, :gecos, :directory, :shell) class Passwd
def to_s "%s:%s:%s:%s:%s:%s:%s" % [account, password, uid, gid, gecos, directory, shell] end
end
jsmith = Passwd.new('jsmith','x',1001, 1000, Gecos.new('Joe Smith', 'Room 1007', '(234)555-8917', '(234)555-0077', 'jsmith@rosettacode.org'), '/home/jsmith', '/bin/bash') jdoe = Passwd.new('jdoe','x',1002, 1000, Gecos.new('Jane Doe', 'Room 1004', '(234)555-8914', '(234)555-0044', 'jdoe@rosettacode.org'), '/home/jdoe', '/bin/bash') xyz = Passwd.new('xyz','x',1003, 1000, Gecos.new('X Yz', 'Room 1003', '(234)555-8913', '(234)555-0033', 'xyz@rosettacode.org'), '/home/xyz', '/bin/bash')
filename = 'append.records.test'
- create the passwd file with two records
File.open(filename, 'w') do |io|
io.puts jsmith io.puts jdoe
end
puts "before appending:" system 'cat', '-n', filename
- append the third record
File.open(filename, 'a') do |io|
io.puts xyz
end
puts "after appending:" system 'cat', '-n', filename</lang>
output
before appending: 1 jsmith:x:1001:1000:Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org:/home/jsmith:/bin/bash 2 jdoe:x:1002:1000:Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org:/home/jdoe:/bin/bash after appending: 1 jsmith:x:1001:1000:Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org:/home/jsmith:/bin/bash 2 jdoe:x:1002:1000:Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org:/home/jdoe:/bin/bash 3 xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/bash
Tcl
Data Representation | IO Library |
Append Possible |
Automatic Append |
Multi-tasking Safe | |
---|---|---|---|---|---|
In core | On disk | ||||
nested lists | Colon/Comma-separated text file | builtin | ☑ | ☑ | ☑ |
Note that appending is only safe on POSIX OSes where the data is written in “small enough” amounts to a local disk. This is a limitation of the OS APIs. <lang tcl># Model the data as nested lists, as that is a natural fit for Tcl set basicRecords {
{
jsmith x 1001 1000 { {Joe Smith} {Room 1007} (234)555-8917 (234)555-0077 jsmith@rosettacode.org } /home/jsmith /bin/bash
} {
jdoe x 1002 1000 { {Jane Doe} {Room 1004} (234)555-8914 (234)555-0044 jdoe@rosettacode.org } /home/jsmith /bin/bash
}
} set addedRecords {
{
xyz x 1003 1000 { {X Yz} {Room 1003} (234)555-8913 (234)555-0033 xyz@rosettacode.org } /home/xyz /bin/bash
}
}
proc printRecords {records fd} {
fconfigure $fd -buffering none foreach record $records {
lset record 4 [join [lindex $record 4] ","] puts -nonewline $fd [join $record ":"]\n
}
} proc readRecords fd {
set result {} foreach line [split [read $fd] "\n"] {
if {$line eq ""} continue set record [split $line ":"] # Special handling for GECOS lset record 4 [split [lindex $record 4] ","] lappend result $record
} return $result
}
- Write basic set
set f [open ./passwd w] printRecords $basicRecords $f close $f
- Append the extra ones
- Use {WRONLY APPEND} on Tcl 8.4
set f [open ./passwd a] printRecords $addedRecords $f close $f
set f [open ./passwd] set recs [readRecords $f] close $f puts "last record is for [lindex $recs end 0], named [lindex $recs end 4 0]"</lang> Which produces this output:
last record is for xyz, named X Yz
And a file with this contents:
jsmith:x:1001:1000:Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org:/home/jsmith:/bin/bash jdoe:x:1002:1000:Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org:/home/jsmith:/bin/bash xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/bash