Is there a way to create special files under Linux that would keep only, say the 100 last written lines? I have a process filling a log file, and I'd like to regularly parse its 100 last lines.
I know I could use some kind of logrotate, but is there a way to create a special file that would fill up till it reaches 100 lines, then, adding a line removes the oldest one, so that the file only keeps 100 lines? (a kind of line-based FIFO)
Thanks a lot
43 Answers
Logs are text(like) files and being that kind, appending new disk blocks to them when it's demanded by new lines is a quick action well supported by any file system.
However, constantly dropping out the first line when a new is coming would mean reorganizing at least some if not all the blocks of the file CONSTANTLY. That would mean big overhead dedicated to logging while one of the most important characteristics we want from logging is that it be lightweight .
File systems are not prepared for this (at least I haven't heard about this type), that's why logrotate/tail/database-backed logging are used where the last records are of importance.
1Presuming that you have a script that processes only the last 100 lines each time it is run, these lines are best captured with the tail command, as it does pretty much what you want to do. The key here is the -n switch which dictates how many lines it should capture, starting from the end.
You can incoprorate tail -n 100 somefile.log in your script directly, or you can periodically run tail -n 100 somefile.log > onlylast100lines.log to create a file with only the last 100 lines. The latter approach will rewrite the target file every time, so no need to delete it between each run.
Elaborating on Jarmunds method, you could make a bash script like this:
while true; do tail -n 100 somefile.log > onlylast100lines.log sleep 5
donethis would parse the last 100 lines of somefile.log into onlylast100lines.log every 5 seconds. The file would be overwritten each time, so it would be always only the last 100 lines. This could be added to your login script or to whatever runlevel you want this to be executed.
1
