Turn off buffering in pipe

Yet an additional means to activate line - buffering result setting for the long_running_command is to make use of the script command that runs your long_running_command in a quasi terminal (pty).

script -q /dev/null long_running_command | print_progress      # (FreeBSD, Mac OS X)
script -q -c "long_running_command" /dev/null | print_progress # (Linux)

For grep, sed and also awk you can compel result to be line buffered. You can use:

grep --line-buffered

Force result to be line buffered. By default, result is line buffered when typical result is a terminal and also block buffered various other - sensible.

sed -u

Make result line buffered.

See this web page for additional information: http://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html

According to this the pipe buffer dimension appears to be embeded in the kernel and also would certainly need you to recompile your kernel to modify.

I do not assume the trouble is with the pipe. It seems like your future procedure is not purging its very own buffer regularly sufficient. Transforming the pipe is buffer dimension would certainly be a hack to get rounded it, yet I do not assume its feasible without restoring the kernel - something you would not intend to do as a hack, as it possibly aversley influence a great deal of various other procedures.

It made use of to be the instance, and also possibly still holds true, that when typical result is contacted an incurable, it is line buffered by default - when a newline is created, the line is contacted the incurable. When typical result is sent out to a pipe, it is totally buffered - so the information is just sent out to the next procedure in the pipe when the typical I/O buffer is loaded.

That is the resource of the problem. I'm not exactly sure whether there is much you can do to repair it without changing the program creating right into the pipe. You can make use of the setvbuf() function with the _IOLBF flag to unconditionally place stdout right into line buffered setting. Yet I do not see a very easy means to apply that on a program. Or the program can do fflush() at ideal factors (after each line of result), yet the very same comment uses.

I intend that if you changed the pipe with a pseudo - incurable, after that the typical I/O collection would certainly assume the result was an incurable (due to the fact that it is a sort of incurable) and also would certainly line buffer instantly. That is an intricate means of managing points, however.

If it is a trouble with the libc changing its buffering/ purging when result does not most likely to an incurable, you need to attempt socat. You can create a bidirectional stream in between virtually any kind of sort of I/O device. Among those is a forked program talking with a pseudo tty.

 socat EXEC:long_running_command,pty,ctty STDIO 

What it does is

• create a pseudo tty
• fork long_running_command with the servant side of the pty as stdin/stdout
• develop a bidirectional stream in between the master side of the pty and also the 2nd address (below it is STDIO)

If this offers you the very same result as long_running_command, after that you can proceed with a pipe.

Edit: Wow Did not see the unbuffer solution! Well, socat is a wonderful device anyhow, so I could simply leave this solution

Another means to skin this pet cat is to make use of the stdbuf program, which becomes part of the GNU Coreutils (FreeBSD additionally has its very own one).

stdbuf -i0 -o0 -e0 command

This switches off buffering entirely for input, result and also mistake. For some applications, line buffering might be preferable for efficiency factors:

stdbuf -oL -eL command

_{Note that it just benefits stdio buffering (printf(), fputs() ) for dynamically connected applications, and also just if that application does not or else readjust the buffering of its typical streams on its own, though that need to cover most applications.}

You can make use of the unbuffer command (which comes as component of the expect plan), as an example

unbuffer long_running_command | print_progress

unbuffer attaches to long_running_command using a pseudoterminal (pty), that makes the system treat it as an interactive procedure, consequently not making use of the 4 - kiB buffering in the pipe that is the most likely source of the hold-up.

For longer pipes, you might need to unbuffer each command (other than the last one), as an example

unbuffer x | unbuffer -p y | z