tee: improve throughput by raw syscall

[oech3]

Remove flush overhead and use raw syscall for POSIX tee requirement.

$ truncate -s 1PB huge
$ gnu9.11/tee<huge|pv>/dev/null
7.13GiB 0:00:02 [3.61GiB/s]
$ target/release/tee<huge|pv>/dev/null
2.56GiB 0:00:01 [2.56GiB/s]
$ target/release/tee-raw<huge|pv>/dev/null
3.35GiB 0:00:01 [3.35GiB/s]

$ cat huge|gnu9.11/tee|pv>/dev/null
2.15GiB 0:00:01 [2.15GiB/s]
$ cat huge|target/release/tee|pv>/dev/null
2.31GiB 0:00:01 [2.31GiB/s]
$ cat huge|target/release/tee-raw|pv>/dev/null
3.00GiB 0:00:01 [3.00GiB/s]

I was really surprised that std::io has such overhead.

[sylvestre]

is it possible to have a codspeed benchmark to evaluate the win?

[sylvestre]

i asked on discord

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/date/date-locale-hour (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/tail-n0f (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/cut/bounded-memory (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/rm/many-dir-entries-vs-OOM is now being skipped but was previously passing.
Congrats! The gnu test tests/basenc/bounded-memory is now passing!

[xtqqczze]

The Write implementation is somewhat unintuitive; it could be removed in favor of a simpler approach with a single method:

fn write(&mut self, buf: &[u8]) -> Result<()>

Additionally, avoiding Write may be preferable if we plan to support a MaybeUninit buffer in the future.

[oech3]

This impl is because rustix::io::write does not support writing to stdout on Windows. If we can override write by similar way, it is preffered.

[xtqqczze]

I still think its's confusing to implement Write on Unix, but not Windows...

[xtqqczze]

I was thinking something like this: b0edaae

[oech3]

No. Because write syscall does not guarantee writing everything. Needs loop and take sum of returned values.
(also rustix::io::write_all is not provided...)

[xtqqczze]

It would solve many problems if we could use std::io::stdio::StdoutRaw. Perhaps this functionality could be implemented as a crate.

[oech3]

Private? Is it need to copy and paste unsafe code to uucore?

[xtqqczze]

The issue we’re working around is that std::io::Stdout uses buffering, which doesn’t match our requirements. Ideally, we should provide our own Stdout implementation that performs unbuffered I/O instead. But that would be a much larger piece of work.

[oech3]

https://docs.rs/rustix/latest/rustix/stdio/fn.stdout.html but unix only...

[xtqqczze]

We already have a solution in uucore::io::OwnedFileDescriptorOrHandle, see #10235.

[oech3]

Windows example (with unsafe) is rust-lang/rust#58326 (comment) and rust-lang/rust#58326 (comment)

[xtqqczze]

I would have expected uucore::io::write_all_raw to take a RawFd. Could you please rename it to write_all?

[oech3]

raw means raw-syscall.

[oech3]

Considering most API is buffered, I think write_all_unbuffered is the best effort.

[xtqqczze]

raw means raw-syscall.

It would be better to move it to a new sys module.

[oech3]

I think new module for 1 fn is overkill and causing confusion as it is clearly io related operation.

[xtqqczze]

How naming it write_all_fd, and exposing it via an extension trait?

pub trait WriteFdExt: AsFd {
    fn write_all_fd(&self, buf: &[u8]) -> io::Result<()> { ... }
}

impl<T: AsFd + ?Sized> WriteFdExt for T {}

[oech3]

Why? I'm OK to have it as a method. But I prefer write_all_unbuffered to clarify usecase.

[oech3]

I'll rename it to something after several PRs depending on it were merged (to avoid CI breakage).

[xtqqczze]

I'm trying a different approach with ManuallyDrop<File> in #12207

[oech3]

I minimized diff by sharing code with cat. #12207 can be closed.
I think we should wait std's change for Windows if it requires additional unsafe.

@sylvestre Would you review this? Benchmark is already enabled.
This PR blocks #11943 to make tee similar speed with RAM's bandwidth.

[oech3]

@sylvestre Is this OK?

[oech3]

thankyou