FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Kernel

 
 
LinkBack Thread Tools
 
Old 06-10-2011, 08:21 AM
Jonathan Nieder
 
Default Bug#629994: sendfile returns early without user-visible reason

tags 629994 + upstream wontfix
quit

Hi Marc,

Marc Lehmann wrote:

> In 2.6.39 (and maybe some earlier versions= of Linux, sendfile supports
> file->file copies.
[...]
> Linux always seems to stop copying at 0x7FFFF000 bytes, without apparent
> reason (such as disk full or another error). This happens with a 64 bit
> kernel btw., so this is not a 32 bit issue either.
>
> This causes many programs to report a "short write", as this is an error
> condition with similar syscalls on files (such as write(2)).

Yes, I can reproduce it[*].

I believe this is from v2.6.16-rc1~169^2~16^2~37 (Relax the
rw_verify_area() error checking, 2006-01-04). In other words, it seems
like it was always this way. (Side note: recently the principle behind
that patch was reaffirmed --- see v2.6.37-rc1~40, readv/writev: do the
same MAX_RW_COUNT truncation that read/write does, 2010-10-29.)

Indeed, read(2) does the same thing (truncates to 7ffff000) and has done
so for five years, though it's a little harder to notice (I had to use
mmap to create a large file-backed buffer to read into.)

Background: even with read(2) and write(2), partial progress does not
necessarily represent an error. For example, on "slow" devices like a
terminal, pipe, or socket, a partial success can indicate interruption
by a signal, and on a named or unnamed pipe it can indicate that fewer
than the requested number of bytes were immediately available. So I am
somewhat curious about these many programs --- why are they expecting
this from sendfile?

The manpage is outdated and does not even indicate that sendfile can be
used to copy a file. Has the size allowed for a single sendfile(2) call
changed over time? Is this a regression or a request for a new feature?

> I think sendfile should either not attempt to copy files, or copy the
> requested number of bytes unless an error occurs

If an application wants to print a useful error message, it has to try
again until sendfile returns -1 so errno can be set.

Anyway, I agree that it would be better for sendfile to return partial
results less often, to make one-off programs easier to write and to
decrease the number of syscalls made, but that doesn't seem worth
exposing problems in low-level driver code that assumes it never has to
write more than fits in an "int" at a given moment. So I'm marking this
wontfix for now.

All that said.

An obvious possible improvement would be to update the manpages to
include information about this. Would you be interested in that, and if
so, can you suggest a wording?

Thanks and hope that helps,
Jonathan
[*]
#define _FILE_OFFSET_BITS 64
#include <sys/sendfile.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <inttypes.h>
#include <limits.h>

static void die(const char *msg)
{
fprintf(stderr, "%s
", msg);
exit(1);
}

int main(int argc, const char *argv[])
{
struct stat st;
int in, out;

if (argc != 3)
die("wrong number of arguments");
in = open(argv[1], O_RDONLY);
out = open(argv[2], O_WRONLY | O_CREAT);
if (in < 0 || out < 0)
die("cannot open file");
if (fstat(in, &st))
die("cannot get input file status");
if (st.st_size > SSIZE_MAX)
die("input file is too big");
printf("sendfile returns %"PRIdMAX"
",
(intmax_t) sendfile(out, in, NULL, st.st_size));
return 0;
}



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610082138.GA29517@elie">http://lists.debian.org/20110610082138.GA29517@elie
 
Old 06-10-2011, 10:15 AM
Marc Lehmann
 
Default Bug#629994: sendfile returns early without user-visible reason

On Fri, Jun 10, 2011 at 03:21:38AM -0500, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Indeed, read(2) does the same thing (truncates to 7ffff000) and has done

What the fuck, it's buggy, indeed:

read(0, ""..., 3298534883328) = 2147479552

> so for five years, though it's a little harder to notice (I had to use
> mmap to create a large file-backed buffer to read into.)

Well, for read, the situation is a bit different, because thats a clear
posix violation. While this is obviously not relevant for sendfile, it of
course makes sense to use posix (or simply traditional unix) I/O semantics
for sendfile as well *iff* read implements posix behaviour.

> Background: even with read(2) and write(2), partial progress does not
> necessarily represent an error.

Thats true _in general_, but in posix/unix/sus it has clearly defined
and user-visible semantics for that, which require that the success case
transfers as many bytes as can be transferred, and not stop a random
amount earlier unless there is an error condition (signal => EINTR if not
restarted, and easily controllable by applications - for example, not
doing anything with signals makes it work):

The value returned may be less than nbyte if the number of bytes left
in the file is less than nbyte, if the read() request was interrupted
by a signal, or if the file is a pipe or FIFO or special file and has
fewer than nbyte bytes immediately available for reading.

In this case, the size of the file was also the sendfile transfer size. If
read() was used, and read wouldn't be non-posix, then a partial read
necessarily indicates an error of some kind.

> For example, on "slow" devices like a terminal, pipe, or socket, a
> partial success can indicate interruption by a signal, and on a named
> or unnamed pipe it can indicate that fewer than the requested number of
> bytes were immediately available.

A file is not a terminal, pipe or socket - I specifically reported a
file->file problem, not a socket->file or file->socket problem (the
former is probably not supported and the latter has a whole lot different
error modes/behvaiour). For files, unless you do signal stuff, a partial
(posix-) read indicates error or end of file.

Applications are well aware of the differences between sockets and files
for example (set nonblocking mode for example to see very different
behaviour).

The biggets difference to your example and my exampel however is that the
posix requireemnts are so strong, and the unix behaviour has been working for
so long, that applications that have short writes or reads when they know
there is more data can rightfully abort the process.

With sockets, posix semantics are different, so the normal behaviour of
applications is to retry.

This works fine as long as the OS follows posix.

> So I am somewhat curious about these many programs --- why are they
> expecting this from sendfile?

Because thats how every unix works now and has in the past, and thats what
the unix standard requires.

I think it is reasonable for programs to expect sendfile to behave like a
synthesized read+write, as opposed to "weird" semantics, and I think it is
reasonable for read() to follow the posix semantics nowadays, it shouldn't
be that hard to implement it, and the portability gain from having posix
behaviour is immense.

That linux apparently fails to implement this for read too, makes it
consistent (which is kind of good), but creates a portability problem for
unix programs.

> The manpage is outdated and does not even indicate that sendfile can be
> used to copy a file.

My manpage clearly alows it:

sendfile() copies data between one file descriptor and another.

There are no other hard requirements listed for sendfile. It mentions that
in 2.6.9, there are extra requirements, but thats obviously not relevant
to (and untrue for) 2.6.39.

That's no different to read or write, both of which also work on file
descriptors and put no other requirements on them.

Since files are accessible via file descriptors, the sendfile manpage
clearly says it can be used to copy a file (or more correctly, to transfer
data from one file to another).

However, the manpage says:

Applications may wish to fall back to read(2)/write(2) in the case
where sendfile() fails with EINVAL or ENOSYS.

And this is in fact what many applications do, try it, and then fall back.

It's also common sense, and the rationale behind the design (cf. Linuses mails
on that topic) - sendfile should implement what the kernel can do more
efficiently, and otherwise signal the application that it should do it itself
(EINVAL). The expected applciation behaviour is just that: flal back to
read/write, and this worked in the past.

The problem is precisely that sendfile changed semantics.

> Has the size allowed for a single sendfile(2) call changed over time?

Implementations following the manpage worked in the past, yes, because
read or write emulation usually uses smaller than 2gb buffers (and if
read(2) would be fixed, it would even work with larger buffer sizes).

The size allowed for a single sendfile was about 0 in earlier versions,
because they returned EINVAL.

> Is this a regression or a request for a new feature?

It seems there are two regressions: read no longer being posix compliant
and sendfile no longer telling applications to use a (working) read/write
loop but instead attempting the copy itself.

> If an application wants to print a useful error message, it has to try
> again until sendfile returns -1 so errno can be set.

Thats clearly just an opinion. The authors of gnu tar and many existing
applications apparently disagree, as do I.

It's widespread behaviour to expect posix semantics nowadays, and quite
reaosnable to expect similar behaviour by sendfile.

In fact, great peril has been brought over the world by introducing so
horribly misdesigned interfaces such as epoll() (and, to a lesser extent,
similar mechanisms in other kernels), that creating consistency in the
form of using posix semangics for any file I/O is clearly a good thing.

But that's just my opinion

> Anyway, I agree that it would be better for sendfile to return partial
> results less often,

I think sendfile should follow the same semantics as unix read(), and
further, linux should follow both defacto historical unix behaviour as
well as posix/sus behaviour and not return partial results in cases not
allowed by posix.

> to make one-off programs easier to write and to decrease the number of
> syscalls made, but that doesn't seem worth

The whole *point* of sendfile is to decrease the number of syscalls, for
high-performance programs. If overhead isn't an issue, then read+write are
much more portable, and typically easier to use.

As such, if sendfile requires extra unnecessary syscalls, this is clearly
a design violation.

> write more than fits in an "int" at a given moment. So I'm marking this
> wontfix for now.

Should I open a separate bug for read(2) then, or will posix compliance
also be a wontfix (a valid position)?

> An obvious possible improvement would be to update the manpages to
> include information about this. Would you be interested in that, and if
> so, can you suggest a wording?

I guess something like that would be fine:

sendfile is not the same as a read+write combination - it may transfer
and return fewer bytes than requested for no user-visible reason.

that would require read(2) to be fixed. A warning that read doesn't
implement posix semantics for file I/O and also errornously might return
partial results might be very useful, too.

--
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ / // / / / schmorp@schmorp.de
-=====/_/_//_/\_,_/ /_/\_



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610101544.GA12349@schmorp.de">http://lists.debian.org/20110610101544.GA12349@schmorp.de
 
Old 06-10-2011, 11:30 AM
Jonathan Nieder
 
Default Bug#629994: sendfile returns early without user-visible reason

tags 629994 + lfs
quit

Hi,

Marc Lehmann wrote:
[out of order for convenience]

> Well, for read, the situation is a bit different, because thats a clear
> posix violation.
[...]
> The value returned may be less than nbyte if the number of bytes left
> in the file is less than nbyte, if the read() request was interrupted
> by a signal, or if the file is a pipe or FIFO or special file and has
> fewer than nbyte bytes immediately available for reading.

That would have been my guess, too. I just stared at the POSIX "write"
manpage for a few minutes, and although it's pretty strongly implied
by the presence of a long list of exceptions:

- if a write() runs out of room (for example, by filling the medium
or hitting the per-process file size limit)
- if the write() is interrupted by a signal
- pipes and FIFOs
- writes with the O_NONBLOCK flag set
- when the number of bytes to be written exceeds SSIZE_MAX
(implementation-defined behavior, truncation presumably possible)
- sockets
- STREAMs

(just like for "read") I didn't find any text that comes out and says
that barring such an exception partial writes are not allowed. The
missing text might be as small as a missing "shall" but the
requirement just didn't seem to be there anywhere obvious.

I would be happy to see the standard clarified. If you're interested,
the people at http://opengroup.org/austin and
http://austingroupbugs.net/ are generally a helpful bunch.

> in posix/unix/sus it has clearly defined
> and user-visible semantics for that, which require that the success case
> transfers as many bytes as can be transferred, and not stop a random
> amount earlier unless there is an error condition (signal => EINTR if not
> restarted, and easily controllable by applications - for example, not
> doing anything with signals makes it work):

Yes, that's mostly true. The idiom "if (read(..., count) == count)"
is not uncommon and is safe when used carefully (for example, when
used with files on disk rather than terminals, tapes, or pipes).

> On Fri, Jun 10, 2011 at 03:21:38AM -0500, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> If an application wants to print a useful error message, it has to try
>> again until sendfile returns -1 so errno can be set.
>
> Thats clearly just an opinion.

Yes, I was clearly overstepping to suggest that an I/O error message
has to include strerror(errno) to be useful. There are plenty of
cases where that isn't true. But isn't it the common case?

> The authors of gnu tar and many existing
> applications apparently disagree, as do I.

GNU tar mostly uses the safe_read function from gnulib, except when
reading the first line of CACHEDIR.TAG files, if I read it correctly.

[...]
> Should I open a separate bug for read(2) then, or will posix compliance
> also be a wontfix (a valid position)?

If you'd like, you can retitle this bug.

Also keep in mind that if you can convince upstream (presumably by
writing a patch, e.g., one to let block devices and in turn
filesystems declare that they are confident about safely handling
large reads/writes) then that "wontfix" tag won't matter in the least.
It's just documentation to summarize the current state of things.

> sendfile is not the same as a read+write combination - it may transfer
> and return fewer bytes than requested for no user-visible reason.
>
> that would require read(2) to be fixed.

If read(2) and write(2) start handling larger chunks, there's
obviously no reason for sendfile(2) not to, too. The whole point of
the exercise is to avoid buffer overflows and other logic errors in
sloppily written, obscure drivers.

Thanks again.

Good night,
Jonathan



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610113048.GB32119@elie">http://lists.debian.org/20110610113048.GB32119@elie
 
Old 06-10-2011, 11:30 AM
Jonathan Nieder
 
Default Bug#629994: sendfile returns early without user-visible reason

tags 629994 + lfs
quit

Hi,

Marc Lehmann wrote:
[out of order for convenience]

> Well, for read, the situation is a bit different, because thats a clear
> posix violation.
[...]
> The value returned may be less than nbyte if the number of bytes left
> in the file is less than nbyte, if the read() request was interrupted
> by a signal, or if the file is a pipe or FIFO or special file and has
> fewer than nbyte bytes immediately available for reading.

That would have been my guess, too. I just stared at the POSIX "write"
manpage for a few minutes, and although it's pretty strongly implied
by the presence of a long list of exceptions:

- if a write() runs out of room (for example, by filling the medium
or hitting the per-process file size limit)
- if the write() is interrupted by a signal
- pipes and FIFOs
- writes with the O_NONBLOCK flag set
- when the number of bytes to be written exceeds SSIZE_MAX
(implementation-defined behavior, truncation presumably possible)
- sockets
- STREAMs

(just like for "read") I didn't find any text that comes out and says
that barring such an exception partial writes are not allowed. The
missing text might be as small as a missing "shall" but the
requirement just didn't seem to be there anywhere obvious.

I would be happy to see the standard clarified. If you're interested,
the people at http://opengroup.org/austin and
http://austingroupbugs.net/ are generally a helpful bunch.

> in posix/unix/sus it has clearly defined
> and user-visible semantics for that, which require that the success case
> transfers as many bytes as can be transferred, and not stop a random
> amount earlier unless there is an error condition (signal => EINTR if not
> restarted, and easily controllable by applications - for example, not
> doing anything with signals makes it work):

Yes, that's mostly true. The idiom "if (read(..., count) == count)"
is not uncommon and is safe when used carefully (for example, when
used with files on disk rather than terminals, tapes, or pipes).

> On Fri, Jun 10, 2011 at 03:21:38AM -0500, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> If an application wants to print a useful error message, it has to try
>> again until sendfile returns -1 so errno can be set.
>
> Thats clearly just an opinion.

Yes, I was clearly overstepping to suggest that an I/O error message
has to include strerror(errno) to be useful. There are plenty of
cases where that isn't true. But isn't it the common case?

> The authors of gnu tar and many existing
> applications apparently disagree, as do I.

GNU tar mostly uses the safe_read function from gnulib, except when
reading the first line of CACHEDIR.TAG files, if I read it correctly.

[...]
> Should I open a separate bug for read(2) then, or will posix compliance
> also be a wontfix (a valid position)?

If you'd like, you can retitle this bug.

Also keep in mind that if you can convince upstream (presumably by
writing a patch, e.g., one to let block devices and in turn
filesystems declare that they are confident about safely handling
large reads/writes) then that "wontfix" tag won't matter in the least.
It's just documentation to summarize the current state of things.

> sendfile is not the same as a read+write combination - it may transfer
> and return fewer bytes than requested for no user-visible reason.
>
> that would require read(2) to be fixed.

If read(2) and write(2) start handling larger chunks, there's
obviously no reason for sendfile(2) not to, too. The whole point of
the exercise is to avoid buffer overflows and other logic errors in
sloppily written, obscure drivers.

Thanks again.

Good night,
Jonathan



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610113048.GB32119@elie">http://lists.debian.org/20110610113048.GB32119@elie
 
Old 06-10-2011, 12:29 PM
Bastian Blank
 
Default Bug#629994: sendfile returns early without user-visible reason

On Fri, Jun 10, 2011 at 12:15:44PM +0200, Marc Lehmann wrote:
> On Fri, Jun 10, 2011 at 03:21:38AM -0500, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > Indeed, read(2) does the same thing (truncates to 7ffff000) and has done
> What the fuck, it's buggy, indeed:
> read(0, ""..., 3298534883328) = 2147479552

What is the bug?

> > Background: even with read(2) and write(2), partial progress does not
> > necessarily represent an error.
> Thats true _in general_, but in posix/unix/sus it has clearly defined
> and user-visible semantics for that,

| Upon successful completion, write() shall return the number of bytes
| actually written to the file associated with fildes. This number shall
| never be greater than nbyte. Otherwise, -1 shall be returned and errno
| set to indicate the error.

> which require that the success case
> transfers as many bytes as can be transferred, and not stop a random
> amount earlier

Please quote the standard on this. Please note that A => B does not
imply !A => !B (A == {not-regular, signaled}, B == short-write).

> This works fine as long as the OS follows posix.

Linux is no fully compliant POSIX system.

Bastian

--
Pain is a thing of the mind. The mind can be controlled.
-- Spock, "Operation -- Annihilate!" stardate 3287.2



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610122946.GA21346@wavehammer.waldi.eu.org">ht tp://lists.debian.org/20110610122946.GA21346@wavehammer.waldi.eu.org
 
Old 06-10-2011, 12:59 PM
Marc Lehmann
 
Default Bug#629994: sendfile returns early without user-visible reason

On Fri, Jun 10, 2011 at 06:30:48AM -0500, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > Well, for read, the situation is a bit different, because thats a clear
> > posix violation.
> [...]
> > The value returned may be less than nbyte if the number of bytes left
> > in the file is less than nbyte, if the read() request was interrupted
> > by a signal, or if the file is a pipe or FIFO or special file and has
> > fewer than nbyte bytes immediately available for reading.
>
> That would have been my guess, too. I just stared at the POSIX "write"

Well, the relevant posix manpage for read is read, not the one for write.
read is clear.

> (just like for "read")

No, read cannot run out of room - the wording is altogether quite different,
and I quoted it above.

It might be less strongly worded, but it's still clear - the standard defines
how a function must behave and exceptions to it.

> I didn't find any text that comes out and says
> that barring such an exception partial writes are not allowed.

The standard specifies the behaviour of write - it doesn't allow
implementations to specify extra behaviour unless it explicitly says so
(for example, with a "may" as is done for read).

> missing text might be as small as a missing "shall" but the
> requirement just didn't seem to be there anywhere obvious.

I quoted them. You should really read the "read" manpage instead of the
write manpage, the functions have quite different return values.

> I would be happy to see the standard clarified. If you're interested,
> the people at http://opengroup.org/austin and
> http://austingroupbugs.net/ are generally a helpful bunch.

The standard is absolutely clear, really.

> Yes, that's mostly true. The idiom "if (read(..., count) == count)"
> is not uncommon and is safe when used carefully (for example, when
> used with files on disk rather than terminals, tapes, or pipes).

Well, not on linux.

> has to include strerror(errno) to be useful. There are plenty of
> cases where that isn't true. But isn't it the common case?

Both are common cases, as is no error checking.

> > The authors of gnu tar and many existing
> > applications apparently disagree, as do I.
>
> GNU tar mostly uses the safe_read function from gnulib, except when
> reading the first line of CACHEDIR.TAG files, if I read it correctly.

It still gives a short read or equivalent error message on short reads,
depending on version and input device/file.

> If you'd like, you can retitle this bug.

I don't know how.

> Also keep in mind that if you can convince upstream (presumably by

I am really busy with writing and maintaining a LOT of free software. I
consider it my duty to report bugs, but if upstream doesn't feel like
fixing it, that's not really my problem. I already workarounded the
sendfile problem in libeio for example, and would have done so in any
case, as buggy kernel versions will be in use for a long time.

> > that would require read(2) to be fixed.
>
> If read(2) and write(2) start handling larger chunks, there's
> obviously no reason for sendfile(2) not to, too. The whole point of
> the exercise is to avoid buffer overflows and other logic errors in
> sloppily written, obscure drivers.

Well, the result is some non-posix code that breaks real-world programs that
work with both posix semantics as well as historic unix semantics.

I wouldn't be surprised if breakage and data loss ensues more in the
future - right now, there are not so many big files around (mostly dvd
etc. images), but files tend to grow.

--
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ / // / / / schmorp@schmorp.de
-=====/_/_//_/\_,_/ /_/\_



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610125928.GA5217@schmorp.de">http://lists.debian.org/20110610125928.GA5217@schmorp.de
 
Old 06-10-2011, 01:08 PM
Marc Lehmann
 
Default Bug#629994: sendfile returns early without user-visible reason

On Fri, Jun 10, 2011 at 02:29:46PM +0200, Bastian Blank <waldi@debian.org> wrote:
> > What the fuck, it's buggy, indeed:
> > read(0, ""..., 3298534883328) = 2147479552
>
> What is the bug?

Please *read* the bug report.

> > transfers as many bytes as can be transferred, and not stop a random
> > amount earlier
>
> Please quote the standard on this.

Please *read* the bug report. You can also read the full read(2) sus
manpage online at www.opengroup.org.

> Please note that A => B does not imply !A => !B (A == {not-regular,
> signaled}, B == short-write).

Reading standards is notoriously difficult, I admit. The behaviour of read
is specified to read the requested number of bytes, if possible.

The standard gives an exception list where applications can deviate from
the behaviour and read less.

For other exceptions, you would have to find a part of the standard that
actually allows it.

So where in the standard is this read behaviour allowed?

> > This works fine as long as the OS follows posix.
>
> Linux is no fully compliant POSIX system.

This is painfully clear. It's also not a unix kernel.

What should be clear to you though, is that posix semantics for read that a
lot of real world programs rely on is useful regardless of whether it is
specified in posix or not, and regardless of whether it is specified

What you also totally miss is that this is also historical unix behaviour,
and linux *is* a unix kernel clone.

If you actually had read the full bug report you could have known that
already.

> Bastian

Please, next time, actually try to read the bug report before sending a
superfluous mail, it just wastes time for those people interested.

--
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ / // / / / schmorp@schmorp.de
-=====/_/_//_/\_,_/ /_/\_



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610130810.GB5217@schmorp.de">http://lists.debian.org/20110610130810.GB5217@schmorp.de
 
Old 06-10-2011, 01:17 PM
Jonathan Nieder
 
Default Bug#629994: sendfile returns early without user-visible reason

clone 629994 -1
reassign -1 manpages-dev 3.27-1
severity -1 = minor
tags -1 = upstream lfs
retitle -1 read/write/readv/writev/sendfile(2): undocumented cap on number of bytes read/written
quit

Marc Lehmann wrote:
(out of order for convenience)

> I am really busy with writing and maintaining a LOT of free software. I
> consider it my duty to report bugs, but if upstream doesn't feel like
> fixing it, that's not really my problem.

Indeed, thanks for reporting it.

>> If you'd like, you can retitle this bug.
>
> I don't know how.

http://www.debian.org/Bugs/Developer

> Well, the relevant posix manpage for read is read, not the one for write.
> read is clear.

I looked at both. I suppose we will have to agree to disagree here ---
in standardese "may" and "shall" both have clear meanings, and unless I
missed something obvious the wording you pointed to did not place the
requirement we are talking about on the implementation.

Not that it matters much, since on one hand the text is obviously less
clear than ideal and there are people in the Austin Group who would be
glad to improve it if asked (presumably by making it match the intent
better) and on the other hand, Linux does not follow Posix rigidly
anyway.

But I had thought I could point to that easy avenue for improving
documentation and portability in the wider world.

Cloning the bug so the confusing behavior can be documented. I will
leave it to the kernel maintainers to decide whether the kernel bug
should be left open.

Regards,
Jonathan



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610131722.GA2827@elie">http://lists.debian.org/20110610131722.GA2827@elie
 
Old 06-10-2011, 01:46 PM
Bastian Blank
 
Default Bug#629994: sendfile returns early without user-visible reason

Please move this discussion somewhere else. I consider this no bug at all
and you failed to convince me.

On Fri, Jun 10, 2011 at 03:08:11PM +0200, Marc Lehmann wrote:
> On Fri, Jun 10, 2011 at 02:29:46PM +0200, Bastian Blank <waldi@debian.org> wrote:
> > > What the fuck, it's buggy, indeed:
> > > read(0, ""..., 3298534883328) = 2147479552
> > What is the bug?
> Please *read* the bug report.

No. The bug report is about sendfile, not read.

> > > transfers as many bytes as can be transferred, and not stop a random
> > > amount earlier
> > Please quote the standard on this.
> Please *read* the bug report. You can also read the full read(2) sus
> manpage online at www.opengroup.org.

Please show the actual URI.
http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html
sounds different for me.

> > Please note that A => B does not imply !A => !B (A == {not-regular,
> > signaled}, B == short-write).
> Reading standards is notoriously difficult, I admit. The behaviour of read
> is specified to read the requested number of bytes, if possible.

Please quote. You have not yet.

> The standard gives an exception list where applications can deviate from
> the behaviour and read less.

It defines some behaviours where it "shall" produce a different result.
They are not defined as exception of something.

> For other exceptions, you would have to find a part of the standard that
> actually allows it.

To find an exception, there needs to be a rule. Where is it?

Bastian

--
Even historians fail to learn from history -- they repeat the same mistakes.
-- John Gill, "Patterns of Force", stardate 2534.7



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610134646.GA23864@wavehammer.waldi.eu.org">ht tp://lists.debian.org/20110610134646.GA23864@wavehammer.waldi.eu.org
 
Old 06-10-2011, 01:47 PM
Marc Lehmann
 
Default Bug#629994: sendfile returns early without user-visible reason

On Fri, Jun 10, 2011 at 08:17:22AM -0500, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > Well, the relevant posix manpage for read is read, not the one for write.
> > read is clear.
>
> I looked at both. I suppose we will have to agree to disagree here ---

Yes, but thats your lack of understanding prose logic, not a valid
disagreement:

> in standardese "may" and "shall" both have clear meanings, and unless I
> missed something obvious the wording you pointed to did not place the
> requirement we are talking about on the implementation.

Thats true - the wording, paraphrased, is "an implementation may return less
under these conditions".

That indeed is true. It is not *required* to return less, only *allowed*
to. It is indeed not "the implemenentation shall return less under these
conditions".

The point is that nowhere else is it *allowed* to return less under
*other* conditions. posix read *has* to try to read those bytes, and linux
read doesn't.

If you really still disagree, then please quote where the standard
*allows* the exception to the semantics of read, namely that it doesn'T
need to attempt to read those bytes, because it's clearly not doing it -
if it attempted that, then it would be able to read them.

This is the source of the confusion. The standard is clear enough here,
the problem is not "may" vs. "shall" but that you ignore that nowhere does
it allow this behaviour. What I quoted is only the list of allowed
exceptions, where the implementation may deviate.

Again, it is absolutely clear both to posix as well as to common sense
what read should do: it should read what i ask it to read. The standard
specifies a list of conditions where it is allowed NOT to do that. It also
specifies a list of things it may do elsewhere, but none of these allow
the behaviour, namely *not* trying to read those bytes.

> Not that it matters much, since on one hand the text is obviously less
> clear than ideal and there are people in the Austin Group who would be

The text is absolutely clear. The problem is that you are confused about the
prose logic. Again, the structure is like this:

a) read does A.
b) read may not do A under these conditions.

According to you, that means

c) read may not do A under any conditions.

But thats not what is written there.

Or maybe you expect this:

a) read shall A.

But thats not true either - the normative portion of the standard is
normative, not just sentences that have "shall" or "may" in them.

> better) and on the other hand, Linux does not follow Posix rigidly
> anyway.

Well, then programs such as dd which are currently broken for large
blocksizes, need to be fixed. Example on a 8.7gb file:

dd if=x of=/dev/null bs=1072x1024x1024
7+1 records in
7+1 records out

dd if=x of=/dev/null bs=3072x1024x1024
0+5 records in
0+5 records out

If you now add a conversion or reblocking you have instant data loss. Some
tars or other programs break with large blocksizes etc.

I find it rather disingenious to argue whether linux is posix or not. As I
wrote multiple times, the behaviour is expected by *existing* programs, is
historically correct, and is expected even by programs in debian, whether
posix or not. gnu programs expect it. You acknowledges this at least
partially yourself. Why bother with totally ridiculous misinterpretations
of POSIX? What's the win for debian to talk this bug down that breaks
existing applications?

To me, it doesn't matter that much if all userspace programs get fixed to
work around this bug, or if the kernel gets fixed, or nothing gets fixed.

However, this is obviously a problem in the kernel, and not with all those
userspace prorgams that work portably on other systems but not on any
gnu/linux system.

I mean, whats the point? Are you really arguing that all these userspace
programs are broken? Good luck convincing their authors.

I also find it weird that there are people replying to me who clearly
didn't even bother to read the bugreport.

> But I had thought I could point to that easy avenue for improving
> documentation and portability in the wider world.

Sure, here's one for dd, you can adapt it for other programs, but don't
expect me to list all of them.

"dd is broken for blocksizes near or large than 2gb, as the linux
kernel isn't capable of reading more then 0x7fff000 bytes in one go,
and dd expects psoix compliance, which linux does not care for.".

(or whatever the limit is)

--
The choice of a Deliantra, the free code+content MORPG
-----==- _GNU_ http://www.deliantra.net
----==-- _ generation
---==---(_)__ __ ____ __ Marc Lehmann
--==---/ / _ / // / / / schmorp@schmorp.de
-=====/_/_//_/\_,_/ /_/\_



--
To UNSUBSCRIBE, email to debian-kernel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110610134706.GA5307@schmorp.de">http://lists.debian.org/20110610134706.GA5307@schmorp.de
 

Thread Tools




All times are GMT. The time now is 10:37 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org