2019-05-19 12:08:55 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
2006-03-30 13:15:30 +00:00
|
|
|
/*
|
|
|
|
* "splice": joining two ropes together by interweaving their strands.
|
|
|
|
*
|
|
|
|
* This is the "extended pipe" functionality, where a pipe is used as
|
|
|
|
* an arbitrary in-memory buffer. Think of a pipe as a small kernel
|
|
|
|
* buffer that you can use to transfer data from one end to the other.
|
|
|
|
*
|
|
|
|
* The traditional unix read/write is extended with a "splice()" operation
|
|
|
|
* that transfers data buffers to or from a pipe buffer.
|
|
|
|
*
|
|
|
|
* Named by Larry McVoy, original implementation from Linus, extended by
|
2006-04-11 11:56:34 +00:00
|
|
|
* Jens to support splicing to files, network, direct splicing, etc and
|
|
|
|
* fixing lots of bugs.
|
2006-03-30 13:15:30 +00:00
|
|
|
*
|
2006-09-04 13:41:16 +00:00
|
|
|
* Copyright (C) 2005-2006 Jens Axboe <axboe@kernel.dk>
|
2006-04-11 11:56:34 +00:00
|
|
|
* Copyright (C) 2005-2006 Linus Torvalds <torvalds@osdl.org>
|
|
|
|
* Copyright (C) 2006 Ingo Molnar <mingo@elte.hu>
|
2006-03-30 13:15:30 +00:00
|
|
|
*
|
|
|
|
*/
|
2016-11-01 13:40:16 +00:00
|
|
|
#include <linux/bvec.h>
|
2006-03-30 13:15:30 +00:00
|
|
|
#include <linux/fs.h>
|
|
|
|
#include <linux/file.h>
|
|
|
|
#include <linux/pagemap.h>
|
2007-06-04 07:59:47 +00:00
|
|
|
#include <linux/splice.h>
|
memcg: synchronized LRU
A big patch for changing memcg's LRU semantics.
Now,
- page_cgroup is linked to mem_cgroup's its own LRU (per zone).
- LRU of page_cgroup is not synchronous with global LRU.
- page and page_cgroup is one-to-one and statically allocated.
- To find page_cgroup is on what LRU, you have to check pc->mem_cgroup as
- lru = page_cgroup_zoneinfo(pc, nid_of_pc, zid_of_pc);
- SwapCache is handled.
And, when we handle LRU list of page_cgroup, we do following.
pc = lookup_page_cgroup(page);
lock_page_cgroup(pc); .....................(1)
mz = page_cgroup_zoneinfo(pc);
spin_lock(&mz->lru_lock);
.....add to LRU
spin_unlock(&mz->lru_lock);
unlock_page_cgroup(pc);
But (1) is spin_lock and we have to be afraid of dead-lock with zone->lru_lock.
So, trylock() is used at (1), now. Without (1), we can't trust "mz" is correct.
This is a trial to remove this dirty nesting of locks.
This patch changes mz->lru_lock to be zone->lru_lock.
Then, above sequence will be written as
spin_lock(&zone->lru_lock); # in vmscan.c or swap.c via global LRU
mem_cgroup_add/remove/etc_lru() {
pc = lookup_page_cgroup(page);
mz = page_cgroup_zoneinfo(pc);
if (PageCgroupUsed(pc)) {
....add to LRU
}
spin_lock(&zone->lru_lock); # in vmscan.c or swap.c via global LRU
This is much simpler.
(*) We're safe even if we don't take lock_page_cgroup(pc). Because..
1. When pc->mem_cgroup can be modified.
- at charge.
- at account_move().
2. at charge
the PCG_USED bit is not set before pc->mem_cgroup is fixed.
3. at account_move()
the page is isolated and not on LRU.
Pros.
- easy for maintenance.
- memcg can make use of laziness of pagevec.
- we don't have to duplicated LRU/Active/Unevictable bit in page_cgroup.
- LRU status of memcg will be synchronized with global LRU's one.
- # of locks are reduced.
- account_move() is simplified very much.
Cons.
- may increase cost of LRU rotation.
(no impact if memcg is not configured.)
Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Li Zefan <lizf@cn.fujitsu.com>
Cc: Balbir Singh <balbir@in.ibm.com>
Cc: Pavel Emelyanov <xemul@openvz.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-01-08 02:08:01 +00:00
|
|
|
#include <linux/memcontrol.h>
|
2006-03-30 13:15:30 +00:00
|
|
|
#include <linux/mm_inline.h>
|
2006-03-30 13:16:46 +00:00
|
|
|
#include <linux/swap.h>
|
2006-04-02 21:04:46 +00:00
|
|
|
#include <linux/writeback.h>
|
2011-11-17 04:57:37 +00:00
|
|
|
#include <linux/export.h>
|
2006-04-02 21:04:46 +00:00
|
|
|
#include <linux/syscalls.h>
|
2006-04-26 08:59:21 +00:00
|
|
|
#include <linux/uio.h>
|
2023-03-22 06:25:19 +00:00
|
|
|
#include <linux/fsnotify.h>
|
2007-07-13 09:44:32 +00:00
|
|
|
#include <linux/security.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 08:04:11 +00:00
|
|
|
#include <linux/gfp.h>
|
2023-06-07 18:19:09 +00:00
|
|
|
#include <linux/net.h>
|
2012-04-05 03:05:35 +00:00
|
|
|
#include <linux/socket.h>
|
2017-02-02 18:15:33 +00:00
|
|
|
#include <linux/sched/signal.h>
|
|
|
|
|
2013-03-20 17:19:30 +00:00
|
|
|
#include "internal.h"
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2023-04-24 22:32:55 +00:00
|
|
|
/*
|
|
|
|
* Splice doesn't support FMODE_NOWAIT. Since pipes may set this flag to
|
|
|
|
* indicate they support non-blocking reads or writes, we must clear it
|
|
|
|
* here if set to avoid blocking other users of this pipe if splice is
|
|
|
|
* being done on it.
|
|
|
|
*/
|
|
|
|
static noinline void noinline pipe_clear_nowait(struct file *file)
|
|
|
|
{
|
|
|
|
fmode_t fmode = READ_ONCE(file->f_mode);
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (!(fmode & FMODE_NOWAIT))
|
|
|
|
break;
|
|
|
|
} while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT));
|
|
|
|
}
|
|
|
|
|
2006-04-02 21:05:09 +00:00
|
|
|
/*
|
|
|
|
* Attempt to steal a page from a pipe buffer. This should perhaps go into
|
|
|
|
* a vm helper function, it's already simplified quite a bit by the
|
|
|
|
* addition of remove_mapping(). If success is returned, the caller may
|
|
|
|
* attempt to reuse this page for another destination.
|
|
|
|
*/
|
2020-05-20 15:58:16 +00:00
|
|
|
static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,
|
|
|
|
struct pipe_buffer *buf)
|
2006-03-30 13:16:46 +00:00
|
|
|
{
|
2022-02-13 03:48:55 +00:00
|
|
|
struct folio *folio = page_folio(buf->page);
|
2006-06-20 13:01:12 +00:00
|
|
|
struct address_space *mapping;
|
2006-03-30 13:16:46 +00:00
|
|
|
|
2022-02-12 04:39:03 +00:00
|
|
|
folio_lock(folio);
|
2006-04-19 13:57:31 +00:00
|
|
|
|
2022-02-12 04:39:03 +00:00
|
|
|
mapping = folio_mapping(folio);
|
2006-06-20 13:01:12 +00:00
|
|
|
if (mapping) {
|
2022-02-12 04:39:03 +00:00
|
|
|
WARN_ON(!folio_test_uptodate(folio));
|
2006-03-30 13:16:46 +00:00
|
|
|
|
2006-06-20 13:01:12 +00:00
|
|
|
/*
|
|
|
|
* At least for ext2 with nobh option, we need to wait on
|
2022-02-12 04:39:03 +00:00
|
|
|
* writeback completing on this folio, since we'll remove it
|
2006-06-20 13:01:12 +00:00
|
|
|
* from the pagecache. Otherwise truncate wont wait on the
|
2022-02-12 04:39:03 +00:00
|
|
|
* folio, allowing the disk blocks to be reused by someone else
|
2006-06-20 13:01:12 +00:00
|
|
|
* before we actually wrote our data to them. fs corruption
|
|
|
|
* ensues.
|
|
|
|
*/
|
2022-02-12 04:39:03 +00:00
|
|
|
folio_wait_writeback(folio);
|
2006-04-02 21:10:32 +00:00
|
|
|
|
mm: merge folio_has_private()/filemap_release_folio() call pairs
Patch series "mm, netfs, fscache: Stop read optimisation when folio
removed from pagecache", v7.
This fixes an optimisation in fscache whereby we don't read from the cache
for a particular file until we know that there's data there that we don't
have in the pagecache. The problem is that I'm no longer using PG_fscache
(aka PG_private_2) to indicate that the page is cached and so I don't get
a notification when a cached page is dropped from the pagecache.
The first patch merges some folio_has_private() and
filemap_release_folio() pairs and introduces a helper,
folio_needs_release(), to indicate if a release is required.
The second patch is the actual fix. Following Willy's suggestions[1], it
adds an AS_RELEASE_ALWAYS flag to an address_space that will make
filemap_release_folio() always call ->release_folio(), even if
PG_private/PG_private_2 aren't set. folio_needs_release() is altered to
add a check for this.
This patch (of 2):
Make filemap_release_folio() check folio_has_private(). Then, in most
cases, where a call to folio_has_private() is immediately followed by a
call to filemap_release_folio(), we can get rid of the test in the pair.
There are a couple of sites in mm/vscan.c that this can't so easily be
done. In shrink_folio_list(), there are actually three cases (something
different is done for incompletely invalidated buffers), but
filemap_release_folio() elides two of them.
In shrink_active_list(), we don't have have the folio lock yet, so the
check allows us to avoid locking the page unnecessarily.
A wrapper function to check if a folio needs release is provided for those
places that still need to do it in the mm/ directory. This will acquire
additional parts to the condition in a future patch.
After this, the only remaining caller of folio_has_private() outside of
mm/ is a check in fuse.
Link: https://lkml.kernel.org/r/20230628104852.3391651-1-dhowells@redhat.com
Link: https://lkml.kernel.org/r/20230628104852.3391651-2-dhowells@redhat.com
Reported-by: Rohith Surabattula <rohiths.msft@gmail.com>
Suggested-by: Matthew Wilcox <willy@infradead.org>
Signed-off-by: David Howells <dhowells@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Steve French <sfrench@samba.org>
Cc: Shyam Prasad N <nspmangalore@gmail.com>
Cc: Rohith Surabattula <rohiths.msft@gmail.com>
Cc: Dave Wysochanski <dwysocha@redhat.com>
Cc: Dominique Martinet <asmadeus@codewreck.org>
Cc: Ilya Dryomov <idryomov@gmail.com>
Cc: "Theodore Ts'o" <tytso@mit.edu>
Cc: Andreas Dilger <adilger.kernel@dilger.ca>
Cc: Xiubo Li <xiubli@redhat.com>
Cc: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-06-28 10:48:51 +00:00
|
|
|
if (!filemap_release_folio(folio, GFP_KERNEL))
|
2008-05-20 19:27:41 +00:00
|
|
|
goto out_unlock;
|
2006-04-02 21:04:46 +00:00
|
|
|
|
2006-06-20 13:01:12 +00:00
|
|
|
/*
|
|
|
|
* If we succeeded in removing the mapping, set LRU flag
|
|
|
|
* and return good.
|
|
|
|
*/
|
2022-02-13 03:48:55 +00:00
|
|
|
if (remove_mapping(mapping, folio)) {
|
2006-06-20 13:01:12 +00:00
|
|
|
buf->flags |= PIPE_BUF_FLAG_LRU;
|
2020-05-20 15:58:16 +00:00
|
|
|
return true;
|
2006-06-20 13:01:12 +00:00
|
|
|
}
|
2006-04-19 13:57:31 +00:00
|
|
|
}
|
2006-03-30 13:16:46 +00:00
|
|
|
|
2006-06-20 13:01:12 +00:00
|
|
|
/*
|
2022-02-12 04:39:03 +00:00
|
|
|
* Raced with truncate or failed to remove folio from current
|
2006-06-20 13:01:12 +00:00
|
|
|
* address space, unlock and return failure.
|
|
|
|
*/
|
2008-05-20 19:27:41 +00:00
|
|
|
out_unlock:
|
2022-02-12 04:39:03 +00:00
|
|
|
folio_unlock(folio);
|
2020-05-20 15:58:16 +00:00
|
|
|
return false;
|
2006-03-30 13:16:46 +00:00
|
|
|
}
|
|
|
|
|
2006-05-03 08:41:33 +00:00
|
|
|
static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe,
|
2006-03-30 13:15:30 +00:00
|
|
|
struct pipe_buffer *buf)
|
|
|
|
{
|
mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros
PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} macros were introduced *long* time
ago with promise that one day it will be possible to implement page
cache with bigger chunks than PAGE_SIZE.
This promise never materialized. And unlikely will.
We have many places where PAGE_CACHE_SIZE assumed to be equal to
PAGE_SIZE. And it's constant source of confusion on whether
PAGE_CACHE_* or PAGE_* constant should be used in a particular case,
especially on the border between fs and mm.
Global switching to PAGE_CACHE_SIZE != PAGE_SIZE would cause to much
breakage to be doable.
Let's stop pretending that pages in page cache are special. They are
not.
The changes are pretty straight-forward:
- <foo> << (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- <foo> >> (PAGE_CACHE_SHIFT - PAGE_SHIFT) -> <foo>;
- PAGE_CACHE_{SIZE,SHIFT,MASK,ALIGN} -> PAGE_{SIZE,SHIFT,MASK,ALIGN};
- page_cache_get() -> get_page();
- page_cache_release() -> put_page();
This patch contains automated changes generated with coccinelle using
script below. For some reason, coccinelle doesn't patch header files.
I've called spatch for them manually.
The only adjustment after coccinelle is revert of changes to
PAGE_CAHCE_ALIGN definition: we are going to drop it later.
There are few places in the code where coccinelle didn't reach. I'll
fix them manually in a separate patch. Comments and documentation also
will be addressed with the separate patch.
virtual patch
@@
expression E;
@@
- E << (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
expression E;
@@
- E >> (PAGE_CACHE_SHIFT - PAGE_SHIFT)
+ E
@@
@@
- PAGE_CACHE_SHIFT
+ PAGE_SHIFT
@@
@@
- PAGE_CACHE_SIZE
+ PAGE_SIZE
@@
@@
- PAGE_CACHE_MASK
+ PAGE_MASK
@@
expression E;
@@
- PAGE_CACHE_ALIGN(E)
+ PAGE_ALIGN(E)
@@
expression E;
@@
- page_cache_get(E)
+ get_page(E)
@@
expression E;
@@
- page_cache_release(E)
+ put_page(E)
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-04-01 12:29:47 +00:00
|
|
|
put_page(buf->page);
|
2006-05-03 08:35:26 +00:00
|
|
|
buf->flags &= ~PIPE_BUF_FLAG_LRU;
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
|
|
|
|
2007-06-12 18:51:32 +00:00
|
|
|
/*
|
|
|
|
* Check whether the contents of buf is OK to access. Since the content
|
|
|
|
* is a page cache page, IO may be in flight.
|
|
|
|
*/
|
2007-06-14 11:10:48 +00:00
|
|
|
static int page_cache_pipe_buf_confirm(struct pipe_inode_info *pipe,
|
|
|
|
struct pipe_buffer *buf)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2023-08-21 14:15:41 +00:00
|
|
|
struct folio *folio = page_folio(buf->page);
|
2006-04-10 07:04:41 +00:00
|
|
|
int err;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2023-08-21 14:15:41 +00:00
|
|
|
if (!folio_test_uptodate(folio)) {
|
|
|
|
folio_lock(folio);
|
2006-04-10 07:04:41 +00:00
|
|
|
|
|
|
|
/*
|
2023-08-21 14:15:41 +00:00
|
|
|
* Folio got truncated/unhashed. This will cause a 0-byte
|
2006-04-11 11:57:21 +00:00
|
|
|
* splice, if this is the first page.
|
2006-04-10 07:04:41 +00:00
|
|
|
*/
|
2023-08-21 14:15:41 +00:00
|
|
|
if (!folio->mapping) {
|
2006-04-10 07:04:41 +00:00
|
|
|
err = -ENODATA;
|
|
|
|
goto error;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2006-04-10 07:04:41 +00:00
|
|
|
/*
|
2006-04-11 11:57:21 +00:00
|
|
|
* Uh oh, read-error from disk.
|
2006-04-10 07:04:41 +00:00
|
|
|
*/
|
2023-08-21 14:15:41 +00:00
|
|
|
if (!folio_test_uptodate(folio)) {
|
2006-04-10 07:04:41 +00:00
|
|
|
err = -EIO;
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
|
2023-08-21 14:15:41 +00:00
|
|
|
/* Folio is ok after all, we are done */
|
|
|
|
folio_unlock(folio);
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
|
|
|
|
2006-05-01 17:59:03 +00:00
|
|
|
return 0;
|
2006-04-10 07:04:41 +00:00
|
|
|
error:
|
2023-08-21 14:15:41 +00:00
|
|
|
folio_unlock(folio);
|
2006-05-01 17:59:03 +00:00
|
|
|
return err;
|
2006-04-11 13:51:17 +00:00
|
|
|
}
|
|
|
|
|
2011-07-26 00:12:32 +00:00
|
|
|
const struct pipe_buf_operations page_cache_pipe_buf_ops = {
|
2020-05-20 15:58:16 +00:00
|
|
|
.confirm = page_cache_pipe_buf_confirm,
|
|
|
|
.release = page_cache_pipe_buf_release,
|
|
|
|
.try_steal = page_cache_pipe_buf_try_steal,
|
|
|
|
.get = generic_pipe_buf_get,
|
2006-03-30 13:15:30 +00:00
|
|
|
};
|
|
|
|
|
2020-05-20 15:58:16 +00:00
|
|
|
static bool user_page_pipe_buf_try_steal(struct pipe_inode_info *pipe,
|
|
|
|
struct pipe_buffer *buf)
|
2006-04-26 08:59:21 +00:00
|
|
|
{
|
2006-05-01 18:02:33 +00:00
|
|
|
if (!(buf->flags & PIPE_BUF_FLAG_GIFT))
|
2020-05-20 15:58:16 +00:00
|
|
|
return false;
|
2006-05-01 18:02:33 +00:00
|
|
|
|
2006-05-03 08:35:26 +00:00
|
|
|
buf->flags |= PIPE_BUF_FLAG_LRU;
|
2020-05-20 15:58:16 +00:00
|
|
|
return generic_pipe_buf_try_steal(pipe, buf);
|
2006-04-26 08:59:21 +00:00
|
|
|
}
|
|
|
|
|
2006-12-13 08:34:04 +00:00
|
|
|
static const struct pipe_buf_operations user_page_pipe_buf_ops = {
|
2020-05-20 15:58:16 +00:00
|
|
|
.release = page_cache_pipe_buf_release,
|
|
|
|
.try_steal = user_page_pipe_buf_try_steal,
|
|
|
|
.get = generic_pipe_buf_get,
|
2006-04-26 08:59:21 +00:00
|
|
|
};
|
|
|
|
|
2011-05-23 17:58:53 +00:00
|
|
|
static void wakeup_pipe_readers(struct pipe_inode_info *pipe)
|
|
|
|
{
|
|
|
|
smp_mb();
|
pipe: use exclusive waits when reading or writing
This makes the pipe code use separate wait-queues and exclusive waiting
for readers and writers, avoiding a nasty thundering herd problem when
there are lots of readers waiting for data on a pipe (or, less commonly,
lots of writers waiting for a pipe to have space).
While this isn't a common occurrence in the traditional "use a pipe as a
data transport" case, where you typically only have a single reader and
a single writer process, there is one common special case: using a pipe
as a source of "locking tokens" rather than for data communication.
In particular, the GNU make jobserver code ends up using a pipe as a way
to limit parallelism, where each job consumes a token by reading a byte
from the jobserver pipe, and releases the token by writing a byte back
to the pipe.
This pattern is fairly traditional on Unix, and works very well, but
will waste a lot of time waking up a lot of processes when only a single
reader needs to be woken up when a writer releases a new token.
A simplified test-case of just this pipe interaction is to create 64
processes, and then pass a single token around between them (this
test-case also intentionally passes another token that gets ignored to
test the "wake up next" logic too, in case anybody wonders about it):
#include <unistd.h>
int main(int argc, char **argv)
{
int fd[2], counters[2];
pipe(fd);
counters[0] = 0;
counters[1] = -1;
write(fd[1], counters, sizeof(counters));
/* 64 processes */
fork(); fork(); fork(); fork(); fork(); fork();
do {
int i;
read(fd[0], &i, sizeof(i));
if (i < 0)
continue;
counters[0] = i+1;
write(fd[1], counters, (1+(i & 1)) *sizeof(int));
} while (counters[0] < 1000000);
return 0;
}
and in a perfect world, passing that token around should only cause one
context switch per transfer, when the writer of a token causes a
directed wakeup of just a single reader.
But with the "writer wakes all readers" model we traditionally had, on
my test box the above case causes more than an order of magnitude more
scheduling: instead of the expected ~1M context switches, "perf stat"
shows
231,852.37 msec task-clock # 15.857 CPUs utilized
11,250,961 context-switches # 0.049 M/sec
616,304 cpu-migrations # 0.003 M/sec
1,648 page-faults # 0.007 K/sec
1,097,903,998,514 cycles # 4.735 GHz
120,781,778,352 instructions # 0.11 insn per cycle
27,997,056,043 branches # 120.754 M/sec
283,581,233 branch-misses # 1.01% of all branches
14.621273891 seconds time elapsed
0.018243000 seconds user
3.611468000 seconds sys
before this commit.
After this commit, I get
5,229.55 msec task-clock # 3.072 CPUs utilized
1,212,233 context-switches # 0.232 M/sec
103,951 cpu-migrations # 0.020 M/sec
1,328 page-faults # 0.254 K/sec
21,307,456,166 cycles # 4.074 GHz
12,947,819,999 instructions # 0.61 insn per cycle
2,881,985,678 branches # 551.096 M/sec
64,267,015 branch-misses # 2.23% of all branches
1.702148350 seconds time elapsed
0.004868000 seconds user
0.110786000 seconds sys
instead. Much better.
[ Note! This kernel improvement seems to be very good at triggering a
race condition in the make jobserver (in GNU make 4.2.1) for me. It's
a long known bug that was fixed back in June 2017 by GNU make commit
b552b0525198 ("[SV 51159] Use a non-blocking read with pselect to
avoid hangs.").
But there wasn't a new release of GNU make until 4.3 on Jan 19 2020,
so a number of distributions may still have the buggy version. Some
have backported the fix to their 4.2.1 release, though, and even
without the fix it's quite timing-dependent whether the bug actually
is hit. ]
Josh Triplett says:
"I've been hammering on your pipe fix patch (switching to exclusive
wait queues) for a month or so, on several different systems, and I've
run into no issues with it. The patch *substantially* improves
parallel build times on large (~100 CPU) systems, both with parallel
make and with other things that use make's pipe-based jobserver.
All current distributions (including stable and long-term stable
distributions) have versions of GNU make that no longer have the
jobserver bug"
Tested-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-12-09 17:48:27 +00:00
|
|
|
if (waitqueue_active(&pipe->rd_wait))
|
|
|
|
wake_up_interruptible(&pipe->rd_wait);
|
2011-05-23 17:58:53 +00:00
|
|
|
kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
|
|
|
|
}
|
|
|
|
|
2007-06-21 11:10:21 +00:00
|
|
|
/**
|
|
|
|
* splice_to_pipe - fill passed data into a pipe
|
|
|
|
* @pipe: pipe to fill
|
|
|
|
* @spd: data to fill
|
|
|
|
*
|
|
|
|
* Description:
|
2007-07-27 06:08:51 +00:00
|
|
|
* @spd contains a map of pages and len/offset tuples, along with
|
2007-06-21 11:10:21 +00:00
|
|
|
* the struct pipe_buf_operations associated with these pages. This
|
|
|
|
* function will link that data to the pipe.
|
|
|
|
*
|
2006-04-02 21:05:09 +00:00
|
|
|
*/
|
2007-06-04 07:59:47 +00:00
|
|
|
ssize_t splice_to_pipe(struct pipe_inode_info *pipe,
|
|
|
|
struct splice_pipe_desc *spd)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2007-06-15 11:14:22 +00:00
|
|
|
unsigned int spd_pages = spd->nr_pages;
|
2019-11-15 13:30:32 +00:00
|
|
|
unsigned int tail = pipe->tail;
|
|
|
|
unsigned int head = pipe->head;
|
|
|
|
unsigned int mask = pipe->ring_size - 1;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret = 0;
|
|
|
|
int page_nr = 0;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2016-03-10 20:19:06 +00:00
|
|
|
if (!spd_pages)
|
|
|
|
return 0;
|
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
if (unlikely(!pipe->readers)) {
|
|
|
|
send_sig(SIGPIPE, current, 0);
|
|
|
|
ret = -EPIPE;
|
|
|
|
goto out;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2019-10-16 15:47:32 +00:00
|
|
|
while (!pipe_full(head, tail, pipe->max_usage)) {
|
2019-11-15 13:30:32 +00:00
|
|
|
struct pipe_buffer *buf = &pipe->bufs[head & mask];
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
buf->page = spd->pages[page_nr];
|
|
|
|
buf->offset = spd->partial[page_nr].offset;
|
|
|
|
buf->len = spd->partial[page_nr].len;
|
|
|
|
buf->private = spd->partial[page_nr].private;
|
|
|
|
buf->ops = spd->ops;
|
2017-02-16 16:49:02 +00:00
|
|
|
buf->flags = 0;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
head++;
|
|
|
|
pipe->head = head;
|
2016-09-18 00:44:45 +00:00
|
|
|
page_nr++;
|
|
|
|
ret += buf->len;
|
2006-04-02 19:46:35 +00:00
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
if (!--spd->nr_pages)
|
2006-03-30 13:15:30 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
if (!ret)
|
|
|
|
ret = -EAGAIN;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
out:
|
2007-06-15 11:14:22 +00:00
|
|
|
while (page_nr < spd_pages)
|
2007-11-07 07:29:47 +00:00
|
|
|
spd->spd_release(spd, page_nr++);
|
2006-03-30 13:15:30 +00:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
2015-05-21 15:00:01 +00:00
|
|
|
EXPORT_SYMBOL_GPL(splice_to_pipe);
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2016-09-18 02:38:20 +00:00
|
|
|
ssize_t add_to_pipe(struct pipe_inode_info *pipe, struct pipe_buffer *buf)
|
|
|
|
{
|
2019-11-15 13:30:32 +00:00
|
|
|
unsigned int head = pipe->head;
|
|
|
|
unsigned int tail = pipe->tail;
|
|
|
|
unsigned int mask = pipe->ring_size - 1;
|
2016-09-18 02:38:20 +00:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (unlikely(!pipe->readers)) {
|
|
|
|
send_sig(SIGPIPE, current, 0);
|
|
|
|
ret = -EPIPE;
|
2019-10-16 15:47:32 +00:00
|
|
|
} else if (pipe_full(head, tail, pipe->max_usage)) {
|
2016-09-18 02:38:20 +00:00
|
|
|
ret = -EAGAIN;
|
|
|
|
} else {
|
2019-11-15 13:30:32 +00:00
|
|
|
pipe->bufs[head & mask] = *buf;
|
|
|
|
pipe->head = head + 1;
|
2016-09-18 02:38:20 +00:00
|
|
|
return buf->len;
|
|
|
|
}
|
2016-09-27 08:45:12 +00:00
|
|
|
pipe_buf_release(pipe, buf);
|
2016-09-18 02:38:20 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(add_to_pipe);
|
|
|
|
|
2010-05-20 08:43:18 +00:00
|
|
|
/*
|
|
|
|
* Check if we need to grow the arrays holding pages and partial page
|
|
|
|
* descriptions.
|
|
|
|
*/
|
2012-06-12 13:24:40 +00:00
|
|
|
int splice_grow_spd(const struct pipe_inode_info *pipe, struct splice_pipe_desc *spd)
|
2010-05-20 08:43:18 +00:00
|
|
|
{
|
2019-10-16 15:47:32 +00:00
|
|
|
unsigned int max_usage = READ_ONCE(pipe->max_usage);
|
2012-06-12 13:24:40 +00:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
spd->nr_pages_max = max_usage;
|
|
|
|
if (max_usage <= PIPE_DEF_BUFFERS)
|
2010-05-20 08:43:18 +00:00
|
|
|
return 0;
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
spd->pages = kmalloc_array(max_usage, sizeof(struct page *), GFP_KERNEL);
|
|
|
|
spd->partial = kmalloc_array(max_usage, sizeof(struct partial_page),
|
treewide: kmalloc() -> kmalloc_array()
The kmalloc() function has a 2-factor argument form, kmalloc_array(). This
patch replaces cases of:
kmalloc(a * b, gfp)
with:
kmalloc_array(a * b, gfp)
as well as handling cases of:
kmalloc(a * b * c, gfp)
with:
kmalloc(array3_size(a, b, c), gfp)
as it's slightly less ugly than:
kmalloc_array(array_size(a, b), c, gfp)
This does, however, attempt to ignore constant size factors like:
kmalloc(4 * 1024, gfp)
though any constants defined via macros get caught up in the conversion.
Any factors with a sizeof() of "unsigned char", "char", and "u8" were
dropped, since they're redundant.
The tools/ directory was manually excluded, since it has its own
implementation of kmalloc().
The Coccinelle script used for this was:
// Fix redundant parens around sizeof().
@@
type TYPE;
expression THING, E;
@@
(
kmalloc(
- (sizeof(TYPE)) * E
+ sizeof(TYPE) * E
, ...)
|
kmalloc(
- (sizeof(THING)) * E
+ sizeof(THING) * E
, ...)
)
// Drop single-byte sizes and redundant parens.
@@
expression COUNT;
typedef u8;
typedef __u8;
@@
(
kmalloc(
- sizeof(u8) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(__u8) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(char) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(unsigned char) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(u8) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(__u8) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(char) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(unsigned char) * COUNT
+ COUNT
, ...)
)
// 2-factor product with sizeof(type/expression) and identifier or constant.
@@
type TYPE;
expression THING;
identifier COUNT_ID;
constant COUNT_CONST;
@@
(
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (COUNT_ID)
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * COUNT_ID
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (COUNT_CONST)
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * COUNT_CONST
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (COUNT_ID)
+ COUNT_ID, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * COUNT_ID
+ COUNT_ID, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (COUNT_CONST)
+ COUNT_CONST, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * COUNT_CONST
+ COUNT_CONST, sizeof(THING)
, ...)
)
// 2-factor product, only identifiers.
@@
identifier SIZE, COUNT;
@@
- kmalloc
+ kmalloc_array
(
- SIZE * COUNT
+ COUNT, SIZE
, ...)
// 3-factor product with 1 sizeof(type) or sizeof(expression), with
// redundant parens removed.
@@
expression THING;
identifier STRIDE, COUNT;
type TYPE;
@@
(
kmalloc(
- sizeof(TYPE) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(THING) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
)
// 3-factor product with 2 sizeof(variable), with redundant parens removed.
@@
expression THING1, THING2;
identifier COUNT;
type TYPE1, TYPE2;
@@
(
kmalloc(
- sizeof(TYPE1) * sizeof(TYPE2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kmalloc(
- sizeof(THING1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(THING1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
)
// 3-factor product, only identifiers, with redundant parens removed.
@@
identifier STRIDE, SIZE, COUNT;
@@
(
kmalloc(
- (COUNT) * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
)
// Any remaining multi-factor products, first at least 3-factor products,
// when they're not all constants...
@@
expression E1, E2, E3;
constant C1, C2, C3;
@@
(
kmalloc(C1 * C2 * C3, ...)
|
kmalloc(
- (E1) * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- (E1) * (E2) * E3
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- (E1) * (E2) * (E3)
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- E1 * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
)
// And then all remaining 2 factors products when they're not all constants,
// keeping sizeof() as the second factor argument.
@@
expression THING, E1, E2;
type TYPE;
constant C1, C2, C3;
@@
(
kmalloc(sizeof(THING) * C2, ...)
|
kmalloc(sizeof(TYPE) * C2, ...)
|
kmalloc(C1 * C2 * C3, ...)
|
kmalloc(C1 * C2, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (E2)
+ E2, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * E2
+ E2, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (E2)
+ E2, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * E2
+ E2, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- (E1) * E2
+ E1, E2
, ...)
|
- kmalloc
+ kmalloc_array
(
- (E1) * (E2)
+ E1, E2
, ...)
|
- kmalloc
+ kmalloc_array
(
- E1 * E2
+ E1, E2
, ...)
)
Signed-off-by: Kees Cook <keescook@chromium.org>
2018-06-12 20:55:00 +00:00
|
|
|
GFP_KERNEL);
|
2010-05-20 08:43:18 +00:00
|
|
|
|
|
|
|
if (spd->pages && spd->partial)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
kfree(spd->pages);
|
|
|
|
kfree(spd->partial);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2012-06-12 13:24:40 +00:00
|
|
|
void splice_shrink_spd(struct splice_pipe_desc *spd)
|
2010-05-20 08:43:18 +00:00
|
|
|
{
|
2012-06-12 13:24:40 +00:00
|
|
|
if (spd->nr_pages_max <= PIPE_DEF_BUFFERS)
|
2010-05-20 08:43:18 +00:00
|
|
|
return;
|
|
|
|
|
|
|
|
kfree(spd->pages);
|
|
|
|
kfree(spd->partial);
|
|
|
|
}
|
|
|
|
|
2023-05-22 13:50:18 +00:00
|
|
|
/**
|
|
|
|
* copy_splice_read - Copy data from a file and splice the copy into a pipe
|
|
|
|
* @in: The file to read from
|
|
|
|
* @ppos: Pointer to the file position to read from
|
|
|
|
* @pipe: The pipe to splice into
|
|
|
|
* @len: The amount to splice
|
|
|
|
* @flags: The SPLICE_F_* flags
|
|
|
|
*
|
|
|
|
* This function allocates a bunch of pages sufficient to hold the requested
|
|
|
|
* amount of data (but limited by the remaining pipe capacity), passes it to
|
|
|
|
* the file's ->read_iter() to read into and then splices the used pages into
|
|
|
|
* the pipe.
|
|
|
|
*
|
|
|
|
* Return: On success, the number of bytes read will be returned and *@ppos
|
|
|
|
* will be updated if appropriate; 0 will be returned if there is no more data
|
|
|
|
* to be read; -EAGAIN will be returned if the pipe had no space, and some
|
|
|
|
* other negative error code will be returned on error. A short read may occur
|
|
|
|
* if the pipe has insufficient space, we reach the end of the data or we hit a
|
|
|
|
* hole.
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
*/
|
2023-05-22 13:49:50 +00:00
|
|
|
ssize_t copy_splice_read(struct file *in, loff_t *ppos,
|
|
|
|
struct pipe_inode_info *pipe,
|
|
|
|
size_t len, unsigned int flags)
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
{
|
|
|
|
struct iov_iter to;
|
|
|
|
struct bio_vec *bv;
|
|
|
|
struct kiocb kiocb;
|
|
|
|
struct page **pages;
|
|
|
|
ssize_t ret;
|
2023-05-22 13:49:51 +00:00
|
|
|
size_t used, npages, chunk, remain, keep = 0;
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Work out how much data we can actually add into the pipe */
|
|
|
|
used = pipe_occupancy(pipe->head, pipe->tail);
|
|
|
|
npages = max_t(ssize_t, pipe->max_usage - used, 0);
|
|
|
|
len = min_t(size_t, len, npages * PAGE_SIZE);
|
|
|
|
npages = DIV_ROUND_UP(len, PAGE_SIZE);
|
|
|
|
|
|
|
|
bv = kzalloc(array_size(npages, sizeof(bv[0])) +
|
|
|
|
array_size(npages, sizeof(struct page *)), GFP_KERNEL);
|
|
|
|
if (!bv)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2023-05-22 13:49:51 +00:00
|
|
|
pages = (struct page **)(bv + npages);
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
npages = alloc_pages_bulk_array(GFP_USER, npages, pages);
|
|
|
|
if (!npages) {
|
|
|
|
kfree(bv);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
|
|
|
remain = len = min_t(size_t, len, npages * PAGE_SIZE);
|
|
|
|
|
|
|
|
for (i = 0; i < npages; i++) {
|
|
|
|
chunk = min_t(size_t, PAGE_SIZE, remain);
|
|
|
|
bv[i].bv_page = pages[i];
|
|
|
|
bv[i].bv_offset = 0;
|
|
|
|
bv[i].bv_len = chunk;
|
|
|
|
remain -= chunk;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Do the I/O */
|
|
|
|
iov_iter_bvec(&to, ITER_DEST, bv, npages, len);
|
|
|
|
init_sync_kiocb(&kiocb, in);
|
|
|
|
kiocb.ki_pos = *ppos;
|
|
|
|
ret = call_read_iter(in, &kiocb, &to);
|
|
|
|
|
|
|
|
if (ret > 0) {
|
2023-05-22 13:49:51 +00:00
|
|
|
keep = DIV_ROUND_UP(ret, PAGE_SIZE);
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
*ppos = kiocb.ki_pos;
|
|
|
|
}
|
|
|
|
|
2023-06-14 14:03:39 +00:00
|
|
|
/*
|
|
|
|
* Callers of ->splice_read() expect -EAGAIN on "can't put anything in
|
|
|
|
* there", rather than -EFAULT.
|
|
|
|
*/
|
|
|
|
if (ret == -EFAULT)
|
|
|
|
ret = -EAGAIN;
|
|
|
|
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
/* Free any pages that didn't get touched at all. */
|
2023-05-22 13:49:51 +00:00
|
|
|
if (keep < npages)
|
|
|
|
release_pages(pages + keep, npages - keep);
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
|
|
|
|
/* Push the remaining pages into the pipe. */
|
2023-05-22 13:49:51 +00:00
|
|
|
remain = ret;
|
|
|
|
for (i = 0; i < keep; i++) {
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
struct pipe_buffer *buf = pipe_head_buf(pipe);
|
|
|
|
|
|
|
|
chunk = min_t(size_t, remain, PAGE_SIZE);
|
|
|
|
*buf = (struct pipe_buffer) {
|
|
|
|
.ops = &default_pipe_buf_ops,
|
|
|
|
.page = bv[i].bv_page,
|
|
|
|
.offset = 0,
|
|
|
|
.len = chunk,
|
|
|
|
};
|
|
|
|
pipe->head++;
|
|
|
|
remain -= chunk;
|
|
|
|
}
|
|
|
|
|
|
|
|
kfree(bv);
|
|
|
|
return ret;
|
|
|
|
}
|
2023-05-22 13:49:50 +00:00
|
|
|
EXPORT_SYMBOL(copy_splice_read);
|
splice: Add a func to do a splice from an O_DIRECT file without ITER_PIPE
Implement a function, direct_file_splice(), that deals with this by using
an ITER_BVEC iterator instead of an ITER_PIPE iterator as the former won't
free its buffers when reverted. The function bulk allocates all the
buffers it thinks it is going to use in advance, does the read
synchronously and only then trims the buffer down. The pages we did use
get pushed into the pipe.
This fixes a problem with the upcoming iov_iter_extract_pages() function,
whereby pages extracted from a non-user-backed iterator such as ITER_PIPE
aren't pinned. __iomap_dio_rw(), however, calls iov_iter_revert() to
shorten the iterator to just the bufferage it is going to use - which has
the side-effect of freeing the excess pipe buffers, even though they're
attached to a bio and may get written to by DMA (thanks to Hillf Danton for
spotting this[1]).
This then causes memory corruption that is particularly noticeable when the
syzbot test[2] is run. The test boils down to:
out = creat(argv[1], 0666);
ftruncate(out, 0x800);
lseek(out, 0x200, SEEK_SET);
in = open(argv[1], O_RDONLY | O_DIRECT | O_NOFOLLOW);
sendfile(out, in, NULL, 0x1dd00);
run repeatedly in parallel. What I think is happening is that ftruncate()
occasionally shortens the DIO read that's about to be made by sendfile's
splice core by reducing i_size.
This should be more efficient for DIO read by virtue of doing a bulk page
allocation, but slightly less efficient by ignoring any partial page in the
pipe.
Reported-by: syzbot+a440341a59e3b7142895@syzkaller.appspotmail.com
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: David Hildenbrand <david@redhat.com>
cc: John Hubbard <jhubbard@nvidia.com>
cc: linux-mm@kvack.org
cc: linux-block@vger.kernel.org
cc: linux-fsdevel@vger.kernel.org
Link: https://lore.kernel.org/r/20230207094731.1390-1-hdanton@sina.com/ [1]
Link: https://lore.kernel.org/r/000000000000b0b3c005f3a09383@google.com/ [2]
Signed-off-by: Steve French <stfrench@microsoft.com>
2023-02-07 10:45:40 +00:00
|
|
|
|
2016-09-22 20:33:12 +00:00
|
|
|
const struct pipe_buf_operations default_pipe_buf_ops = {
|
2020-05-20 15:58:16 +00:00
|
|
|
.release = generic_pipe_buf_release,
|
|
|
|
.try_steal = generic_pipe_buf_try_steal,
|
|
|
|
.get = generic_pipe_buf_get,
|
2009-05-07 13:37:36 +00:00
|
|
|
};
|
|
|
|
|
2014-01-22 18:36:57 +00:00
|
|
|
/* Pipe buffer operations for a socket and similar. */
|
|
|
|
const struct pipe_buf_operations nosteal_pipe_buf_ops = {
|
2020-05-20 15:58:16 +00:00
|
|
|
.release = generic_pipe_buf_release,
|
|
|
|
.get = generic_pipe_buf_get,
|
2014-01-22 18:36:57 +00:00
|
|
|
};
|
|
|
|
EXPORT_SYMBOL(nosteal_pipe_buf_ops);
|
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
static void wakeup_pipe_writers(struct pipe_inode_info *pipe)
|
|
|
|
{
|
|
|
|
smp_mb();
|
pipe: use exclusive waits when reading or writing
This makes the pipe code use separate wait-queues and exclusive waiting
for readers and writers, avoiding a nasty thundering herd problem when
there are lots of readers waiting for data on a pipe (or, less commonly,
lots of writers waiting for a pipe to have space).
While this isn't a common occurrence in the traditional "use a pipe as a
data transport" case, where you typically only have a single reader and
a single writer process, there is one common special case: using a pipe
as a source of "locking tokens" rather than for data communication.
In particular, the GNU make jobserver code ends up using a pipe as a way
to limit parallelism, where each job consumes a token by reading a byte
from the jobserver pipe, and releases the token by writing a byte back
to the pipe.
This pattern is fairly traditional on Unix, and works very well, but
will waste a lot of time waking up a lot of processes when only a single
reader needs to be woken up when a writer releases a new token.
A simplified test-case of just this pipe interaction is to create 64
processes, and then pass a single token around between them (this
test-case also intentionally passes another token that gets ignored to
test the "wake up next" logic too, in case anybody wonders about it):
#include <unistd.h>
int main(int argc, char **argv)
{
int fd[2], counters[2];
pipe(fd);
counters[0] = 0;
counters[1] = -1;
write(fd[1], counters, sizeof(counters));
/* 64 processes */
fork(); fork(); fork(); fork(); fork(); fork();
do {
int i;
read(fd[0], &i, sizeof(i));
if (i < 0)
continue;
counters[0] = i+1;
write(fd[1], counters, (1+(i & 1)) *sizeof(int));
} while (counters[0] < 1000000);
return 0;
}
and in a perfect world, passing that token around should only cause one
context switch per transfer, when the writer of a token causes a
directed wakeup of just a single reader.
But with the "writer wakes all readers" model we traditionally had, on
my test box the above case causes more than an order of magnitude more
scheduling: instead of the expected ~1M context switches, "perf stat"
shows
231,852.37 msec task-clock # 15.857 CPUs utilized
11,250,961 context-switches # 0.049 M/sec
616,304 cpu-migrations # 0.003 M/sec
1,648 page-faults # 0.007 K/sec
1,097,903,998,514 cycles # 4.735 GHz
120,781,778,352 instructions # 0.11 insn per cycle
27,997,056,043 branches # 120.754 M/sec
283,581,233 branch-misses # 1.01% of all branches
14.621273891 seconds time elapsed
0.018243000 seconds user
3.611468000 seconds sys
before this commit.
After this commit, I get
5,229.55 msec task-clock # 3.072 CPUs utilized
1,212,233 context-switches # 0.232 M/sec
103,951 cpu-migrations # 0.020 M/sec
1,328 page-faults # 0.254 K/sec
21,307,456,166 cycles # 4.074 GHz
12,947,819,999 instructions # 0.61 insn per cycle
2,881,985,678 branches # 551.096 M/sec
64,267,015 branch-misses # 2.23% of all branches
1.702148350 seconds time elapsed
0.004868000 seconds user
0.110786000 seconds sys
instead. Much better.
[ Note! This kernel improvement seems to be very good at triggering a
race condition in the make jobserver (in GNU make 4.2.1) for me. It's
a long known bug that was fixed back in June 2017 by GNU make commit
b552b0525198 ("[SV 51159] Use a non-blocking read with pselect to
avoid hangs.").
But there wasn't a new release of GNU make until 4.3 on Jan 19 2020,
so a number of distributions may still have the buggy version. Some
have backported the fix to their 4.2.1 release, though, and even
without the fix it's quite timing-dependent whether the bug actually
is hit. ]
Josh Triplett says:
"I've been hammering on your pipe fix patch (switching to exclusive
wait queues) for a month or so, on several different systems, and I've
run into no issues with it. The patch *substantially* improves
parallel build times on large (~100 CPU) systems, both with parallel
make and with other things that use make's pipe-based jobserver.
All current distributions (including stable and long-term stable
distributions) have versions of GNU make that no longer have the
jobserver bug"
Tested-by: Josh Triplett <josh@joshtriplett.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-12-09 17:48:27 +00:00
|
|
|
if (waitqueue_active(&pipe->wr_wait))
|
|
|
|
wake_up_interruptible(&pipe->wr_wait);
|
2009-04-14 17:48:36 +00:00
|
|
|
kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
|
|
|
|
}
|
|
|
|
|
2007-06-21 11:10:21 +00:00
|
|
|
/**
|
2009-04-14 17:48:36 +00:00
|
|
|
* splice_from_pipe_feed - feed available data from a pipe to a file
|
2007-06-21 11:10:21 +00:00
|
|
|
* @pipe: pipe to splice from
|
|
|
|
* @sd: information to @actor
|
|
|
|
* @actor: handler that splices the data
|
|
|
|
*
|
|
|
|
* Description:
|
2009-04-14 17:48:36 +00:00
|
|
|
* This function loops over the pipe and calls @actor to do the
|
|
|
|
* actual moving of a single struct pipe_buffer to the desired
|
|
|
|
* destination. It returns when there's no more buffers left in
|
|
|
|
* the pipe or if the requested number of bytes (@sd->total_len)
|
|
|
|
* have been copied. It returns a positive number (one) if the
|
|
|
|
* pipe needs to be filled with more data, zero if the required
|
|
|
|
* number of bytes have been copied and -errno on error.
|
2007-06-21 11:10:21 +00:00
|
|
|
*
|
2009-04-14 17:48:36 +00:00
|
|
|
* This, together with splice_from_pipe_{begin,end,next}, may be
|
|
|
|
* used to implement the functionality of __splice_from_pipe() when
|
|
|
|
* locking is required around copying the pipe buffers to the
|
|
|
|
* destination.
|
2006-04-02 21:05:09 +00:00
|
|
|
*/
|
2014-04-05 08:35:49 +00:00
|
|
|
static int splice_from_pipe_feed(struct pipe_inode_info *pipe, struct splice_desc *sd,
|
2009-04-14 17:48:36 +00:00
|
|
|
splice_actor *actor)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2019-11-15 13:30:32 +00:00
|
|
|
unsigned int head = pipe->head;
|
|
|
|
unsigned int tail = pipe->tail;
|
|
|
|
unsigned int mask = pipe->ring_size - 1;
|
2009-04-14 17:48:36 +00:00
|
|
|
int ret;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2019-12-06 20:40:35 +00:00
|
|
|
while (!pipe_empty(head, tail)) {
|
2019-11-15 13:30:32 +00:00
|
|
|
struct pipe_buffer *buf = &pipe->bufs[tail & mask];
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
sd->len = buf->len;
|
|
|
|
if (sd->len > sd->total_len)
|
|
|
|
sd->len = sd->total_len;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2016-09-27 08:45:12 +00:00
|
|
|
ret = pipe_buf_confirm(pipe, buf);
|
2010-12-17 07:56:44 +00:00
|
|
|
if (unlikely(ret)) {
|
2009-04-14 17:48:36 +00:00
|
|
|
if (ret == -ENODATA)
|
|
|
|
ret = 0;
|
|
|
|
return ret;
|
|
|
|
}
|
2010-12-17 07:56:44 +00:00
|
|
|
|
|
|
|
ret = actor(pipe, buf, sd);
|
|
|
|
if (ret <= 0)
|
|
|
|
return ret;
|
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
buf->offset += ret;
|
|
|
|
buf->len -= ret;
|
|
|
|
|
|
|
|
sd->num_spliced += ret;
|
|
|
|
sd->len -= ret;
|
|
|
|
sd->pos += ret;
|
|
|
|
sd->total_len -= ret;
|
|
|
|
|
|
|
|
if (!buf->len) {
|
2016-09-27 08:45:12 +00:00
|
|
|
pipe_buf_release(pipe, buf);
|
2019-11-15 13:30:32 +00:00
|
|
|
tail++;
|
|
|
|
pipe->tail = tail;
|
2013-03-21 15:01:38 +00:00
|
|
|
if (pipe->files)
|
2009-04-14 17:48:36 +00:00
|
|
|
sd->need_wakeup = true;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
if (!sd->total_len)
|
|
|
|
return 0;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
return 1;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
splice: teach splice pipe reading about empty pipe buffers
Tetsuo Handa reports that splice() can return 0 before the real EOF, if
the data in the splice source pipe is an empty pipe buffer. That empty
pipe buffer case doesn't happen in any normal situation, but you can
trigger it by doing a write to a pipe that fails due to a page fault.
Tetsuo has a test-case to show the behavior:
#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
const int fd = open("/tmp/testfile", O_WRONLY | O_CREAT, 0600);
int pipe_fd[2] = { -1, -1 };
pipe(pipe_fd);
write(pipe_fd[1], NULL, 4096);
/* This splice() should wait unless interrupted. */
return !splice(pipe_fd[0], NULL, fd, NULL, 65536, 0);
}
which results in
write(5, NULL, 4096) = -1 EFAULT (Bad address)
splice(4, NULL, 3, NULL, 65536, 0) = 0
and this can confuse splice() users into believing they have hit EOF
prematurely.
The issue was introduced when the pipe write code started pre-allocating
the pipe buffers before copying data from user space.
This is modified verion of Tetsuo's original patch.
Fixes: a194dfe6e6f6 ("pipe: Rearrange sequence in pipe_write() to preallocate slot")
Link:https://lore.kernel.org/linux-fsdevel/20201005121339.4063-1-penguin-kernel@I-love.SAKURA.ne.jp/
Reported-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Acked-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-05 18:26:27 +00:00
|
|
|
/* We know we have a pipe buffer, but maybe it's empty? */
|
|
|
|
static inline bool eat_empty_buffer(struct pipe_inode_info *pipe)
|
|
|
|
{
|
|
|
|
unsigned int tail = pipe->tail;
|
|
|
|
unsigned int mask = pipe->ring_size - 1;
|
|
|
|
struct pipe_buffer *buf = &pipe->bufs[tail & mask];
|
|
|
|
|
|
|
|
if (unlikely(!buf->len)) {
|
|
|
|
pipe_buf_release(pipe, buf);
|
|
|
|
pipe->tail = tail+1;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
/**
|
|
|
|
* splice_from_pipe_next - wait for some data to splice from
|
|
|
|
* @pipe: pipe to splice from
|
|
|
|
* @sd: information about the splice operation
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* This function will wait for some data and return a positive
|
|
|
|
* value (one) if pipe buffers are available. It will return zero
|
|
|
|
* or -errno if no more data needs to be spliced.
|
|
|
|
*/
|
2014-04-05 08:35:49 +00:00
|
|
|
static int splice_from_pipe_next(struct pipe_inode_info *pipe, struct splice_desc *sd)
|
2009-04-14 17:48:36 +00:00
|
|
|
{
|
2015-11-23 12:09:50 +00:00
|
|
|
/*
|
|
|
|
* Check for signal early to make process killable when there are
|
|
|
|
* always buffers available
|
|
|
|
*/
|
|
|
|
if (signal_pending(current))
|
|
|
|
return -ERESTARTSYS;
|
|
|
|
|
splice: teach splice pipe reading about empty pipe buffers
Tetsuo Handa reports that splice() can return 0 before the real EOF, if
the data in the splice source pipe is an empty pipe buffer. That empty
pipe buffer case doesn't happen in any normal situation, but you can
trigger it by doing a write to a pipe that fails due to a page fault.
Tetsuo has a test-case to show the behavior:
#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
const int fd = open("/tmp/testfile", O_WRONLY | O_CREAT, 0600);
int pipe_fd[2] = { -1, -1 };
pipe(pipe_fd);
write(pipe_fd[1], NULL, 4096);
/* This splice() should wait unless interrupted. */
return !splice(pipe_fd[0], NULL, fd, NULL, 65536, 0);
}
which results in
write(5, NULL, 4096) = -1 EFAULT (Bad address)
splice(4, NULL, 3, NULL, 65536, 0) = 0
and this can confuse splice() users into believing they have hit EOF
prematurely.
The issue was introduced when the pipe write code started pre-allocating
the pipe buffers before copying data from user space.
This is modified verion of Tetsuo's original patch.
Fixes: a194dfe6e6f6 ("pipe: Rearrange sequence in pipe_write() to preallocate slot")
Link:https://lore.kernel.org/linux-fsdevel/20201005121339.4063-1-penguin-kernel@I-love.SAKURA.ne.jp/
Reported-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Acked-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-05 18:26:27 +00:00
|
|
|
repeat:
|
2019-11-15 13:30:32 +00:00
|
|
|
while (pipe_empty(pipe->head, pipe->tail)) {
|
2009-04-14 17:48:36 +00:00
|
|
|
if (!pipe->writers)
|
|
|
|
return 0;
|
2006-04-25 13:42:00 +00:00
|
|
|
|
pipe: remove 'waiting_writers' merging logic
This code is ancient, and goes back to when we only had a single page
for the pipe buffers. The exact history is hidden in the mists of time
(ie "before git", and in fact predates the BK repository too).
At that long-ago point in time, it actually helped to try to merge big
back-and-forth pipe reads and writes, and not limit pipe reads to the
single pipe buffer in length just because that was all we had at a time.
However, since then we've expanded the pipe buffers to multiple pages,
and this logic really doesn't seem to make sense. And a lot of it is
somewhat questionable (ie "hmm, the user asked for a non-blocking read,
but we see that there's a writer pending, so let's wait anyway to get
the extra data that the writer will have").
But more importantly, it makes the "go to sleep" logic much less
obvious, and considering the wakeup issues we've had, I want to make for
less of those kinds of things.
Cc: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-12-07 21:21:01 +00:00
|
|
|
if (sd->num_spliced)
|
2009-04-14 17:48:36 +00:00
|
|
|
return 0;
|
2006-04-11 11:57:21 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
if (sd->flags & SPLICE_F_NONBLOCK)
|
|
|
|
return -EAGAIN;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
if (signal_pending(current))
|
|
|
|
return -ERESTARTSYS;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
if (sd->need_wakeup) {
|
|
|
|
wakeup_pipe_writers(pipe);
|
|
|
|
sd->need_wakeup = false;
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
|
|
|
|
pipe: remove pipe_wait() and fix wakeup race with splice
The pipe splice code still used the old model of waiting for pipe IO by
using a non-specific "pipe_wait()" that waited for any pipe event to
happen, which depended on all pipe IO being entirely serialized by the
pipe lock. So by checking the state you were waiting for, and then
adding yourself to the wait queue before dropping the lock, you were
guaranteed to see all the wakeups.
Strictly speaking, the actual wakeups were not done under the lock, but
the pipe_wait() model still worked, because since the waiter held the
lock when checking whether it should sleep, it would always see the
current state, and the wakeup was always done after updating the state.
However, commit 0ddad21d3e99 ("pipe: use exclusive waits when reading or
writing") split the single wait-queue into two, and in the process also
made the "wait for event" code wait for _two_ wait queues, and that then
showed a race with the wakers that were not serialized by the pipe lock.
It's only splice that used that "pipe_wait()" model, so the problem
wasn't obvious, but Josef Bacik reports:
"I hit a hang with fstest btrfs/187, which does a btrfs send into
/dev/null. This works by creating a pipe, the write side is given to
the kernel to write into, and the read side is handed to a thread that
splices into a file, in this case /dev/null.
The box that was hung had the write side stuck here [pipe_write] and
the read side stuck here [splice_from_pipe_next -> pipe_wait].
[ more details about pipe_wait() scenario ]
The problem is we're doing the prepare_to_wait, which sets our state
each time, however we can be woken up either with reads or writes. In
the case above we race with the WRITER waking us up, and re-set our
state to INTERRUPTIBLE, and thus never break out of schedule"
Josef had a patch that avoided the issue in pipe_wait() by just making
it set the state only once, but the deeper problem is that pipe_wait()
depends on a level of synchonization by the pipe mutex that it really
shouldn't. And the whole "wait for any pipe state change" model really
isn't very good to begin with.
So rather than trying to work around things in pipe_wait(), remove that
legacy model of "wait for arbitrary pipe event" entirely, and actually
create functions that wait for the pipe actually being readable or
writable, and can do so without depending on the pipe lock serializing
everything.
Fixes: 0ddad21d3e99 ("pipe: use exclusive waits when reading or writing")
Link: https://lore.kernel.org/linux-fsdevel/bfa88b5ad6f069b2b679316b9e495a970130416c.1601567868.git.josef@toxicpanda.com/
Reported-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-and-tested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-02 02:14:36 +00:00
|
|
|
pipe_wait_readable(pipe);
|
2009-04-14 17:48:36 +00:00
|
|
|
}
|
2006-04-02 19:46:35 +00:00
|
|
|
|
splice: teach splice pipe reading about empty pipe buffers
Tetsuo Handa reports that splice() can return 0 before the real EOF, if
the data in the splice source pipe is an empty pipe buffer. That empty
pipe buffer case doesn't happen in any normal situation, but you can
trigger it by doing a write to a pipe that fails due to a page fault.
Tetsuo has a test-case to show the behavior:
#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
const int fd = open("/tmp/testfile", O_WRONLY | O_CREAT, 0600);
int pipe_fd[2] = { -1, -1 };
pipe(pipe_fd);
write(pipe_fd[1], NULL, 4096);
/* This splice() should wait unless interrupted. */
return !splice(pipe_fd[0], NULL, fd, NULL, 65536, 0);
}
which results in
write(5, NULL, 4096) = -1 EFAULT (Bad address)
splice(4, NULL, 3, NULL, 65536, 0) = 0
and this can confuse splice() users into believing they have hit EOF
prematurely.
The issue was introduced when the pipe write code started pre-allocating
the pipe buffers before copying data from user space.
This is modified verion of Tetsuo's original patch.
Fixes: a194dfe6e6f6 ("pipe: Rearrange sequence in pipe_write() to preallocate slot")
Link:https://lore.kernel.org/linux-fsdevel/20201005121339.4063-1-penguin-kernel@I-love.SAKURA.ne.jp/
Reported-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Acked-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-05 18:26:27 +00:00
|
|
|
if (eat_empty_buffer(pipe))
|
|
|
|
goto repeat;
|
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
return 1;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
/**
|
|
|
|
* splice_from_pipe_begin - start splicing from pipe
|
2009-04-17 02:09:55 +00:00
|
|
|
* @sd: information about the splice operation
|
2009-04-14 17:48:36 +00:00
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* This function should be called before a loop containing
|
|
|
|
* splice_from_pipe_next() and splice_from_pipe_feed() to
|
|
|
|
* initialize the necessary fields of @sd.
|
|
|
|
*/
|
2014-04-05 08:35:49 +00:00
|
|
|
static void splice_from_pipe_begin(struct splice_desc *sd)
|
2009-04-14 17:48:36 +00:00
|
|
|
{
|
|
|
|
sd->num_spliced = 0;
|
|
|
|
sd->need_wakeup = false;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
/**
|
|
|
|
* splice_from_pipe_end - finish splicing from pipe
|
|
|
|
* @pipe: pipe to splice from
|
|
|
|
* @sd: information about the splice operation
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* This function will wake up pipe writers if necessary. It should
|
|
|
|
* be called after a loop containing splice_from_pipe_next() and
|
|
|
|
* splice_from_pipe_feed().
|
|
|
|
*/
|
2014-04-05 08:35:49 +00:00
|
|
|
static void splice_from_pipe_end(struct pipe_inode_info *pipe, struct splice_desc *sd)
|
2009-04-14 17:48:36 +00:00
|
|
|
{
|
|
|
|
if (sd->need_wakeup)
|
|
|
|
wakeup_pipe_writers(pipe);
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
/**
|
|
|
|
* __splice_from_pipe - splice data from a pipe to given actor
|
|
|
|
* @pipe: pipe to splice from
|
|
|
|
* @sd: information to @actor
|
|
|
|
* @actor: handler that splices the data
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* This function does little more than loop over the pipe and call
|
|
|
|
* @actor to do the actual moving of a single struct pipe_buffer to
|
2023-06-07 18:19:09 +00:00
|
|
|
* the desired destination. See pipe_to_file, pipe_to_sendmsg, or
|
2009-04-14 17:48:36 +00:00
|
|
|
* pipe_to_user.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
ssize_t __splice_from_pipe(struct pipe_inode_info *pipe, struct splice_desc *sd,
|
|
|
|
splice_actor *actor)
|
|
|
|
{
|
|
|
|
int ret;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2009-04-14 17:48:36 +00:00
|
|
|
splice_from_pipe_begin(sd);
|
|
|
|
do {
|
2015-11-23 12:09:51 +00:00
|
|
|
cond_resched();
|
2009-04-14 17:48:36 +00:00
|
|
|
ret = splice_from_pipe_next(pipe, sd);
|
|
|
|
if (ret > 0)
|
|
|
|
ret = splice_from_pipe_feed(pipe, sd, actor);
|
|
|
|
} while (ret > 0);
|
|
|
|
splice_from_pipe_end(pipe, sd);
|
|
|
|
|
|
|
|
return sd->num_spliced ? sd->num_spliced : ret;
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
2007-03-21 12:11:02 +00:00
|
|
|
EXPORT_SYMBOL(__splice_from_pipe);
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2007-06-21 11:10:21 +00:00
|
|
|
/**
|
|
|
|
* splice_from_pipe - splice data from a pipe to a file
|
|
|
|
* @pipe: pipe to splice from
|
|
|
|
* @out: file to splice to
|
|
|
|
* @ppos: position in @out
|
|
|
|
* @len: how many bytes to splice
|
|
|
|
* @flags: splice modifier flags
|
|
|
|
* @actor: handler that splices the data
|
|
|
|
*
|
|
|
|
* Description:
|
2009-04-14 17:48:37 +00:00
|
|
|
* See __splice_from_pipe. This function locks the pipe inode,
|
2007-06-21 11:10:21 +00:00
|
|
|
* otherwise it's identical to __splice_from_pipe().
|
|
|
|
*
|
|
|
|
*/
|
2006-10-17 16:43:07 +00:00
|
|
|
ssize_t splice_from_pipe(struct pipe_inode_info *pipe, struct file *out,
|
|
|
|
loff_t *ppos, size_t len, unsigned int flags,
|
|
|
|
splice_actor *actor)
|
|
|
|
{
|
|
|
|
ssize_t ret;
|
2007-06-12 19:17:17 +00:00
|
|
|
struct splice_desc sd = {
|
|
|
|
.total_len = len,
|
|
|
|
.flags = flags,
|
|
|
|
.pos = *ppos,
|
2007-06-14 11:08:55 +00:00
|
|
|
.u.file = out,
|
2007-06-12 19:17:17 +00:00
|
|
|
};
|
2006-10-17 16:43:07 +00:00
|
|
|
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_lock(pipe);
|
2007-06-12 19:17:17 +00:00
|
|
|
ret = __splice_from_pipe(pipe, &sd, actor);
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_unlock(pipe);
|
2006-10-17 16:43:07 +00:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-04-05 08:27:08 +00:00
|
|
|
/**
|
|
|
|
* iter_file_splice_write - splice data from a pipe to a file
|
|
|
|
* @pipe: pipe info
|
|
|
|
* @out: file to write to
|
|
|
|
* @ppos: position in @out
|
|
|
|
* @len: number of bytes to splice
|
|
|
|
* @flags: splice modifier flags
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* Will either move or copy pages (determined by @flags options) from
|
|
|
|
* the given pipe inode to the given file.
|
|
|
|
* This one is ->write_iter-based.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
ssize_t
|
|
|
|
iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out,
|
|
|
|
loff_t *ppos, size_t len, unsigned int flags)
|
|
|
|
{
|
|
|
|
struct splice_desc sd = {
|
|
|
|
.total_len = len,
|
|
|
|
.flags = flags,
|
|
|
|
.pos = *ppos,
|
|
|
|
.u.file = out,
|
|
|
|
};
|
2019-10-16 15:47:32 +00:00
|
|
|
int nbufs = pipe->max_usage;
|
2023-11-23 17:51:44 +00:00
|
|
|
struct bio_vec *array;
|
2014-04-05 08:27:08 +00:00
|
|
|
ssize_t ret;
|
|
|
|
|
2023-11-23 17:51:44 +00:00
|
|
|
if (!out->f_op->write_iter)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
array = kcalloc(nbufs, sizeof(struct bio_vec), GFP_KERNEL);
|
2014-04-05 08:27:08 +00:00
|
|
|
if (unlikely(!array))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
pipe_lock(pipe);
|
|
|
|
|
|
|
|
splice_from_pipe_begin(&sd);
|
|
|
|
while (sd.total_len) {
|
2023-11-23 17:51:44 +00:00
|
|
|
struct kiocb kiocb;
|
2014-04-05 08:27:08 +00:00
|
|
|
struct iov_iter from;
|
2019-12-06 20:40:35 +00:00
|
|
|
unsigned int head, tail, mask;
|
2014-04-05 08:27:08 +00:00
|
|
|
size_t left;
|
2019-11-15 13:30:32 +00:00
|
|
|
int n;
|
2014-04-05 08:27:08 +00:00
|
|
|
|
|
|
|
ret = splice_from_pipe_next(pipe, &sd);
|
|
|
|
if (ret <= 0)
|
|
|
|
break;
|
|
|
|
|
2019-10-16 15:47:32 +00:00
|
|
|
if (unlikely(nbufs < pipe->max_usage)) {
|
2014-04-05 08:27:08 +00:00
|
|
|
kfree(array);
|
2019-10-16 15:47:32 +00:00
|
|
|
nbufs = pipe->max_usage;
|
2014-04-05 08:27:08 +00:00
|
|
|
array = kcalloc(nbufs, sizeof(struct bio_vec),
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!array) {
|
|
|
|
ret = -ENOMEM;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-12-06 20:40:35 +00:00
|
|
|
head = pipe->head;
|
|
|
|
tail = pipe->tail;
|
|
|
|
mask = pipe->ring_size - 1;
|
|
|
|
|
2014-04-05 08:27:08 +00:00
|
|
|
/* build the vector */
|
|
|
|
left = sd.total_len;
|
2021-01-09 16:02:57 +00:00
|
|
|
for (n = 0; !pipe_empty(head, tail) && left && n < nbufs; tail++) {
|
2019-11-15 13:30:32 +00:00
|
|
|
struct pipe_buffer *buf = &pipe->bufs[tail & mask];
|
2014-04-05 08:27:08 +00:00
|
|
|
size_t this_len = buf->len;
|
|
|
|
|
2021-01-09 16:02:57 +00:00
|
|
|
/* zero-length bvecs are not supported, skip them */
|
|
|
|
if (!this_len)
|
|
|
|
continue;
|
|
|
|
this_len = min(this_len, left);
|
2014-04-05 08:27:08 +00:00
|
|
|
|
2016-09-27 08:45:12 +00:00
|
|
|
ret = pipe_buf_confirm(pipe, buf);
|
2014-04-05 08:27:08 +00:00
|
|
|
if (unlikely(ret)) {
|
|
|
|
if (ret == -ENODATA)
|
|
|
|
ret = 0;
|
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
|
2023-02-03 15:06:28 +00:00
|
|
|
bvec_set_page(&array[n], buf->page, this_len,
|
|
|
|
buf->offset);
|
2014-04-05 08:27:08 +00:00
|
|
|
left -= this_len;
|
2021-01-09 16:02:57 +00:00
|
|
|
n++;
|
2014-04-05 08:27:08 +00:00
|
|
|
}
|
|
|
|
|
2022-09-16 00:25:47 +00:00
|
|
|
iov_iter_bvec(&from, ITER_SOURCE, array, n, sd.total_len - left);
|
2023-11-23 17:51:44 +00:00
|
|
|
init_sync_kiocb(&kiocb, out);
|
|
|
|
kiocb.ki_pos = sd.pos;
|
|
|
|
ret = call_write_iter(out, &kiocb, &from);
|
|
|
|
sd.pos = kiocb.ki_pos;
|
2014-04-05 08:27:08 +00:00
|
|
|
if (ret <= 0)
|
|
|
|
break;
|
|
|
|
|
|
|
|
sd.num_spliced += ret;
|
|
|
|
sd.total_len -= ret;
|
2015-01-25 20:11:59 +00:00
|
|
|
*ppos = sd.pos;
|
2014-04-05 08:27:08 +00:00
|
|
|
|
|
|
|
/* dismiss the fully eaten buffers, adjust the partial one */
|
2019-11-15 13:30:32 +00:00
|
|
|
tail = pipe->tail;
|
2014-04-05 08:27:08 +00:00
|
|
|
while (ret) {
|
2019-11-15 13:30:32 +00:00
|
|
|
struct pipe_buffer *buf = &pipe->bufs[tail & mask];
|
2014-04-05 08:27:08 +00:00
|
|
|
if (ret >= buf->len) {
|
|
|
|
ret -= buf->len;
|
|
|
|
buf->len = 0;
|
2016-09-27 08:45:12 +00:00
|
|
|
pipe_buf_release(pipe, buf);
|
2019-11-15 13:30:32 +00:00
|
|
|
tail++;
|
|
|
|
pipe->tail = tail;
|
2014-04-05 08:27:08 +00:00
|
|
|
if (pipe->files)
|
|
|
|
sd.need_wakeup = true;
|
|
|
|
} else {
|
|
|
|
buf->offset += ret;
|
|
|
|
buf->len -= ret;
|
|
|
|
ret = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
done:
|
|
|
|
kfree(array);
|
|
|
|
splice_from_pipe_end(pipe, &sd);
|
|
|
|
|
|
|
|
pipe_unlock(pipe);
|
|
|
|
|
|
|
|
if (sd.num_spliced)
|
|
|
|
ret = sd.num_spliced;
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
EXPORT_SYMBOL(iter_file_splice_write);
|
|
|
|
|
2023-06-07 18:19:09 +00:00
|
|
|
#ifdef CONFIG_NET
|
2006-04-02 21:05:09 +00:00
|
|
|
/**
|
2023-06-07 18:19:09 +00:00
|
|
|
* splice_to_socket - splice data from a pipe to a socket
|
2007-06-21 11:10:21 +00:00
|
|
|
* @pipe: pipe to splice from
|
2006-04-02 21:05:09 +00:00
|
|
|
* @out: socket to write to
|
2007-06-21 11:10:21 +00:00
|
|
|
* @ppos: position in @out
|
2006-04-02 21:05:09 +00:00
|
|
|
* @len: number of bytes to splice
|
|
|
|
* @flags: splice modifier flags
|
|
|
|
*
|
2007-06-21 11:10:21 +00:00
|
|
|
* Description:
|
|
|
|
* Will send @len bytes from the pipe to a network socket. No data copying
|
|
|
|
* is involved.
|
2006-04-02 21:05:09 +00:00
|
|
|
*
|
|
|
|
*/
|
2023-06-07 18:19:09 +00:00
|
|
|
ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out,
|
|
|
|
loff_t *ppos, size_t len, unsigned int flags)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2023-06-07 18:19:09 +00:00
|
|
|
struct socket *sock = sock_from_file(out);
|
|
|
|
struct bio_vec bvec[16];
|
|
|
|
struct msghdr msg = {};
|
|
|
|
ssize_t ret = 0;
|
|
|
|
size_t spliced = 0;
|
|
|
|
bool need_wakeup = false;
|
|
|
|
|
|
|
|
pipe_lock(pipe);
|
|
|
|
|
|
|
|
while (len > 0) {
|
|
|
|
unsigned int head, tail, mask, bc = 0;
|
|
|
|
size_t remain = len;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check for signal early to make process killable when there
|
|
|
|
* are always buffers available
|
|
|
|
*/
|
|
|
|
ret = -ERESTARTSYS;
|
|
|
|
if (signal_pending(current))
|
|
|
|
break;
|
|
|
|
|
|
|
|
while (pipe_empty(pipe->head, pipe->tail)) {
|
|
|
|
ret = 0;
|
|
|
|
if (!pipe->writers)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (spliced)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
ret = -EAGAIN;
|
|
|
|
if (flags & SPLICE_F_NONBLOCK)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
ret = -ERESTARTSYS;
|
|
|
|
if (signal_pending(current))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (need_wakeup) {
|
|
|
|
wakeup_pipe_writers(pipe);
|
|
|
|
need_wakeup = false;
|
|
|
|
}
|
|
|
|
|
|
|
|
pipe_wait_readable(pipe);
|
|
|
|
}
|
|
|
|
|
|
|
|
head = pipe->head;
|
|
|
|
tail = pipe->tail;
|
|
|
|
mask = pipe->ring_size - 1;
|
|
|
|
|
|
|
|
while (!pipe_empty(head, tail)) {
|
|
|
|
struct pipe_buffer *buf = &pipe->bufs[tail & mask];
|
|
|
|
size_t seg;
|
|
|
|
|
|
|
|
if (!buf->len) {
|
|
|
|
tail++;
|
|
|
|
continue;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2023-06-07 18:19:09 +00:00
|
|
|
seg = min_t(size_t, remain, buf->len);
|
|
|
|
|
|
|
|
ret = pipe_buf_confirm(pipe, buf);
|
|
|
|
if (unlikely(ret)) {
|
|
|
|
if (ret == -ENODATA)
|
|
|
|
ret = 0;
|
|
|
|
break;
|
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2023-06-07 18:19:09 +00:00
|
|
|
bvec_set_page(&bvec[bc++], buf->page, seg, buf->offset);
|
|
|
|
remain -= seg;
|
2023-06-14 10:09:48 +00:00
|
|
|
if (remain == 0 || bc >= ARRAY_SIZE(bvec))
|
2023-06-07 18:19:09 +00:00
|
|
|
break;
|
2023-06-14 10:09:48 +00:00
|
|
|
tail++;
|
2023-06-07 18:19:09 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!bc)
|
|
|
|
break;
|
|
|
|
|
|
|
|
msg.msg_flags = MSG_SPLICE_PAGES;
|
|
|
|
if (flags & SPLICE_F_MORE)
|
|
|
|
msg.msg_flags |= MSG_MORE;
|
|
|
|
if (remain && pipe_occupancy(pipe->head, tail) > 0)
|
|
|
|
msg.msg_flags |= MSG_MORE;
|
2023-07-24 17:39:04 +00:00
|
|
|
if (out->f_flags & O_NONBLOCK)
|
|
|
|
msg.msg_flags |= MSG_DONTWAIT;
|
2023-06-07 18:19:09 +00:00
|
|
|
|
|
|
|
iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, bc,
|
|
|
|
len - remain);
|
|
|
|
ret = sock_sendmsg(sock, &msg);
|
|
|
|
if (ret <= 0)
|
|
|
|
break;
|
|
|
|
|
|
|
|
spliced += ret;
|
|
|
|
len -= ret;
|
|
|
|
tail = pipe->tail;
|
|
|
|
while (ret > 0) {
|
|
|
|
struct pipe_buffer *buf = &pipe->bufs[tail & mask];
|
|
|
|
size_t seg = min_t(size_t, ret, buf->len);
|
|
|
|
|
|
|
|
buf->offset += seg;
|
|
|
|
buf->len -= seg;
|
|
|
|
ret -= seg;
|
|
|
|
|
|
|
|
if (!buf->len) {
|
|
|
|
pipe_buf_release(pipe, buf);
|
|
|
|
tail++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tail != pipe->tail) {
|
|
|
|
pipe->tail = tail;
|
|
|
|
if (pipe->files)
|
|
|
|
need_wakeup = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
pipe_unlock(pipe);
|
|
|
|
if (need_wakeup)
|
|
|
|
wakeup_pipe_writers(pipe);
|
|
|
|
return spliced ?: ret;
|
|
|
|
}
|
|
|
|
#endif
|
2006-03-31 04:06:13 +00:00
|
|
|
|
2020-09-03 14:22:34 +00:00
|
|
|
static int warn_unsupported(struct file *file, const char *op)
|
|
|
|
{
|
|
|
|
pr_debug_ratelimited(
|
|
|
|
"splice %s not supported for file %pD4 (pid: %d comm: %.20s)\n",
|
|
|
|
op, file, current->pid, current->comm);
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2006-04-02 21:05:09 +00:00
|
|
|
/*
|
|
|
|
* Attempt to initiate a splice from pipe to file.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t do_splice_from(struct pipe_inode_info *pipe, struct file *out,
|
|
|
|
loff_t *ppos, size_t len, unsigned int flags)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2020-09-03 14:22:34 +00:00
|
|
|
if (unlikely(!out->f_op->splice_write))
|
|
|
|
return warn_unsupported(out, "write");
|
|
|
|
return out->f_op->splice_write(pipe, out, ppos, len, flags);
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
|
|
|
|
2023-06-07 18:19:10 +00:00
|
|
|
/*
|
|
|
|
* Indicate to the caller that there was a premature EOF when reading from the
|
|
|
|
* source and the caller didn't indicate they would be sending more data after
|
|
|
|
* this.
|
|
|
|
*/
|
|
|
|
static void do_splice_eof(struct splice_desc *sd)
|
|
|
|
{
|
|
|
|
if (sd->splice_eof)
|
|
|
|
sd->splice_eof(sd);
|
|
|
|
}
|
|
|
|
|
2023-11-22 12:27:02 +00:00
|
|
|
/*
|
|
|
|
* Callers already called rw_verify_area() on the entire range.
|
|
|
|
* No need to call it for sub ranges.
|
2006-04-02 21:05:09 +00:00
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t do_splice_read(struct file *in, loff_t *ppos,
|
|
|
|
struct pipe_inode_info *pipe, size_t len,
|
|
|
|
unsigned int flags)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2021-01-25 04:49:04 +00:00
|
|
|
unsigned int p_space;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2006-04-11 11:56:09 +00:00
|
|
|
if (unlikely(!(in->f_mode & FMODE_READ)))
|
2006-03-30 13:15:30 +00:00
|
|
|
return -EBADF;
|
2023-05-22 13:49:53 +00:00
|
|
|
if (!len)
|
|
|
|
return 0;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2021-01-25 04:49:04 +00:00
|
|
|
/* Don't try to read more the pipe has space for. */
|
|
|
|
p_space = pipe->max_usage - pipe_occupancy(pipe->head, pipe->tail);
|
|
|
|
len = min_t(size_t, len, p_space << PAGE_SHIFT);
|
|
|
|
|
2016-04-02 18:56:58 +00:00
|
|
|
if (unlikely(len > MAX_RW_COUNT))
|
|
|
|
len = MAX_RW_COUNT;
|
|
|
|
|
2020-09-03 14:22:34 +00:00
|
|
|
if (unlikely(!in->f_op->splice_read))
|
|
|
|
return warn_unsupported(in, "read");
|
2023-05-22 13:49:54 +00:00
|
|
|
/*
|
2023-05-22 13:49:55 +00:00
|
|
|
* O_DIRECT and DAX don't deal with the pagecache, so we allocate a
|
|
|
|
* buffer, copy into it and splice that into the pipe.
|
2023-05-22 13:49:54 +00:00
|
|
|
*/
|
2023-05-22 13:49:55 +00:00
|
|
|
if ((in->f_flags & O_DIRECT) || IS_DAX(in->f_mapping->host))
|
2023-05-22 13:49:54 +00:00
|
|
|
return copy_splice_read(in, ppos, pipe, len, flags);
|
2020-09-03 14:22:34 +00:00
|
|
|
return in->f_op->splice_read(in, ppos, pipe, len, flags);
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
2023-11-22 12:27:02 +00:00
|
|
|
|
|
|
|
/**
|
|
|
|
* vfs_splice_read - Read data from a file and splice it into a pipe
|
|
|
|
* @in: File to splice from
|
|
|
|
* @ppos: Input file offset
|
|
|
|
* @pipe: Pipe to splice to
|
|
|
|
* @len: Number of bytes to splice
|
|
|
|
* @flags: Splice modifier flags (SPLICE_F_*)
|
|
|
|
*
|
|
|
|
* Splice the requested amount of data from the input file to the pipe. This
|
|
|
|
* is synchronous as the caller must hold the pipe lock across the entire
|
|
|
|
* operation.
|
|
|
|
*
|
|
|
|
* If successful, it returns the amount of data spliced, 0 if it hit the EOF or
|
|
|
|
* a hole and a negative error code otherwise.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t vfs_splice_read(struct file *in, loff_t *ppos,
|
|
|
|
struct pipe_inode_info *pipe, size_t len,
|
|
|
|
unsigned int flags)
|
2023-11-22 12:27:02 +00:00
|
|
|
{
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret;
|
2023-11-22 12:27:02 +00:00
|
|
|
|
|
|
|
ret = rw_verify_area(READ, in, ppos, len);
|
|
|
|
if (unlikely(ret < 0))
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
return do_splice_read(in, ppos, pipe, len, flags);
|
|
|
|
}
|
2023-05-22 13:49:52 +00:00
|
|
|
EXPORT_SYMBOL_GPL(vfs_splice_read);
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2007-06-21 11:10:21 +00:00
|
|
|
/**
|
|
|
|
* splice_direct_to_actor - splices data directly between two non-pipes
|
|
|
|
* @in: file to splice from
|
|
|
|
* @sd: actor information on where to splice to
|
|
|
|
* @actor: handles the data splicing
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* This is a special case helper to splice directly between two
|
|
|
|
* points, without requiring an explicit pipe. Internally an allocated
|
2007-07-27 06:08:51 +00:00
|
|
|
* pipe is cached in the process, and reused during the lifetime of
|
2007-06-21 11:10:21 +00:00
|
|
|
* that process.
|
|
|
|
*
|
2007-06-12 19:17:17 +00:00
|
|
|
*/
|
|
|
|
ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
|
|
|
|
splice_direct_actor *actor)
|
2006-04-11 11:52:07 +00:00
|
|
|
{
|
|
|
|
struct pipe_inode_info *pipe;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret, bytes;
|
2007-06-12 19:17:17 +00:00
|
|
|
size_t len;
|
splice: sendfile() at once fails for big files
Using sendfile with below small program to get MD5 sums of some files,
it appear that big files (over 64kbytes with 4k pages system) get a
wrong MD5 sum while small files get the correct sum.
This program uses sendfile() to send a file to an AF_ALG socket
for hashing.
/* md5sum2.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <linux/if_alg.h>
int main(int argc, char **argv)
{
int sk = socket(AF_ALG, SOCK_SEQPACKET, 0);
struct stat st;
struct sockaddr_alg sa = {
.salg_family = AF_ALG,
.salg_type = "hash",
.salg_name = "md5",
};
int n;
bind(sk, (struct sockaddr*)&sa, sizeof(sa));
for (n = 1; n < argc; n++) {
int size;
int offset = 0;
char buf[4096];
int fd;
int sko;
int i;
fd = open(argv[n], O_RDONLY);
sko = accept(sk, NULL, 0);
fstat(fd, &st);
size = st.st_size;
sendfile(sko, fd, &offset, size);
size = read(sko, buf, sizeof(buf));
for (i = 0; i < size; i++)
printf("%2.2x", buf[i]);
printf(" %s\n", argv[n]);
close(fd);
close(sko);
}
exit(0);
}
Test below is done using official linux patch files. First result is
with a software based md5sum. Second result is with the program above.
root@vgoip:~# ls -l patch-3.6.*
-rw-r--r-- 1 root root 64011 Aug 24 12:01 patch-3.6.2.gz
-rw-r--r-- 1 root root 94131 Aug 24 12:01 patch-3.6.3.gz
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
5fd77b24e68bb24dcc72d6e57c64790e patch-3.6.3.gz
After investivation, it appears that sendfile() sends the files by blocks
of 64kbytes (16 times PAGE_SIZE). The problem is that at the end of each
block, the SPLICE_F_MORE flag is missing, therefore the hashing operation
is reset as if it was the end of the file.
This patch adds SPLICE_F_MORE to the flags when more data is pending.
With the patch applied, we get the correct sums:
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06 15:26:47 +00:00
|
|
|
int i, flags, more;
|
2006-04-11 11:52:07 +00:00
|
|
|
|
|
|
|
/*
|
2022-06-29 13:06:58 +00:00
|
|
|
* We require the input to be seekable, as we don't want to randomly
|
|
|
|
* drop data for eg socket -> socket splicing. Use the piped splicing
|
|
|
|
* for that!
|
2006-04-11 11:52:07 +00:00
|
|
|
*/
|
2022-06-29 13:06:58 +00:00
|
|
|
if (unlikely(!(in->f_mode & FMODE_LSEEK)))
|
2006-04-11 11:52:07 +00:00
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* neither in nor out is a pipe, setup an internal pipe attached to
|
|
|
|
* 'out' and transfer the wanted data from 'in' to 'out' through that
|
|
|
|
*/
|
|
|
|
pipe = current->splice_pipe;
|
2006-04-11 11:56:09 +00:00
|
|
|
if (unlikely(!pipe)) {
|
2013-03-21 15:04:15 +00:00
|
|
|
pipe = alloc_pipe_info();
|
2006-04-11 11:52:07 +00:00
|
|
|
if (!pipe)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We don't have an immediate reader, but we'll read the stuff
|
2006-04-26 12:39:29 +00:00
|
|
|
* out of the pipe right after the splice_to_pipe(). So set
|
2006-04-11 11:52:07 +00:00
|
|
|
* PIPE_READERS appropriately.
|
|
|
|
*/
|
|
|
|
pipe->readers = 1;
|
|
|
|
|
|
|
|
current->splice_pipe = pipe;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2006-04-11 11:57:21 +00:00
|
|
|
* Do the splice.
|
2006-04-11 11:52:07 +00:00
|
|
|
*/
|
|
|
|
bytes = 0;
|
2007-06-12 19:17:17 +00:00
|
|
|
len = sd->total_len;
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
|
|
|
|
/* Don't block on output, we have to drain the direct pipe. */
|
2007-06-12 19:17:17 +00:00
|
|
|
flags = sd->flags;
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
sd->flags &= ~SPLICE_F_NONBLOCK;
|
2007-06-12 19:17:17 +00:00
|
|
|
|
|
|
|
/*
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
* We signal MORE until we've read sufficient data to fulfill the
|
|
|
|
* request and we keep signalling it if the caller set it.
|
2007-06-12 19:17:17 +00:00
|
|
|
*/
|
splice: sendfile() at once fails for big files
Using sendfile with below small program to get MD5 sums of some files,
it appear that big files (over 64kbytes with 4k pages system) get a
wrong MD5 sum while small files get the correct sum.
This program uses sendfile() to send a file to an AF_ALG socket
for hashing.
/* md5sum2.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <linux/if_alg.h>
int main(int argc, char **argv)
{
int sk = socket(AF_ALG, SOCK_SEQPACKET, 0);
struct stat st;
struct sockaddr_alg sa = {
.salg_family = AF_ALG,
.salg_type = "hash",
.salg_name = "md5",
};
int n;
bind(sk, (struct sockaddr*)&sa, sizeof(sa));
for (n = 1; n < argc; n++) {
int size;
int offset = 0;
char buf[4096];
int fd;
int sko;
int i;
fd = open(argv[n], O_RDONLY);
sko = accept(sk, NULL, 0);
fstat(fd, &st);
size = st.st_size;
sendfile(sko, fd, &offset, size);
size = read(sko, buf, sizeof(buf));
for (i = 0; i < size; i++)
printf("%2.2x", buf[i]);
printf(" %s\n", argv[n]);
close(fd);
close(sko);
}
exit(0);
}
Test below is done using official linux patch files. First result is
with a software based md5sum. Second result is with the program above.
root@vgoip:~# ls -l patch-3.6.*
-rw-r--r-- 1 root root 64011 Aug 24 12:01 patch-3.6.2.gz
-rw-r--r-- 1 root root 94131 Aug 24 12:01 patch-3.6.3.gz
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
5fd77b24e68bb24dcc72d6e57c64790e patch-3.6.3.gz
After investivation, it appears that sendfile() sends the files by blocks
of 64kbytes (16 times PAGE_SIZE). The problem is that at the end of each
block, the SPLICE_F_MORE flag is missing, therefore the hashing operation
is reset as if it was the end of the file.
This patch adds SPLICE_F_MORE to the flags when more data is pending.
With the patch applied, we get the correct sums:
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06 15:26:47 +00:00
|
|
|
more = sd->flags & SPLICE_F_MORE;
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
sd->flags |= SPLICE_F_MORE;
|
2006-04-11 11:52:07 +00:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
WARN_ON_ONCE(!pipe_empty(pipe->head, pipe->tail));
|
2018-11-30 18:37:49 +00:00
|
|
|
|
2006-04-11 11:52:07 +00:00
|
|
|
while (len) {
|
2007-07-13 12:11:43 +00:00
|
|
|
size_t read_len;
|
2008-05-09 11:28:36 +00:00
|
|
|
loff_t pos = sd->pos, prev_pos = pos;
|
2006-04-11 11:52:07 +00:00
|
|
|
|
2023-11-22 12:27:02 +00:00
|
|
|
ret = do_splice_read(in, &pos, pipe, len, flags);
|
2007-07-13 12:11:43 +00:00
|
|
|
if (unlikely(ret <= 0))
|
2023-06-07 18:19:10 +00:00
|
|
|
goto read_failure;
|
2006-04-11 11:52:07 +00:00
|
|
|
|
|
|
|
read_len = ret;
|
2007-06-12 19:17:17 +00:00
|
|
|
sd->total_len = read_len;
|
2006-04-11 11:52:07 +00:00
|
|
|
|
splice: sendfile() at once fails for big files
Using sendfile with below small program to get MD5 sums of some files,
it appear that big files (over 64kbytes with 4k pages system) get a
wrong MD5 sum while small files get the correct sum.
This program uses sendfile() to send a file to an AF_ALG socket
for hashing.
/* md5sum2.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <linux/if_alg.h>
int main(int argc, char **argv)
{
int sk = socket(AF_ALG, SOCK_SEQPACKET, 0);
struct stat st;
struct sockaddr_alg sa = {
.salg_family = AF_ALG,
.salg_type = "hash",
.salg_name = "md5",
};
int n;
bind(sk, (struct sockaddr*)&sa, sizeof(sa));
for (n = 1; n < argc; n++) {
int size;
int offset = 0;
char buf[4096];
int fd;
int sko;
int i;
fd = open(argv[n], O_RDONLY);
sko = accept(sk, NULL, 0);
fstat(fd, &st);
size = st.st_size;
sendfile(sko, fd, &offset, size);
size = read(sko, buf, sizeof(buf));
for (i = 0; i < size; i++)
printf("%2.2x", buf[i]);
printf(" %s\n", argv[n]);
close(fd);
close(sko);
}
exit(0);
}
Test below is done using official linux patch files. First result is
with a software based md5sum. Second result is with the program above.
root@vgoip:~# ls -l patch-3.6.*
-rw-r--r-- 1 root root 64011 Aug 24 12:01 patch-3.6.2.gz
-rw-r--r-- 1 root root 94131 Aug 24 12:01 patch-3.6.3.gz
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
5fd77b24e68bb24dcc72d6e57c64790e patch-3.6.3.gz
After investivation, it appears that sendfile() sends the files by blocks
of 64kbytes (16 times PAGE_SIZE). The problem is that at the end of each
block, the SPLICE_F_MORE flag is missing, therefore the hashing operation
is reset as if it was the end of the file.
This patch adds SPLICE_F_MORE to the flags when more data is pending.
With the patch applied, we get the correct sums:
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06 15:26:47 +00:00
|
|
|
/*
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
* If we now have sufficient data to fulfill the request then
|
|
|
|
* we clear SPLICE_F_MORE if it was not set initially.
|
splice: sendfile() at once fails for big files
Using sendfile with below small program to get MD5 sums of some files,
it appear that big files (over 64kbytes with 4k pages system) get a
wrong MD5 sum while small files get the correct sum.
This program uses sendfile() to send a file to an AF_ALG socket
for hashing.
/* md5sum2.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <linux/if_alg.h>
int main(int argc, char **argv)
{
int sk = socket(AF_ALG, SOCK_SEQPACKET, 0);
struct stat st;
struct sockaddr_alg sa = {
.salg_family = AF_ALG,
.salg_type = "hash",
.salg_name = "md5",
};
int n;
bind(sk, (struct sockaddr*)&sa, sizeof(sa));
for (n = 1; n < argc; n++) {
int size;
int offset = 0;
char buf[4096];
int fd;
int sko;
int i;
fd = open(argv[n], O_RDONLY);
sko = accept(sk, NULL, 0);
fstat(fd, &st);
size = st.st_size;
sendfile(sko, fd, &offset, size);
size = read(sko, buf, sizeof(buf));
for (i = 0; i < size; i++)
printf("%2.2x", buf[i]);
printf(" %s\n", argv[n]);
close(fd);
close(sko);
}
exit(0);
}
Test below is done using official linux patch files. First result is
with a software based md5sum. Second result is with the program above.
root@vgoip:~# ls -l patch-3.6.*
-rw-r--r-- 1 root root 64011 Aug 24 12:01 patch-3.6.2.gz
-rw-r--r-- 1 root root 94131 Aug 24 12:01 patch-3.6.3.gz
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
5fd77b24e68bb24dcc72d6e57c64790e patch-3.6.3.gz
After investivation, it appears that sendfile() sends the files by blocks
of 64kbytes (16 times PAGE_SIZE). The problem is that at the end of each
block, the SPLICE_F_MORE flag is missing, therefore the hashing operation
is reset as if it was the end of the file.
This patch adds SPLICE_F_MORE to the flags when more data is pending.
With the patch applied, we get the correct sums:
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06 15:26:47 +00:00
|
|
|
*/
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
if (read_len >= len && !more)
|
splice: sendfile() at once fails for big files
Using sendfile with below small program to get MD5 sums of some files,
it appear that big files (over 64kbytes with 4k pages system) get a
wrong MD5 sum while small files get the correct sum.
This program uses sendfile() to send a file to an AF_ALG socket
for hashing.
/* md5sum2.c */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <linux/if_alg.h>
int main(int argc, char **argv)
{
int sk = socket(AF_ALG, SOCK_SEQPACKET, 0);
struct stat st;
struct sockaddr_alg sa = {
.salg_family = AF_ALG,
.salg_type = "hash",
.salg_name = "md5",
};
int n;
bind(sk, (struct sockaddr*)&sa, sizeof(sa));
for (n = 1; n < argc; n++) {
int size;
int offset = 0;
char buf[4096];
int fd;
int sko;
int i;
fd = open(argv[n], O_RDONLY);
sko = accept(sk, NULL, 0);
fstat(fd, &st);
size = st.st_size;
sendfile(sko, fd, &offset, size);
size = read(sko, buf, sizeof(buf));
for (i = 0; i < size; i++)
printf("%2.2x", buf[i]);
printf(" %s\n", argv[n]);
close(fd);
close(sko);
}
exit(0);
}
Test below is done using official linux patch files. First result is
with a software based md5sum. Second result is with the program above.
root@vgoip:~# ls -l patch-3.6.*
-rw-r--r-- 1 root root 64011 Aug 24 12:01 patch-3.6.2.gz
-rw-r--r-- 1 root root 94131 Aug 24 12:01 patch-3.6.3.gz
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
5fd77b24e68bb24dcc72d6e57c64790e patch-3.6.3.gz
After investivation, it appears that sendfile() sends the files by blocks
of 64kbytes (16 times PAGE_SIZE). The problem is that at the end of each
block, the SPLICE_F_MORE flag is missing, therefore the hashing operation
is reset as if it was the end of the file.
This patch adds SPLICE_F_MORE to the flags when more data is pending.
With the patch applied, we get the correct sums:
root@vgoip:~# md5sum patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
root@vgoip:~# ./md5sum2 patch-3.6.*
b3ffb9848196846f31b2ff133d2d6443 patch-3.6.2.gz
c5e8f687878457db77cb7158c38a7e43 patch-3.6.3.gz
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Jens Axboe <axboe@fb.com>
2015-05-06 15:26:47 +00:00
|
|
|
sd->flags &= ~SPLICE_F_MORE;
|
splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor()
splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and,
as a result, it incorrectly signals/fails to signal MSG_MORE when splicing
to a socket. The problem I'm seeing happens when a short splice occurs
because we got a short read due to hitting the EOF on a file: as the length
read (read_len) is less than the remaining size to be spliced (len),
SPLICE_F_MORE (and thus MSG_MORE) is set.
The issue is that, for the moment, we have no way to know *why* the short
read occurred and so can't make a good decision on whether we *should* keep
MSG_MORE set.
MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set
incorrectly under some circumstances - for example if a short read fills a
single pipe_buffer, but the next read would return more (seqfile can do
this).
This was observed with the multi_chunk_sendfile tests in the tls kselftest
program. Some of those tests would hang and time out when the last chunk
of file was less than the sendfile request size:
build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile
This has been observed before[2] and worked around in AF_TLS[3].
Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if
we haven't yet hit the requested operation size. SPLICE_F_MORE remains
signalled if the user passed it in to splice() but otherwise gets cleared
when we've read sufficient data to fulfill the request.
If, however, we get a premature EOF from ->splice_read(), have sent at
least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof()
will be invoked.
Signed-off-by: David Howells <dhowells@redhat.com>
cc: Linus Torvalds <torvalds@linux-foundation.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: Christoph Hellwig <hch@lst.de>
cc: Al Viro <viro@zeniv.linux.org.uk>
cc: Matthew Wilcox <willy@infradead.org>
cc: Jan Kara <jack@suse.cz>
cc: Jeff Layton <jlayton@kernel.org>
cc: David Hildenbrand <david@redhat.com>
cc: Christian Brauner <brauner@kernel.org>
cc: Chuck Lever <chuck.lever@oracle.com>
cc: Boris Pismenny <borisp@nvidia.com>
cc: John Fastabend <john.fastabend@gmail.com>
cc: linux-mm@kvack.org
Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1]
Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2]
Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3]
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-07 18:19:16 +00:00
|
|
|
|
2006-04-11 11:52:07 +00:00
|
|
|
/*
|
|
|
|
* NOTE: nonblocking mode only applies to the input. We
|
|
|
|
* must not do the output in nonblocking mode as then we
|
|
|
|
* could get stuck data in the internal pipe:
|
|
|
|
*/
|
2007-06-12 19:17:17 +00:00
|
|
|
ret = actor(pipe, sd);
|
2008-05-09 11:28:36 +00:00
|
|
|
if (unlikely(ret <= 0)) {
|
|
|
|
sd->pos = prev_pos;
|
2006-04-11 11:52:07 +00:00
|
|
|
goto out_release;
|
2008-05-09 11:28:36 +00:00
|
|
|
}
|
2006-04-11 11:52:07 +00:00
|
|
|
|
|
|
|
bytes += ret;
|
|
|
|
len -= ret;
|
2007-07-16 12:41:49 +00:00
|
|
|
sd->pos = pos;
|
2006-04-11 11:52:07 +00:00
|
|
|
|
2008-05-09 11:28:36 +00:00
|
|
|
if (ret < read_len) {
|
|
|
|
sd->pos = prev_pos + ret;
|
2007-07-13 12:11:43 +00:00
|
|
|
goto out_release;
|
2008-05-09 11:28:36 +00:00
|
|
|
}
|
2006-04-11 11:52:07 +00:00
|
|
|
}
|
|
|
|
|
2008-01-29 20:05:57 +00:00
|
|
|
done:
|
2019-11-15 13:30:32 +00:00
|
|
|
pipe->tail = pipe->head = 0;
|
2008-01-30 11:24:48 +00:00
|
|
|
file_accessed(in);
|
2006-04-11 11:52:07 +00:00
|
|
|
return bytes;
|
|
|
|
|
2023-06-07 18:19:10 +00:00
|
|
|
read_failure:
|
|
|
|
/*
|
|
|
|
* If the user did *not* set SPLICE_F_MORE *and* we didn't hit that
|
|
|
|
* "use all of len" case that cleared SPLICE_F_MORE, *and* we did a
|
|
|
|
* "->splice_in()" that returned EOF (ie zero) *and* we have sent at
|
|
|
|
* least 1 byte *then* we will also do the ->splice_eof() call.
|
|
|
|
*/
|
|
|
|
if (ret == 0 && !more && len > 0 && bytes)
|
|
|
|
do_splice_eof(sd);
|
2006-04-11 11:52:07 +00:00
|
|
|
out_release:
|
|
|
|
/*
|
|
|
|
* If we did an incomplete transfer we must release
|
|
|
|
* the pipe buffers in question:
|
|
|
|
*/
|
2019-11-15 13:30:32 +00:00
|
|
|
for (i = 0; i < pipe->ring_size; i++) {
|
|
|
|
struct pipe_buffer *buf = &pipe->bufs[i];
|
2006-04-11 11:52:07 +00:00
|
|
|
|
2016-09-27 08:45:12 +00:00
|
|
|
if (buf->ops)
|
|
|
|
pipe_buf_release(pipe, buf);
|
2006-04-11 11:52:07 +00:00
|
|
|
}
|
|
|
|
|
2008-01-29 20:05:57 +00:00
|
|
|
if (!bytes)
|
|
|
|
bytes = ret;
|
2007-06-12 19:17:17 +00:00
|
|
|
|
2008-01-29 20:05:57 +00:00
|
|
|
goto done;
|
2007-06-12 19:17:17 +00:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(splice_direct_to_actor);
|
|
|
|
|
|
|
|
static int direct_splice_actor(struct pipe_inode_info *pipe,
|
|
|
|
struct splice_desc *sd)
|
|
|
|
{
|
2007-06-14 11:08:55 +00:00
|
|
|
struct file *file = sd->u.file;
|
2023-11-30 14:16:23 +00:00
|
|
|
long ret;
|
|
|
|
|
|
|
|
file_start_write(file);
|
|
|
|
ret = do_splice_from(pipe, file, sd->opos, sd->total_len, sd->flags);
|
|
|
|
file_end_write(file);
|
|
|
|
return ret;
|
|
|
|
}
|
2007-06-12 19:17:17 +00:00
|
|
|
|
2023-11-30 14:16:23 +00:00
|
|
|
static int splice_file_range_actor(struct pipe_inode_info *pipe,
|
|
|
|
struct splice_desc *sd)
|
|
|
|
{
|
|
|
|
struct file *file = sd->u.file;
|
|
|
|
|
|
|
|
return do_splice_from(pipe, file, sd->opos, sd->total_len, sd->flags);
|
2007-06-12 19:17:17 +00:00
|
|
|
}
|
|
|
|
|
2023-06-07 18:19:10 +00:00
|
|
|
static void direct_file_splice_eof(struct splice_desc *sd)
|
|
|
|
{
|
|
|
|
struct file *file = sd->u.file;
|
|
|
|
|
|
|
|
if (file->f_op->splice_eof)
|
|
|
|
file->f_op->splice_eof(file);
|
|
|
|
}
|
|
|
|
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t do_splice_direct_actor(struct file *in, loff_t *ppos,
|
|
|
|
struct file *out, loff_t *opos,
|
|
|
|
size_t len, unsigned int flags,
|
|
|
|
splice_direct_actor *actor)
|
2007-06-12 19:17:17 +00:00
|
|
|
{
|
|
|
|
struct splice_desc sd = {
|
|
|
|
.len = len,
|
|
|
|
.total_len = len,
|
|
|
|
.flags = flags,
|
|
|
|
.pos = *ppos,
|
2007-06-14 11:08:55 +00:00
|
|
|
.u.file = out,
|
2023-06-07 18:19:10 +00:00
|
|
|
.splice_eof = direct_file_splice_eof,
|
2013-06-20 14:58:36 +00:00
|
|
|
.opos = opos,
|
2007-06-12 19:17:17 +00:00
|
|
|
};
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret;
|
2007-06-12 19:17:17 +00:00
|
|
|
|
2013-06-19 11:41:54 +00:00
|
|
|
if (unlikely(!(out->f_mode & FMODE_WRITE)))
|
|
|
|
return -EBADF;
|
|
|
|
|
|
|
|
if (unlikely(out->f_flags & O_APPEND))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2023-11-30 14:16:22 +00:00
|
|
|
ret = splice_direct_to_actor(in, &sd, actor);
|
2007-07-13 12:11:43 +00:00
|
|
|
if (ret > 0)
|
2008-05-09 11:28:36 +00:00
|
|
|
*ppos = sd.pos;
|
2007-07-13 12:11:43 +00:00
|
|
|
|
2007-06-12 19:17:17 +00:00
|
|
|
return ret;
|
2006-04-11 11:52:07 +00:00
|
|
|
}
|
2023-11-30 14:16:22 +00:00
|
|
|
/**
|
|
|
|
* do_splice_direct - splices data directly between two files
|
|
|
|
* @in: file to splice from
|
|
|
|
* @ppos: input file offset
|
|
|
|
* @out: file to splice to
|
|
|
|
* @opos: output file offset
|
|
|
|
* @len: number of bytes to splice
|
|
|
|
* @flags: splice modifier flags
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* For use by do_sendfile(). splice can easily emulate sendfile, but
|
|
|
|
* doing it in the application would incur an extra system call
|
|
|
|
* (splice in + splice out, as compared to just sendfile()). So this helper
|
|
|
|
* can splice directly through a process-private pipe.
|
|
|
|
*
|
|
|
|
* Callers already called rw_verify_area() on the entire range.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t do_splice_direct(struct file *in, loff_t *ppos, struct file *out,
|
|
|
|
loff_t *opos, size_t len, unsigned int flags)
|
2023-11-30 14:16:22 +00:00
|
|
|
{
|
|
|
|
return do_splice_direct_actor(in, ppos, out, opos, len, flags,
|
|
|
|
direct_splice_actor);
|
|
|
|
}
|
2014-10-23 22:14:35 +00:00
|
|
|
EXPORT_SYMBOL(do_splice_direct);
|
2006-04-11 11:52:07 +00:00
|
|
|
|
2023-11-30 14:16:22 +00:00
|
|
|
/**
|
|
|
|
* splice_file_range - splices data between two files for copy_file_range()
|
|
|
|
* @in: file to splice from
|
|
|
|
* @ppos: input file offset
|
|
|
|
* @out: file to splice to
|
|
|
|
* @opos: output file offset
|
|
|
|
* @len: number of bytes to splice
|
|
|
|
*
|
|
|
|
* Description:
|
|
|
|
* For use by generic_copy_file_range() and ->copy_file_range() methods.
|
2023-11-30 14:16:23 +00:00
|
|
|
* Like do_splice_direct(), but vfs_copy_file_range() already holds
|
|
|
|
* start_file_write() on @out file.
|
2023-11-30 14:16:22 +00:00
|
|
|
*
|
|
|
|
* Callers already called rw_verify_area() on the entire range.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t splice_file_range(struct file *in, loff_t *ppos, struct file *out,
|
|
|
|
loff_t *opos, size_t len)
|
2023-11-30 14:16:22 +00:00
|
|
|
{
|
|
|
|
lockdep_assert(file_write_started(out));
|
|
|
|
|
|
|
|
return do_splice_direct_actor(in, ppos, out, opos, len, 0,
|
2023-11-30 14:16:23 +00:00
|
|
|
splice_file_range_actor);
|
2023-11-30 14:16:22 +00:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(splice_file_range);
|
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
static int wait_for_space(struct pipe_inode_info *pipe, unsigned flags)
|
|
|
|
{
|
2016-12-21 18:59:34 +00:00
|
|
|
for (;;) {
|
|
|
|
if (unlikely(!pipe->readers)) {
|
|
|
|
send_sig(SIGPIPE, current, 0);
|
|
|
|
return -EPIPE;
|
|
|
|
}
|
2019-10-16 15:47:32 +00:00
|
|
|
if (!pipe_full(pipe->head, pipe->tail, pipe->max_usage))
|
2016-12-21 18:59:34 +00:00
|
|
|
return 0;
|
2016-09-18 00:44:45 +00:00
|
|
|
if (flags & SPLICE_F_NONBLOCK)
|
|
|
|
return -EAGAIN;
|
|
|
|
if (signal_pending(current))
|
|
|
|
return -ERESTARTSYS;
|
pipe: remove pipe_wait() and fix wakeup race with splice
The pipe splice code still used the old model of waiting for pipe IO by
using a non-specific "pipe_wait()" that waited for any pipe event to
happen, which depended on all pipe IO being entirely serialized by the
pipe lock. So by checking the state you were waiting for, and then
adding yourself to the wait queue before dropping the lock, you were
guaranteed to see all the wakeups.
Strictly speaking, the actual wakeups were not done under the lock, but
the pipe_wait() model still worked, because since the waiter held the
lock when checking whether it should sleep, it would always see the
current state, and the wakeup was always done after updating the state.
However, commit 0ddad21d3e99 ("pipe: use exclusive waits when reading or
writing") split the single wait-queue into two, and in the process also
made the "wait for event" code wait for _two_ wait queues, and that then
showed a race with the wakers that were not serialized by the pipe lock.
It's only splice that used that "pipe_wait()" model, so the problem
wasn't obvious, but Josef Bacik reports:
"I hit a hang with fstest btrfs/187, which does a btrfs send into
/dev/null. This works by creating a pipe, the write side is given to
the kernel to write into, and the read side is handed to a thread that
splices into a file, in this case /dev/null.
The box that was hung had the write side stuck here [pipe_write] and
the read side stuck here [splice_from_pipe_next -> pipe_wait].
[ more details about pipe_wait() scenario ]
The problem is we're doing the prepare_to_wait, which sets our state
each time, however we can be woken up either with reads or writes. In
the case above we race with the WRITER waking us up, and re-set our
state to INTERRUPTIBLE, and thus never break out of schedule"
Josef had a patch that avoided the issue in pipe_wait() by just making
it set the state only once, but the deeper problem is that pipe_wait()
depends on a level of synchonization by the pipe mutex that it really
shouldn't. And the whole "wait for any pipe state change" model really
isn't very good to begin with.
So rather than trying to work around things in pipe_wait(), remove that
legacy model of "wait for arbitrary pipe event" entirely, and actually
create functions that wait for the pipe actually being readable or
writable, and can do so without depending on the pipe lock serializing
everything.
Fixes: 0ddad21d3e99 ("pipe: use exclusive waits when reading or writing")
Link: https://lore.kernel.org/linux-fsdevel/bfa88b5ad6f069b2b679316b9e495a970130416c.1601567868.git.josef@toxicpanda.com/
Reported-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-and-tested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-02 02:14:36 +00:00
|
|
|
pipe_wait_writable(pipe);
|
2016-09-18 00:44:45 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
static int splice_pipe_to_pipe(struct pipe_inode_info *ipipe,
|
|
|
|
struct pipe_inode_info *opipe,
|
|
|
|
size_t len, unsigned int flags);
|
2006-11-04 11:49:32 +00:00
|
|
|
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t splice_file_to_pipe(struct file *in,
|
|
|
|
struct pipe_inode_info *opipe,
|
|
|
|
loff_t *offset,
|
|
|
|
size_t len, unsigned int flags)
|
2021-01-26 03:23:03 +00:00
|
|
|
{
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret;
|
2021-01-26 03:23:03 +00:00
|
|
|
|
|
|
|
pipe_lock(opipe);
|
|
|
|
ret = wait_for_space(opipe, flags);
|
|
|
|
if (!ret)
|
2023-11-22 12:27:03 +00:00
|
|
|
ret = do_splice_read(in, offset, opipe, len, flags);
|
2021-01-26 03:23:03 +00:00
|
|
|
pipe_unlock(opipe);
|
|
|
|
if (ret > 0)
|
|
|
|
wakeup_pipe_readers(opipe);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2006-04-02 21:05:09 +00:00
|
|
|
/*
|
|
|
|
* Determine where to splice to/from.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t do_splice(struct file *in, loff_t *off_in, struct file *out,
|
|
|
|
loff_t *off_out, size_t len, unsigned int flags)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
struct pipe_inode_info *ipipe;
|
|
|
|
struct pipe_inode_info *opipe;
|
2013-06-20 14:58:36 +00:00
|
|
|
loff_t offset;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2020-05-04 19:39:35 +00:00
|
|
|
if (unlikely(!(in->f_mode & FMODE_READ) ||
|
|
|
|
!(out->f_mode & FMODE_WRITE)))
|
|
|
|
return -EBADF;
|
|
|
|
|
pipe: Add general notification queue support
Make it possible to have a general notification queue built on top of a
standard pipe. Notifications are 'spliced' into the pipe and then read
out. splice(), vmsplice() and sendfile() are forbidden on pipes used for
notifications as post_one_notification() cannot take pipe->mutex. This
means that notifications could be posted in between individual pipe
buffers, making iov_iter_revert() difficult to effect.
The way the notification queue is used is:
(1) An application opens a pipe with a special flag and indicates the
number of messages it wishes to be able to queue at once (this can
only be set once):
pipe2(fds, O_NOTIFICATION_PIPE);
ioctl(fds[0], IOC_WATCH_QUEUE_SET_SIZE, queue_depth);
(2) The application then uses poll() and read() as normal to extract data
from the pipe. read() will return multiple notifications if the
buffer is big enough, but it will not split a notification across
buffers - rather it will return a short read or EMSGSIZE.
Notification messages include a length in the header so that the
caller can split them up.
Each message has a header that describes it:
struct watch_notification {
__u32 type:24;
__u32 subtype:8;
__u32 info;
};
The type indicates the source (eg. mount tree changes, superblock events,
keyring changes, block layer events) and the subtype indicates the event
type (eg. mount, unmount; EIO, EDQUOT; link, unlink). The info field
indicates a number of things, including the entry length, an ID assigned to
a watchpoint contributing to this buffer and type-specific flags.
Supplementary data, such as the key ID that generated an event, can be
attached in additional slots. The maximum message size is 127 bytes.
Messages may not be padded or aligned, so there is no guarantee, for
example, that the notification type will be on a 4-byte bounary.
Signed-off-by: David Howells <dhowells@redhat.com>
2020-01-14 17:07:11 +00:00
|
|
|
ipipe = get_pipe_info(in, true);
|
|
|
|
opipe = get_pipe_info(out, true);
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
|
|
|
|
if (ipipe && opipe) {
|
|
|
|
if (off_in || off_out)
|
|
|
|
return -ESPIPE;
|
|
|
|
|
|
|
|
/* Splicing to self would be fun, but... */
|
|
|
|
if (ipipe == opipe)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2019-02-07 15:45:19 +00:00
|
|
|
if ((in->f_flags | out->f_flags) & O_NONBLOCK)
|
|
|
|
flags |= SPLICE_F_NONBLOCK;
|
|
|
|
|
2023-07-03 14:42:13 +00:00
|
|
|
ret = splice_pipe_to_pipe(ipipe, opipe, len, flags);
|
|
|
|
} else if (ipipe) {
|
2006-04-10 13:18:58 +00:00
|
|
|
if (off_in)
|
|
|
|
return -ESPIPE;
|
2006-04-11 11:52:07 +00:00
|
|
|
if (off_out) {
|
2010-06-29 11:10:36 +00:00
|
|
|
if (!(out->f_mode & FMODE_PWRITE))
|
2006-04-11 11:52:07 +00:00
|
|
|
return -EINVAL;
|
2020-10-22 20:15:51 +00:00
|
|
|
offset = *off_out;
|
2013-06-20 14:58:36 +00:00
|
|
|
} else {
|
|
|
|
offset = out->f_pos;
|
|
|
|
}
|
2006-04-10 13:18:58 +00:00
|
|
|
|
2013-06-19 11:41:54 +00:00
|
|
|
if (unlikely(out->f_flags & O_APPEND))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
ret = rw_verify_area(WRITE, out, &offset, len);
|
|
|
|
if (unlikely(ret < 0))
|
|
|
|
return ret;
|
|
|
|
|
2019-02-07 15:45:19 +00:00
|
|
|
if (in->f_flags & O_NONBLOCK)
|
|
|
|
flags |= SPLICE_F_NONBLOCK;
|
|
|
|
|
2013-05-24 00:07:11 +00:00
|
|
|
file_start_write(out);
|
2013-06-20 14:58:36 +00:00
|
|
|
ret = do_splice_from(ipipe, out, &offset, len, flags);
|
2013-05-24 00:07:11 +00:00
|
|
|
file_end_write(out);
|
2006-04-19 13:57:05 +00:00
|
|
|
|
2013-06-20 14:58:36 +00:00
|
|
|
if (!off_out)
|
|
|
|
out->f_pos = offset;
|
2020-10-22 20:15:51 +00:00
|
|
|
else
|
|
|
|
*off_out = offset;
|
2023-07-03 14:42:13 +00:00
|
|
|
} else if (opipe) {
|
2006-04-10 13:18:58 +00:00
|
|
|
if (off_out)
|
|
|
|
return -ESPIPE;
|
2006-04-11 11:52:07 +00:00
|
|
|
if (off_in) {
|
2010-06-29 11:10:36 +00:00
|
|
|
if (!(in->f_mode & FMODE_PREAD))
|
2006-04-11 11:52:07 +00:00
|
|
|
return -EINVAL;
|
2020-10-22 20:15:51 +00:00
|
|
|
offset = *off_in;
|
2013-06-20 14:58:36 +00:00
|
|
|
} else {
|
|
|
|
offset = in->f_pos;
|
|
|
|
}
|
2006-04-10 13:18:58 +00:00
|
|
|
|
2023-11-22 12:27:03 +00:00
|
|
|
ret = rw_verify_area(READ, in, &offset, len);
|
|
|
|
if (unlikely(ret < 0))
|
|
|
|
return ret;
|
|
|
|
|
2019-02-07 15:45:19 +00:00
|
|
|
if (out->f_flags & O_NONBLOCK)
|
|
|
|
flags |= SPLICE_F_NONBLOCK;
|
|
|
|
|
2021-01-26 03:23:03 +00:00
|
|
|
ret = splice_file_to_pipe(in, opipe, &offset, len, flags);
|
2023-03-22 06:25:19 +00:00
|
|
|
|
2013-06-20 14:58:36 +00:00
|
|
|
if (!off_in)
|
|
|
|
in->f_pos = offset;
|
2020-10-22 20:15:51 +00:00
|
|
|
else
|
|
|
|
*off_in = offset;
|
2023-07-03 14:42:13 +00:00
|
|
|
} else {
|
|
|
|
ret = -EINVAL;
|
|
|
|
}
|
2006-04-19 13:57:05 +00:00
|
|
|
|
2023-07-03 14:42:13 +00:00
|
|
|
if (ret > 0) {
|
|
|
|
/*
|
|
|
|
* Generate modify out before access in:
|
|
|
|
* do_splice_from() may've already sent modify out,
|
|
|
|
* and this ensures the events get merged.
|
|
|
|
*/
|
|
|
|
fsnotify_modify(out);
|
|
|
|
fsnotify_access(in);
|
2006-04-10 13:18:58 +00:00
|
|
|
}
|
2006-03-30 13:15:30 +00:00
|
|
|
|
2023-07-03 14:42:13 +00:00
|
|
|
return ret;
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
|
|
|
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t __do_splice(struct file *in, loff_t __user *off_in,
|
|
|
|
struct file *out, loff_t __user *off_out,
|
|
|
|
size_t len, unsigned int flags)
|
2020-10-22 20:15:51 +00:00
|
|
|
{
|
|
|
|
struct pipe_inode_info *ipipe;
|
|
|
|
struct pipe_inode_info *opipe;
|
|
|
|
loff_t offset, *__off_in = NULL, *__off_out = NULL;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret;
|
2020-10-22 20:15:51 +00:00
|
|
|
|
|
|
|
ipipe = get_pipe_info(in, true);
|
|
|
|
opipe = get_pipe_info(out, true);
|
|
|
|
|
2023-04-24 22:32:55 +00:00
|
|
|
if (ipipe) {
|
|
|
|
if (off_in)
|
|
|
|
return -ESPIPE;
|
|
|
|
pipe_clear_nowait(in);
|
|
|
|
}
|
|
|
|
if (opipe) {
|
|
|
|
if (off_out)
|
|
|
|
return -ESPIPE;
|
|
|
|
pipe_clear_nowait(out);
|
|
|
|
}
|
2020-10-22 20:15:51 +00:00
|
|
|
|
|
|
|
if (off_out) {
|
|
|
|
if (copy_from_user(&offset, off_out, sizeof(loff_t)))
|
|
|
|
return -EFAULT;
|
|
|
|
__off_out = &offset;
|
|
|
|
}
|
|
|
|
if (off_in) {
|
|
|
|
if (copy_from_user(&offset, off_in, sizeof(loff_t)))
|
|
|
|
return -EFAULT;
|
|
|
|
__off_in = &offset;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = do_splice(in, __off_in, out, __off_out, len, flags);
|
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
if (__off_out && copy_to_user(off_out, __off_out, sizeof(loff_t)))
|
|
|
|
return -EFAULT;
|
|
|
|
if (__off_in && copy_to_user(off_in, __off_in, sizeof(loff_t)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t iter_to_pipe(struct iov_iter *from,
|
|
|
|
struct pipe_inode_info *pipe,
|
|
|
|
unsigned int flags)
|
2006-04-26 08:59:21 +00:00
|
|
|
{
|
2016-09-18 02:38:20 +00:00
|
|
|
struct pipe_buffer buf = {
|
|
|
|
.ops = &user_page_pipe_buf_ops,
|
|
|
|
.flags = flags
|
|
|
|
};
|
|
|
|
size_t total = 0;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret = 0;
|
2016-09-18 02:38:20 +00:00
|
|
|
|
2022-06-09 15:07:52 +00:00
|
|
|
while (iov_iter_count(from)) {
|
2016-09-18 02:38:20 +00:00
|
|
|
struct page *pages[16];
|
2022-06-09 15:07:52 +00:00
|
|
|
ssize_t left;
|
2016-09-18 00:25:06 +00:00
|
|
|
size_t start;
|
2022-06-09 15:07:52 +00:00
|
|
|
int i, n;
|
2016-09-18 00:25:06 +00:00
|
|
|
|
2022-06-09 15:07:52 +00:00
|
|
|
left = iov_iter_get_pages2(from, pages, ~0UL, 16, &start);
|
|
|
|
if (left <= 0) {
|
|
|
|
ret = left;
|
2016-09-18 02:38:20 +00:00
|
|
|
break;
|
|
|
|
}
|
2016-09-18 00:25:06 +00:00
|
|
|
|
2022-06-09 15:07:52 +00:00
|
|
|
n = DIV_ROUND_UP(left + start, PAGE_SIZE);
|
|
|
|
for (i = 0; i < n; i++) {
|
|
|
|
int size = min_t(int, left, PAGE_SIZE - start);
|
|
|
|
|
|
|
|
buf.page = pages[i];
|
|
|
|
buf.offset = start;
|
|
|
|
buf.len = size;
|
|
|
|
ret = add_to_pipe(pipe, &buf);
|
|
|
|
if (unlikely(ret < 0)) {
|
|
|
|
iov_iter_revert(from, left);
|
|
|
|
// this one got dropped by add_to_pipe()
|
|
|
|
while (++i < n)
|
|
|
|
put_page(pages[i]);
|
|
|
|
goto out;
|
2016-09-18 02:38:20 +00:00
|
|
|
}
|
2022-06-09 15:07:52 +00:00
|
|
|
total += ret;
|
|
|
|
left -= size;
|
|
|
|
start = 0;
|
2006-04-26 08:59:21 +00:00
|
|
|
}
|
|
|
|
}
|
2022-06-09 15:07:52 +00:00
|
|
|
out:
|
2016-09-18 02:38:20 +00:00
|
|
|
return total ? total : ret;
|
2006-04-26 08:59:21 +00:00
|
|
|
}
|
|
|
|
|
2007-06-14 11:08:55 +00:00
|
|
|
static int pipe_to_user(struct pipe_inode_info *pipe, struct pipe_buffer *buf,
|
|
|
|
struct splice_desc *sd)
|
|
|
|
{
|
2014-02-03 23:19:51 +00:00
|
|
|
int n = copy_page_to_iter(buf->page, buf->offset, sd->len, sd->u.data);
|
|
|
|
return n == sd->len ? n : -EFAULT;
|
2007-06-14 11:08:55 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* For lack of a better implementation, implement vmsplice() to userspace
|
|
|
|
* as a simple copy of the pipes pages to the user iov.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t vmsplice_to_user(struct file *file, struct iov_iter *iter,
|
|
|
|
unsigned int flags)
|
2007-06-14 11:08:55 +00:00
|
|
|
{
|
pipe: Add general notification queue support
Make it possible to have a general notification queue built on top of a
standard pipe. Notifications are 'spliced' into the pipe and then read
out. splice(), vmsplice() and sendfile() are forbidden on pipes used for
notifications as post_one_notification() cannot take pipe->mutex. This
means that notifications could be posted in between individual pipe
buffers, making iov_iter_revert() difficult to effect.
The way the notification queue is used is:
(1) An application opens a pipe with a special flag and indicates the
number of messages it wishes to be able to queue at once (this can
only be set once):
pipe2(fds, O_NOTIFICATION_PIPE);
ioctl(fds[0], IOC_WATCH_QUEUE_SET_SIZE, queue_depth);
(2) The application then uses poll() and read() as normal to extract data
from the pipe. read() will return multiple notifications if the
buffer is big enough, but it will not split a notification across
buffers - rather it will return a short read or EMSGSIZE.
Notification messages include a length in the header so that the
caller can split them up.
Each message has a header that describes it:
struct watch_notification {
__u32 type:24;
__u32 subtype:8;
__u32 info;
};
The type indicates the source (eg. mount tree changes, superblock events,
keyring changes, block layer events) and the subtype indicates the event
type (eg. mount, unmount; EIO, EDQUOT; link, unlink). The info field
indicates a number of things, including the entry length, an ID assigned to
a watchpoint contributing to this buffer and type-specific flags.
Supplementary data, such as the key ID that generated an event, can be
attached in additional slots. The maximum message size is 127 bytes.
Messages may not be padded or aligned, so there is no guarantee, for
example, that the notification type will be on a 4-byte bounary.
Signed-off-by: David Howells <dhowells@redhat.com>
2020-01-14 17:07:11 +00:00
|
|
|
struct pipe_inode_info *pipe = get_pipe_info(file, true);
|
2018-05-27 01:39:52 +00:00
|
|
|
struct splice_desc sd = {
|
|
|
|
.total_len = iov_iter_count(iter),
|
|
|
|
.flags = flags,
|
|
|
|
.u.data = iter
|
|
|
|
};
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret = 0;
|
2007-06-14 11:08:55 +00:00
|
|
|
|
|
|
|
if (!pipe)
|
|
|
|
return -EBADF;
|
|
|
|
|
2023-04-24 22:32:55 +00:00
|
|
|
pipe_clear_nowait(file);
|
|
|
|
|
2015-03-21 23:17:55 +00:00
|
|
|
if (sd.total_len) {
|
|
|
|
pipe_lock(pipe);
|
|
|
|
ret = __splice_from_pipe(pipe, &sd, pipe_to_user);
|
|
|
|
pipe_unlock(pipe);
|
|
|
|
}
|
2007-06-14 11:08:55 +00:00
|
|
|
|
2023-07-03 14:42:17 +00:00
|
|
|
if (ret > 0)
|
|
|
|
fsnotify_access(file);
|
|
|
|
|
2007-06-14 11:08:55 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2006-04-26 08:59:21 +00:00
|
|
|
/*
|
|
|
|
* vmsplice splices a user address range into a pipe. It can be thought of
|
|
|
|
* as splice-from-memory, where the regular splice is splice-from-file (or
|
|
|
|
* to file). In both cases the output is a pipe, naturally.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t vmsplice_to_pipe(struct file *file, struct iov_iter *iter,
|
|
|
|
unsigned int flags)
|
2006-04-26 08:59:21 +00:00
|
|
|
{
|
2006-11-04 11:49:32 +00:00
|
|
|
struct pipe_inode_info *pipe;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret = 0;
|
2016-09-18 02:38:20 +00:00
|
|
|
unsigned buf_flag = 0;
|
|
|
|
|
|
|
|
if (flags & SPLICE_F_GIFT)
|
|
|
|
buf_flag = PIPE_BUF_FLAG_GIFT;
|
2006-04-26 08:59:21 +00:00
|
|
|
|
pipe: Add general notification queue support
Make it possible to have a general notification queue built on top of a
standard pipe. Notifications are 'spliced' into the pipe and then read
out. splice(), vmsplice() and sendfile() are forbidden on pipes used for
notifications as post_one_notification() cannot take pipe->mutex. This
means that notifications could be posted in between individual pipe
buffers, making iov_iter_revert() difficult to effect.
The way the notification queue is used is:
(1) An application opens a pipe with a special flag and indicates the
number of messages it wishes to be able to queue at once (this can
only be set once):
pipe2(fds, O_NOTIFICATION_PIPE);
ioctl(fds[0], IOC_WATCH_QUEUE_SET_SIZE, queue_depth);
(2) The application then uses poll() and read() as normal to extract data
from the pipe. read() will return multiple notifications if the
buffer is big enough, but it will not split a notification across
buffers - rather it will return a short read or EMSGSIZE.
Notification messages include a length in the header so that the
caller can split them up.
Each message has a header that describes it:
struct watch_notification {
__u32 type:24;
__u32 subtype:8;
__u32 info;
};
The type indicates the source (eg. mount tree changes, superblock events,
keyring changes, block layer events) and the subtype indicates the event
type (eg. mount, unmount; EIO, EDQUOT; link, unlink). The info field
indicates a number of things, including the entry length, an ID assigned to
a watchpoint contributing to this buffer and type-specific flags.
Supplementary data, such as the key ID that generated an event, can be
attached in additional slots. The maximum message size is 127 bytes.
Messages may not be padded or aligned, so there is no guarantee, for
example, that the notification type will be on a 4-byte bounary.
Signed-off-by: David Howells <dhowells@redhat.com>
2020-01-14 17:07:11 +00:00
|
|
|
pipe = get_pipe_info(file, true);
|
2006-11-04 11:49:32 +00:00
|
|
|
if (!pipe)
|
2006-04-26 08:59:21 +00:00
|
|
|
return -EBADF;
|
|
|
|
|
2023-04-24 22:32:55 +00:00
|
|
|
pipe_clear_nowait(file);
|
|
|
|
|
2016-09-18 00:44:45 +00:00
|
|
|
pipe_lock(pipe);
|
|
|
|
ret = wait_for_space(pipe, flags);
|
2016-09-18 02:38:20 +00:00
|
|
|
if (!ret)
|
2018-05-27 01:39:52 +00:00
|
|
|
ret = iter_to_pipe(iter, pipe, buf_flag);
|
2016-09-18 00:44:45 +00:00
|
|
|
pipe_unlock(pipe);
|
2023-07-03 14:42:17 +00:00
|
|
|
if (ret > 0) {
|
2016-09-18 00:44:45 +00:00
|
|
|
wakeup_pipe_readers(pipe);
|
2023-07-03 14:42:17 +00:00
|
|
|
fsnotify_modify(file);
|
|
|
|
}
|
2010-05-20 08:43:18 +00:00
|
|
|
return ret;
|
2006-04-26 08:59:21 +00:00
|
|
|
}
|
|
|
|
|
2018-05-27 01:39:52 +00:00
|
|
|
static int vmsplice_type(struct fd f, int *type)
|
|
|
|
{
|
|
|
|
if (!f.file)
|
|
|
|
return -EBADF;
|
|
|
|
if (f.file->f_mode & FMODE_WRITE) {
|
2022-09-16 00:25:47 +00:00
|
|
|
*type = ITER_SOURCE;
|
2018-05-27 01:39:52 +00:00
|
|
|
} else if (f.file->f_mode & FMODE_READ) {
|
2022-09-16 00:25:47 +00:00
|
|
|
*type = ITER_DEST;
|
2018-05-27 01:39:52 +00:00
|
|
|
} else {
|
|
|
|
fdput(f);
|
|
|
|
return -EBADF;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2007-06-14 11:08:55 +00:00
|
|
|
/*
|
|
|
|
* Note that vmsplice only really supports true splicing _from_ user memory
|
|
|
|
* to a pipe, not the other way around. Splicing from user memory is a simple
|
|
|
|
* operation that can be supported without any funky alignment restrictions
|
|
|
|
* or nasty vm tricks. We simply map in the user memory and fill them into
|
|
|
|
* a pipe. The reverse isn't quite as easy, though. There are two possible
|
|
|
|
* solutions for that:
|
|
|
|
*
|
|
|
|
* - memcpy() the data internally, at which point we might as well just
|
|
|
|
* do a regular read() on the buffer anyway.
|
|
|
|
* - Lots of nasty vm tricks, that are neither fast nor flexible (it
|
|
|
|
* has restriction limitations on both ends of the pipe).
|
|
|
|
*
|
|
|
|
* Currently we punt and implement it as a normal copy, see pipe_to_user().
|
|
|
|
*
|
|
|
|
*/
|
2018-05-27 01:39:52 +00:00
|
|
|
SYSCALL_DEFINE4(vmsplice, int, fd, const struct iovec __user *, uiov,
|
2018-03-17 14:00:24 +00:00
|
|
|
unsigned long, nr_segs, unsigned int, flags)
|
|
|
|
{
|
2018-05-27 01:39:52 +00:00
|
|
|
struct iovec iovstack[UIO_FASTIOV];
|
|
|
|
struct iovec *iov = iovstack;
|
|
|
|
struct iov_iter iter;
|
2019-05-14 22:02:22 +00:00
|
|
|
ssize_t error;
|
2018-05-27 01:39:52 +00:00
|
|
|
struct fd f;
|
|
|
|
int type;
|
|
|
|
|
2020-09-25 04:51:44 +00:00
|
|
|
if (unlikely(flags & ~SPLICE_F_ALL))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2018-05-27 01:39:52 +00:00
|
|
|
f = fdget(fd);
|
|
|
|
error = vmsplice_type(f, &type);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
|
|
|
error = import_iovec(type, uiov, nr_segs,
|
|
|
|
ARRAY_SIZE(iovstack), &iov, &iter);
|
2020-09-25 04:51:44 +00:00
|
|
|
if (error < 0)
|
|
|
|
goto out_fdput;
|
2018-03-17 14:00:24 +00:00
|
|
|
|
2020-09-25 04:51:44 +00:00
|
|
|
if (!iov_iter_count(&iter))
|
|
|
|
error = 0;
|
2022-09-16 00:25:47 +00:00
|
|
|
else if (type == ITER_SOURCE)
|
2020-09-25 04:51:44 +00:00
|
|
|
error = vmsplice_to_pipe(f.file, &iter, flags);
|
|
|
|
else
|
|
|
|
error = vmsplice_to_user(f.file, &iter, flags);
|
2018-05-27 01:39:52 +00:00
|
|
|
|
2020-09-25 04:51:44 +00:00
|
|
|
kfree(iov);
|
|
|
|
out_fdput:
|
2018-05-27 01:39:52 +00:00
|
|
|
fdput(f);
|
|
|
|
return error;
|
2013-03-02 15:19:56 +00:00
|
|
|
}
|
|
|
|
|
2009-01-14 13:14:33 +00:00
|
|
|
SYSCALL_DEFINE6(splice, int, fd_in, loff_t __user *, off_in,
|
|
|
|
int, fd_out, loff_t __user *, off_out,
|
|
|
|
size_t, len, unsigned int, flags)
|
2006-03-30 13:15:30 +00:00
|
|
|
{
|
2012-08-28 16:52:22 +00:00
|
|
|
struct fd in, out;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t error;
|
2006-03-30 13:15:30 +00:00
|
|
|
|
|
|
|
if (unlikely(!len))
|
|
|
|
return 0;
|
|
|
|
|
2016-12-10 18:17:32 +00:00
|
|
|
if (unlikely(flags & ~SPLICE_F_ALL))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2006-03-30 13:15:30 +00:00
|
|
|
error = -EBADF;
|
2012-08-28 16:52:22 +00:00
|
|
|
in = fdget(fd_in);
|
|
|
|
if (in.file) {
|
2020-05-04 19:39:35 +00:00
|
|
|
out = fdget(fd_out);
|
|
|
|
if (out.file) {
|
2020-10-22 20:15:51 +00:00
|
|
|
error = __do_splice(in.file, off_in, out.file, off_out,
|
2023-12-12 09:44:36 +00:00
|
|
|
len, flags);
|
2020-05-04 19:39:35 +00:00
|
|
|
fdput(out);
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
2012-08-28 16:52:22 +00:00
|
|
|
fdput(in);
|
2006-03-30 13:15:30 +00:00
|
|
|
}
|
|
|
|
return error;
|
|
|
|
}
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
/*
|
|
|
|
* Make sure there's data to read. Wait for input if we can, otherwise
|
|
|
|
* return an appropriate error.
|
|
|
|
*/
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
static int ipipe_prep(struct pipe_inode_info *pipe, unsigned int flags)
|
2006-07-10 09:00:01 +00:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
/*
|
2019-11-15 13:30:32 +00:00
|
|
|
* Check the pipe occupancy without the inode lock first. This function
|
2006-07-10 09:00:01 +00:00
|
|
|
* is speculative anyways, so missing one is ok.
|
|
|
|
*/
|
2019-11-15 13:30:32 +00:00
|
|
|
if (!pipe_empty(pipe->head, pipe->tail))
|
2006-07-10 09:00:01 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
ret = 0;
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_lock(pipe);
|
2006-07-10 09:00:01 +00:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
while (pipe_empty(pipe->head, pipe->tail)) {
|
2006-07-10 09:00:01 +00:00
|
|
|
if (signal_pending(current)) {
|
|
|
|
ret = -ERESTARTSYS;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (!pipe->writers)
|
|
|
|
break;
|
pipe: remove 'waiting_writers' merging logic
This code is ancient, and goes back to when we only had a single page
for the pipe buffers. The exact history is hidden in the mists of time
(ie "before git", and in fact predates the BK repository too).
At that long-ago point in time, it actually helped to try to merge big
back-and-forth pipe reads and writes, and not limit pipe reads to the
single pipe buffer in length just because that was all we had at a time.
However, since then we've expanded the pipe buffers to multiple pages,
and this logic really doesn't seem to make sense. And a lot of it is
somewhat questionable (ie "hmm, the user asked for a non-blocking read,
but we see that there's a writer pending, so let's wait anyway to get
the extra data that the writer will have").
But more importantly, it makes the "go to sleep" logic much less
obvious, and considering the wakeup issues we've had, I want to make for
less of those kinds of things.
Cc: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-12-07 21:21:01 +00:00
|
|
|
if (flags & SPLICE_F_NONBLOCK) {
|
|
|
|
ret = -EAGAIN;
|
|
|
|
break;
|
2006-07-10 09:00:01 +00:00
|
|
|
}
|
pipe: remove pipe_wait() and fix wakeup race with splice
The pipe splice code still used the old model of waiting for pipe IO by
using a non-specific "pipe_wait()" that waited for any pipe event to
happen, which depended on all pipe IO being entirely serialized by the
pipe lock. So by checking the state you were waiting for, and then
adding yourself to the wait queue before dropping the lock, you were
guaranteed to see all the wakeups.
Strictly speaking, the actual wakeups were not done under the lock, but
the pipe_wait() model still worked, because since the waiter held the
lock when checking whether it should sleep, it would always see the
current state, and the wakeup was always done after updating the state.
However, commit 0ddad21d3e99 ("pipe: use exclusive waits when reading or
writing") split the single wait-queue into two, and in the process also
made the "wait for event" code wait for _two_ wait queues, and that then
showed a race with the wakers that were not serialized by the pipe lock.
It's only splice that used that "pipe_wait()" model, so the problem
wasn't obvious, but Josef Bacik reports:
"I hit a hang with fstest btrfs/187, which does a btrfs send into
/dev/null. This works by creating a pipe, the write side is given to
the kernel to write into, and the read side is handed to a thread that
splices into a file, in this case /dev/null.
The box that was hung had the write side stuck here [pipe_write] and
the read side stuck here [splice_from_pipe_next -> pipe_wait].
[ more details about pipe_wait() scenario ]
The problem is we're doing the prepare_to_wait, which sets our state
each time, however we can be woken up either with reads or writes. In
the case above we race with the WRITER waking us up, and re-set our
state to INTERRUPTIBLE, and thus never break out of schedule"
Josef had a patch that avoided the issue in pipe_wait() by just making
it set the state only once, but the deeper problem is that pipe_wait()
depends on a level of synchonization by the pipe mutex that it really
shouldn't. And the whole "wait for any pipe state change" model really
isn't very good to begin with.
So rather than trying to work around things in pipe_wait(), remove that
legacy model of "wait for arbitrary pipe event" entirely, and actually
create functions that wait for the pipe actually being readable or
writable, and can do so without depending on the pipe lock serializing
everything.
Fixes: 0ddad21d3e99 ("pipe: use exclusive waits when reading or writing")
Link: https://lore.kernel.org/linux-fsdevel/bfa88b5ad6f069b2b679316b9e495a970130416c.1601567868.git.josef@toxicpanda.com/
Reported-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-and-tested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-02 02:14:36 +00:00
|
|
|
pipe_wait_readable(pipe);
|
2006-07-10 09:00:01 +00:00
|
|
|
}
|
|
|
|
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_unlock(pipe);
|
2006-07-10 09:00:01 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure there's writeable room. Wait for room if we can, otherwise
|
|
|
|
* return an appropriate error.
|
|
|
|
*/
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
static int opipe_prep(struct pipe_inode_info *pipe, unsigned int flags)
|
2006-07-10 09:00:01 +00:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
/*
|
2019-11-15 13:30:32 +00:00
|
|
|
* Check pipe occupancy without the inode lock first. This function
|
2006-07-10 09:00:01 +00:00
|
|
|
* is speculative anyways, so missing one is ok.
|
|
|
|
*/
|
2020-05-19 23:51:59 +00:00
|
|
|
if (!pipe_full(pipe->head, pipe->tail, pipe->max_usage))
|
2006-07-10 09:00:01 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
ret = 0;
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_lock(pipe);
|
2006-07-10 09:00:01 +00:00
|
|
|
|
2019-10-16 15:47:32 +00:00
|
|
|
while (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) {
|
2006-07-10 09:00:01 +00:00
|
|
|
if (!pipe->readers) {
|
|
|
|
send_sig(SIGPIPE, current, 0);
|
|
|
|
ret = -EPIPE;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (flags & SPLICE_F_NONBLOCK) {
|
|
|
|
ret = -EAGAIN;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (signal_pending(current)) {
|
|
|
|
ret = -ERESTARTSYS;
|
|
|
|
break;
|
|
|
|
}
|
pipe: remove pipe_wait() and fix wakeup race with splice
The pipe splice code still used the old model of waiting for pipe IO by
using a non-specific "pipe_wait()" that waited for any pipe event to
happen, which depended on all pipe IO being entirely serialized by the
pipe lock. So by checking the state you were waiting for, and then
adding yourself to the wait queue before dropping the lock, you were
guaranteed to see all the wakeups.
Strictly speaking, the actual wakeups were not done under the lock, but
the pipe_wait() model still worked, because since the waiter held the
lock when checking whether it should sleep, it would always see the
current state, and the wakeup was always done after updating the state.
However, commit 0ddad21d3e99 ("pipe: use exclusive waits when reading or
writing") split the single wait-queue into two, and in the process also
made the "wait for event" code wait for _two_ wait queues, and that then
showed a race with the wakers that were not serialized by the pipe lock.
It's only splice that used that "pipe_wait()" model, so the problem
wasn't obvious, but Josef Bacik reports:
"I hit a hang with fstest btrfs/187, which does a btrfs send into
/dev/null. This works by creating a pipe, the write side is given to
the kernel to write into, and the read side is handed to a thread that
splices into a file, in this case /dev/null.
The box that was hung had the write side stuck here [pipe_write] and
the read side stuck here [splice_from_pipe_next -> pipe_wait].
[ more details about pipe_wait() scenario ]
The problem is we're doing the prepare_to_wait, which sets our state
each time, however we can be woken up either with reads or writes. In
the case above we race with the WRITER waking us up, and re-set our
state to INTERRUPTIBLE, and thus never break out of schedule"
Josef had a patch that avoided the issue in pipe_wait() by just making
it set the state only once, but the deeper problem is that pipe_wait()
depends on a level of synchonization by the pipe mutex that it really
shouldn't. And the whole "wait for any pipe state change" model really
isn't very good to begin with.
So rather than trying to work around things in pipe_wait(), remove that
legacy model of "wait for arbitrary pipe event" entirely, and actually
create functions that wait for the pipe actually being readable or
writable, and can do so without depending on the pipe lock serializing
everything.
Fixes: 0ddad21d3e99 ("pipe: use exclusive waits when reading or writing")
Link: https://lore.kernel.org/linux-fsdevel/bfa88b5ad6f069b2b679316b9e495a970130416c.1601567868.git.josef@toxicpanda.com/
Reported-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-and-tested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2020-10-02 02:14:36 +00:00
|
|
|
pipe_wait_writable(pipe);
|
2006-07-10 09:00:01 +00:00
|
|
|
}
|
|
|
|
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_unlock(pipe);
|
2006-07-10 09:00:01 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
/*
|
|
|
|
* Splice contents of ipipe to opipe.
|
|
|
|
*/
|
|
|
|
static int splice_pipe_to_pipe(struct pipe_inode_info *ipipe,
|
|
|
|
struct pipe_inode_info *opipe,
|
|
|
|
size_t len, unsigned int flags)
|
|
|
|
{
|
|
|
|
struct pipe_buffer *ibuf, *obuf;
|
2019-11-15 13:30:32 +00:00
|
|
|
unsigned int i_head, o_head;
|
|
|
|
unsigned int i_tail, o_tail;
|
|
|
|
unsigned int i_mask, o_mask;
|
|
|
|
int ret = 0;
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
bool input_wakeup = false;
|
|
|
|
|
|
|
|
|
|
|
|
retry:
|
|
|
|
ret = ipipe_prep(ipipe, flags);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
ret = opipe_prep(opipe, flags);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Potential ABBA deadlock, work around it by ordering lock
|
|
|
|
* grabbing by pipe info address. Otherwise two different processes
|
|
|
|
* could deadlock (one doing tee from A -> B, the other from B -> A).
|
|
|
|
*/
|
|
|
|
pipe_double_lock(ipipe, opipe);
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
i_tail = ipipe->tail;
|
|
|
|
i_mask = ipipe->ring_size - 1;
|
|
|
|
o_head = opipe->head;
|
|
|
|
o_mask = opipe->ring_size - 1;
|
|
|
|
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
do {
|
2019-11-15 13:30:32 +00:00
|
|
|
size_t o_len;
|
|
|
|
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
if (!opipe->readers) {
|
|
|
|
send_sig(SIGPIPE, current, 0);
|
|
|
|
if (!ret)
|
|
|
|
ret = -EPIPE;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
i_head = ipipe->head;
|
|
|
|
o_tail = opipe->tail;
|
|
|
|
|
|
|
|
if (pipe_empty(i_head, i_tail) && !ipipe->writers)
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
break;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Cannot make any progress, because either the input
|
|
|
|
* pipe is empty or the output pipe is full.
|
|
|
|
*/
|
2019-11-15 13:30:32 +00:00
|
|
|
if (pipe_empty(i_head, i_tail) ||
|
2019-10-16 15:47:32 +00:00
|
|
|
pipe_full(o_head, o_tail, opipe->max_usage)) {
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
/* Already processed some buffers, break */
|
|
|
|
if (ret)
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (flags & SPLICE_F_NONBLOCK) {
|
|
|
|
ret = -EAGAIN;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We raced with another reader/writer and haven't
|
|
|
|
* managed to process any buffers. A zero return
|
|
|
|
* value means EOF, so retry instead.
|
|
|
|
*/
|
|
|
|
pipe_unlock(ipipe);
|
|
|
|
pipe_unlock(opipe);
|
|
|
|
goto retry;
|
|
|
|
}
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
ibuf = &ipipe->bufs[i_tail & i_mask];
|
|
|
|
obuf = &opipe->bufs[o_head & o_mask];
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
|
|
|
|
if (len >= ibuf->len) {
|
|
|
|
/*
|
|
|
|
* Simply move the whole buffer from ipipe to opipe
|
|
|
|
*/
|
|
|
|
*obuf = *ibuf;
|
|
|
|
ibuf->ops = NULL;
|
2019-11-15 13:30:32 +00:00
|
|
|
i_tail++;
|
|
|
|
ipipe->tail = i_tail;
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
input_wakeup = true;
|
2019-11-15 13:30:32 +00:00
|
|
|
o_len = obuf->len;
|
|
|
|
o_head++;
|
|
|
|
opipe->head = o_head;
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Get a reference to this pipe buffer,
|
|
|
|
* so we can copy the contents over.
|
|
|
|
*/
|
2019-04-05 21:02:10 +00:00
|
|
|
if (!pipe_buf_get(ipipe, ibuf)) {
|
|
|
|
if (ret == 0)
|
|
|
|
ret = -EFAULT;
|
|
|
|
break;
|
|
|
|
}
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
*obuf = *ibuf;
|
|
|
|
|
|
|
|
/*
|
2020-05-20 15:58:12 +00:00
|
|
|
* Don't inherit the gift and merge flags, we need to
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
* prevent multiple steals of this page.
|
|
|
|
*/
|
|
|
|
obuf->flags &= ~PIPE_BUF_FLAG_GIFT;
|
2020-05-20 15:58:12 +00:00
|
|
|
obuf->flags &= ~PIPE_BUF_FLAG_CAN_MERGE;
|
splice: don't merge into linked buffers
Before this patch, it was possible for two pipes to affect each other after
data had been transferred between them with tee():
============
$ cat tee_test.c
int main(void) {
int pipe_a[2];
if (pipe(pipe_a)) err(1, "pipe");
int pipe_b[2];
if (pipe(pipe_b)) err(1, "pipe");
if (write(pipe_a[1], "abcd", 4) != 4) err(1, "write");
if (tee(pipe_a[0], pipe_b[1], 2, 0) != 2) err(1, "tee");
if (write(pipe_b[1], "xx", 2) != 2) err(1, "write");
char buf[5];
if (read(pipe_a[0], buf, 4) != 4) err(1, "read");
buf[4] = 0;
printf("got back: '%s'\n", buf);
}
$ gcc -o tee_test tee_test.c
$ ./tee_test
got back: 'abxx'
$
============
As suggested by Al Viro, fix it by creating a separate type for
non-mergeable pipe buffers, then changing the types of buffers in
splice_pipe_to_pipe() and link_pipe().
Cc: <stable@vger.kernel.org>
Fixes: 7c77f0b3f920 ("splice: implement pipe to pipe splicing")
Fixes: 70524490ee2e ("[PATCH] splice: add support for sys_tee()")
Suggested-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2019-01-23 14:19:17 +00:00
|
|
|
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
obuf->len = len;
|
2019-11-15 13:30:32 +00:00
|
|
|
ibuf->offset += len;
|
|
|
|
ibuf->len -= len;
|
|
|
|
o_len = len;
|
|
|
|
o_head++;
|
|
|
|
opipe->head = o_head;
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
}
|
2019-11-15 13:30:32 +00:00
|
|
|
ret += o_len;
|
|
|
|
len -= o_len;
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
} while (len);
|
|
|
|
|
|
|
|
pipe_unlock(ipipe);
|
|
|
|
pipe_unlock(opipe);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we put data in the output pipe, wakeup any potential readers.
|
|
|
|
*/
|
2011-05-23 17:58:53 +00:00
|
|
|
if (ret > 0)
|
|
|
|
wakeup_pipe_readers(opipe);
|
|
|
|
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
if (input_wakeup)
|
|
|
|
wakeup_pipe_writers(ipipe);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2006-04-11 13:51:17 +00:00
|
|
|
/*
|
|
|
|
* Link contents of ipipe to opipe.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
static ssize_t link_pipe(struct pipe_inode_info *ipipe,
|
|
|
|
struct pipe_inode_info *opipe,
|
|
|
|
size_t len, unsigned int flags)
|
2006-04-11 13:51:17 +00:00
|
|
|
{
|
|
|
|
struct pipe_buffer *ibuf, *obuf;
|
2019-11-15 13:30:32 +00:00
|
|
|
unsigned int i_head, o_head;
|
|
|
|
unsigned int i_tail, o_tail;
|
|
|
|
unsigned int i_mask, o_mask;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret = 0;
|
2006-04-11 13:51:17 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Potential ABBA deadlock, work around it by ordering lock
|
2009-04-14 17:48:41 +00:00
|
|
|
* grabbing by pipe info address. Otherwise two different processes
|
2006-04-11 13:51:17 +00:00
|
|
|
* could deadlock (one doing tee from A -> B, the other from B -> A).
|
|
|
|
*/
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_double_lock(ipipe, opipe);
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
i_tail = ipipe->tail;
|
|
|
|
i_mask = ipipe->ring_size - 1;
|
|
|
|
o_head = opipe->head;
|
|
|
|
o_mask = opipe->ring_size - 1;
|
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
do {
|
2006-04-11 13:51:17 +00:00
|
|
|
if (!opipe->readers) {
|
|
|
|
send_sig(SIGPIPE, current, 0);
|
|
|
|
if (!ret)
|
|
|
|
ret = -EPIPE;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
i_head = ipipe->head;
|
|
|
|
o_tail = opipe->tail;
|
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
/*
|
2019-11-15 13:30:32 +00:00
|
|
|
* If we have iterated all input buffers or run out of
|
2006-07-10 09:00:01 +00:00
|
|
|
* output room, break.
|
|
|
|
*/
|
2019-11-15 13:30:32 +00:00
|
|
|
if (pipe_empty(i_head, i_tail) ||
|
2019-10-16 15:47:32 +00:00
|
|
|
pipe_full(o_head, o_tail, opipe->max_usage))
|
2006-07-10 09:00:01 +00:00
|
|
|
break;
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2019-11-15 13:30:32 +00:00
|
|
|
ibuf = &ipipe->bufs[i_tail & i_mask];
|
|
|
|
obuf = &opipe->bufs[o_head & o_mask];
|
2006-04-11 13:51:17 +00:00
|
|
|
|
|
|
|
/*
|
2006-07-10 09:00:01 +00:00
|
|
|
* Get a reference to this pipe buffer,
|
|
|
|
* so we can copy the contents over.
|
2006-04-11 13:51:17 +00:00
|
|
|
*/
|
2019-04-05 21:02:10 +00:00
|
|
|
if (!pipe_buf_get(ipipe, ibuf)) {
|
|
|
|
if (ret == 0)
|
|
|
|
ret = -EFAULT;
|
|
|
|
break;
|
|
|
|
}
|
2006-07-10 09:00:01 +00:00
|
|
|
|
|
|
|
*obuf = *ibuf;
|
|
|
|
|
2006-04-19 13:56:40 +00:00
|
|
|
/*
|
2020-05-20 15:58:12 +00:00
|
|
|
* Don't inherit the gift and merge flag, we need to prevent
|
|
|
|
* multiple steals of this page.
|
2006-04-19 13:56:40 +00:00
|
|
|
*/
|
2006-07-10 09:00:01 +00:00
|
|
|
obuf->flags &= ~PIPE_BUF_FLAG_GIFT;
|
2020-05-20 15:58:12 +00:00
|
|
|
obuf->flags &= ~PIPE_BUF_FLAG_CAN_MERGE;
|
splice: don't merge into linked buffers
Before this patch, it was possible for two pipes to affect each other after
data had been transferred between them with tee():
============
$ cat tee_test.c
int main(void) {
int pipe_a[2];
if (pipe(pipe_a)) err(1, "pipe");
int pipe_b[2];
if (pipe(pipe_b)) err(1, "pipe");
if (write(pipe_a[1], "abcd", 4) != 4) err(1, "write");
if (tee(pipe_a[0], pipe_b[1], 2, 0) != 2) err(1, "tee");
if (write(pipe_b[1], "xx", 2) != 2) err(1, "write");
char buf[5];
if (read(pipe_a[0], buf, 4) != 4) err(1, "read");
buf[4] = 0;
printf("got back: '%s'\n", buf);
}
$ gcc -o tee_test tee_test.c
$ ./tee_test
got back: 'abxx'
$
============
As suggested by Al Viro, fix it by creating a separate type for
non-mergeable pipe buffers, then changing the types of buffers in
splice_pipe_to_pipe() and link_pipe().
Cc: <stable@vger.kernel.org>
Fixes: 7c77f0b3f920 ("splice: implement pipe to pipe splicing")
Fixes: 70524490ee2e ("[PATCH] splice: add support for sys_tee()")
Suggested-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2019-01-23 14:19:17 +00:00
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
if (obuf->len > len)
|
|
|
|
obuf->len = len;
|
|
|
|
ret += obuf->len;
|
|
|
|
len -= obuf->len;
|
2019-11-15 13:30:32 +00:00
|
|
|
|
|
|
|
o_head++;
|
|
|
|
opipe->head = o_head;
|
|
|
|
i_tail++;
|
2006-07-10 09:00:01 +00:00
|
|
|
} while (len);
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2009-04-14 17:48:41 +00:00
|
|
|
pipe_unlock(ipipe);
|
|
|
|
pipe_unlock(opipe);
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
/*
|
|
|
|
* If we put data in the output pipe, wakeup any potential readers.
|
|
|
|
*/
|
2011-05-23 17:58:53 +00:00
|
|
|
if (ret > 0)
|
|
|
|
wakeup_pipe_readers(opipe);
|
2006-04-11 13:51:17 +00:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is a tee(1) implementation that works on pipes. It doesn't copy
|
|
|
|
* any data, it simply references the 'in' pages on the 'out' pipe.
|
|
|
|
* The 'flags' used are the SPLICE_F_* variants, currently the only
|
|
|
|
* applicable one is SPLICE_F_NONBLOCK.
|
|
|
|
*/
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t do_tee(struct file *in, struct file *out, size_t len,
|
|
|
|
unsigned int flags)
|
2006-04-11 13:51:17 +00:00
|
|
|
{
|
pipe: Add general notification queue support
Make it possible to have a general notification queue built on top of a
standard pipe. Notifications are 'spliced' into the pipe and then read
out. splice(), vmsplice() and sendfile() are forbidden on pipes used for
notifications as post_one_notification() cannot take pipe->mutex. This
means that notifications could be posted in between individual pipe
buffers, making iov_iter_revert() difficult to effect.
The way the notification queue is used is:
(1) An application opens a pipe with a special flag and indicates the
number of messages it wishes to be able to queue at once (this can
only be set once):
pipe2(fds, O_NOTIFICATION_PIPE);
ioctl(fds[0], IOC_WATCH_QUEUE_SET_SIZE, queue_depth);
(2) The application then uses poll() and read() as normal to extract data
from the pipe. read() will return multiple notifications if the
buffer is big enough, but it will not split a notification across
buffers - rather it will return a short read or EMSGSIZE.
Notification messages include a length in the header so that the
caller can split them up.
Each message has a header that describes it:
struct watch_notification {
__u32 type:24;
__u32 subtype:8;
__u32 info;
};
The type indicates the source (eg. mount tree changes, superblock events,
keyring changes, block layer events) and the subtype indicates the event
type (eg. mount, unmount; EIO, EDQUOT; link, unlink). The info field
indicates a number of things, including the entry length, an ID assigned to
a watchpoint contributing to this buffer and type-specific flags.
Supplementary data, such as the key ID that generated an event, can be
attached in additional slots. The maximum message size is 127 bytes.
Messages may not be padded or aligned, so there is no guarantee, for
example, that the notification type will be on a 4-byte bounary.
Signed-off-by: David Howells <dhowells@redhat.com>
2020-01-14 17:07:11 +00:00
|
|
|
struct pipe_inode_info *ipipe = get_pipe_info(in, true);
|
|
|
|
struct pipe_inode_info *opipe = get_pipe_info(out, true);
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t ret = -EINVAL;
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2020-05-04 19:39:35 +00:00
|
|
|
if (unlikely(!(in->f_mode & FMODE_READ) ||
|
|
|
|
!(out->f_mode & FMODE_WRITE)))
|
|
|
|
return -EBADF;
|
|
|
|
|
2006-04-11 13:51:17 +00:00
|
|
|
/*
|
2006-07-10 09:00:01 +00:00
|
|
|
* Duplicate the contents of ipipe to opipe without actually
|
|
|
|
* copying the data.
|
2006-04-11 13:51:17 +00:00
|
|
|
*/
|
2006-07-10 09:00:01 +00:00
|
|
|
if (ipipe && opipe && ipipe != opipe) {
|
2019-02-07 15:45:19 +00:00
|
|
|
if ((in->f_flags | out->f_flags) & O_NONBLOCK)
|
|
|
|
flags |= SPLICE_F_NONBLOCK;
|
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
/*
|
|
|
|
* Keep going, unless we encounter an error. The ipipe/opipe
|
|
|
|
* ordering doesn't really matter.
|
|
|
|
*/
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
ret = ipipe_prep(ipipe, flags);
|
2006-07-10 09:00:01 +00:00
|
|
|
if (!ret) {
|
splice: implement pipe to pipe splicing
Allow splice(2) to work when both the input and the output is a pipe.
Based on the impementation of the tee(2) syscall, but instead of
duplicating the buffer references move the buffers from the input pipe
to the output pipe.
Moving the whole buffer only succeeds if the full length of the buffer
is spliced. Otherwise duplicate the buffer, just like tee(2), set the
length of the output buffer and advance the offset on the input
buffer.
Since splice is operating on two pipes, special care needs to be taken
with locking to prevent AN ABBA deadlock. Again this is done
similarly to the tee(2) syscall, first preparing the input and output
pipes so there's data to consume and space for that data, and then
doing the move operation while holding both locks.
If other processes are doing I/O on the same pipes parallel to the
splice, then by the time both inodes are locked there might be no
buffers left to move, or no space to move them to. In this case retry
the whole operation, including the preparation phase. This could lead
to starvation, but I'm not sure if that's serious enough to worry
about.
Signed-off-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
2009-05-07 13:37:35 +00:00
|
|
|
ret = opipe_prep(opipe, flags);
|
2008-02-20 09:34:51 +00:00
|
|
|
if (!ret)
|
2006-07-10 09:00:01 +00:00
|
|
|
ret = link_pipe(ipipe, opipe, len, flags);
|
|
|
|
}
|
|
|
|
}
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2023-07-03 14:42:21 +00:00
|
|
|
if (ret > 0) {
|
|
|
|
fsnotify_access(in);
|
|
|
|
fsnotify_modify(out);
|
|
|
|
}
|
|
|
|
|
2006-07-10 09:00:01 +00:00
|
|
|
return ret;
|
2006-04-11 13:51:17 +00:00
|
|
|
}
|
|
|
|
|
2009-01-14 13:14:33 +00:00
|
|
|
SYSCALL_DEFINE4(tee, int, fdin, int, fdout, size_t, len, unsigned int, flags)
|
2006-04-11 13:51:17 +00:00
|
|
|
{
|
2020-05-04 19:39:35 +00:00
|
|
|
struct fd in, out;
|
2023-12-12 09:44:36 +00:00
|
|
|
ssize_t error;
|
2006-04-11 13:51:17 +00:00
|
|
|
|
2016-12-10 18:17:32 +00:00
|
|
|
if (unlikely(flags & ~SPLICE_F_ALL))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2006-04-11 13:51:17 +00:00
|
|
|
if (unlikely(!len))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
error = -EBADF;
|
2012-08-28 16:52:22 +00:00
|
|
|
in = fdget(fdin);
|
|
|
|
if (in.file) {
|
2020-05-04 19:39:35 +00:00
|
|
|
out = fdget(fdout);
|
|
|
|
if (out.file) {
|
|
|
|
error = do_tee(in.file, out.file, len, flags);
|
|
|
|
fdput(out);
|
2006-04-11 13:51:17 +00:00
|
|
|
}
|
2012-08-28 16:52:22 +00:00
|
|
|
fdput(in);
|
2006-04-11 13:51:17 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return error;
|
|
|
|
}
|