2005-01-05 22:34:37 +00:00
|
|
|
/*-
|
1998-09-15 06:33:23 +00:00
|
|
|
* Data structures and definitions for CAM peripheral ("type") drivers.
|
|
|
|
|
*
|
2023-05-10 15:40:58 +00:00
|
|
|
* SPDX-License-Identifier: BSD-2-Clause
|
2017-11-27 15:12:43 +00:00
|
|
|
*
|
1998-09-15 06:33:23 +00:00
|
|
|
* Copyright (c) 1997, 1998 Justin T. Gibbs.
|
|
|
|
|
* All rights reserved.
|
|
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions, and the following disclaimer,
|
|
|
|
|
* without modification, immediately at the beginning of the file.
|
|
|
|
|
* 2. The name of the author may not be used to endorse or promote products
|
|
|
|
|
* derived from this software without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
|
|
|
|
|
* ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _CAM_CAM_PERIPH_H
|
|
|
|
|
#define _CAM_CAM_PERIPH_H 1
|
|
|
|
|
|
|
|
|
|
#include <sys/queue.h>
|
2007-04-19 22:46:26 +00:00
|
|
|
#include <cam/cam_sim.h>
|
1998-09-15 06:33:23 +00:00
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
Extract eventfilter declarations to sys/_eventfilter.h
This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.
EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).
As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.
LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).
No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.
2019-05-20 00:38:23 +00:00
|
|
|
#include <sys/lock.h>
|
|
|
|
|
#include <sys/mutex.h>
|
2018-03-14 17:53:37 +00:00
|
|
|
#include <sys/sysctl.h>
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
#include <sys/taskqueue.h>
|
1998-09-15 06:33:23 +00:00
|
|
|
|
cam: allocate CCBs from UMA for SCSI and ATA IO
This patch makes it possible for CAM to use small CCBs allocated
from an periph-specific UMA zone instead of the usual, huge ones.
The end result is that CCBs issued via da(4) take 544B (size of
ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB,
rounded up by malloc(9)). For ATA it's 272B. We waste less
memory, we avoid zeroing the unused 1kB, and it should be easier
to allocate those CCBs in low memory conditions. It should also
be possible to use uma_zone_reserve(9) to improve behaviour
in low memory conditions even further.
Note that this does not change the size, or the layout, of CCBs
as such. CCBs get allocated in various different ways, in particular
on the stack, and I don't want to redo all that. Instead, this
provides an opt-in mechanism for the periph to declare "my start()
callback is fine with receiving a CCB allocated from this UMA zone".
In other words, most of the code works exactly as it used to; the
change only happens to IOs issued by xpt_run_allockq(), which
is - conveniently - pretty much all that matters for performance.
The reason for doing it this way is that it's pretty small, localized
change, and can be implemented gradually and iteratively: take a
periph, make sure its start() callback only casts the CCBs it takes
to a particular type of CCB, for example ccb_scsiio, and that it only
casts CCBs returned by cam_periph_getccb() to that type, then add UMA
zone for that size, and declare it safe to XPT.
This is disabled by default. Set 'kern.cam.ada.enable_uma_ccbs=1'
and 'kern.cam.da.enable_uma_ccbs=1' tunables to enable it. Testing
is welcome; I will flip the default to enable in two weeks from now.
Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28674
2021-05-15 10:17:22 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
|
|
2013-03-29 11:53:20 +00:00
|
|
|
#include <cam/cam_xpt.h>
|
|
|
|
|
|
2003-03-08 08:01:31 +00:00
|
|
|
struct devstat;
|
|
|
|
|
|
2000-05-07 18:04:50 +00:00
|
|
|
extern struct cam_periph *xpt_periph;
|
2000-04-03 11:11:12 +00:00
|
|
|
|
2001-02-07 07:05:59 +00:00
|
|
|
extern struct periph_driver **periph_drivers;
|
|
|
|
|
void periphdriver_register(void *);
|
2017-03-07 17:41:08 +00:00
|
|
|
int periphdriver_unregister(void *);
|
MFp4: Large set of CAM inprovements.
- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.
2010-01-28 08:41:30 +00:00
|
|
|
void periphdriver_init(int level);
|
2001-02-07 07:05:59 +00:00
|
|
|
|
|
|
|
|
#include <sys/module.h>
|
|
|
|
|
#define PERIPHDRIVER_DECLARE(name, driver) \
|
|
|
|
|
static int name ## _modevent(module_t mod, int type, void *data) \
|
|
|
|
|
{ \
|
|
|
|
|
switch (type) { \
|
|
|
|
|
case MOD_LOAD: \
|
|
|
|
|
periphdriver_register(data); \
|
|
|
|
|
break; \
|
|
|
|
|
case MOD_UNLOAD: \
|
2017-03-07 17:41:08 +00:00
|
|
|
return (periphdriver_unregister(data)); \
|
2004-07-15 08:26:07 +00:00
|
|
|
default: \
|
|
|
|
|
return EOPNOTSUPP; \
|
2001-02-07 07:05:59 +00:00
|
|
|
} \
|
|
|
|
|
return 0; \
|
|
|
|
|
} \
|
|
|
|
|
static moduledata_t name ## _mod = { \
|
|
|
|
|
#name, \
|
|
|
|
|
name ## _modevent, \
|
|
|
|
|
(void *)&driver \
|
|
|
|
|
}; \
|
|
|
|
|
DECLARE_MODULE(name, name ## _mod, SI_SUB_DRIVERS, SI_ORDER_ANY); \
|
|
|
|
|
MODULE_DEPEND(name, cam, 1, 1, 1)
|
1998-09-15 06:33:23 +00:00
|
|
|
|
2017-03-07 17:41:08 +00:00
|
|
|
/*
|
|
|
|
|
* Callback informing the peripheral driver it can perform it's
|
|
|
|
|
* initialization since the XPT is now fully initialized.
|
|
|
|
|
*/
|
|
|
|
|
typedef void (periph_init_t)(void);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Callback requesting the peripheral driver to remove its instances
|
|
|
|
|
* and shutdown, if possible.
|
|
|
|
|
*/
|
|
|
|
|
typedef int (periph_deinit_t)(void);
|
1998-09-15 06:33:23 +00:00
|
|
|
|
|
|
|
|
struct periph_driver {
|
2017-03-07 17:41:08 +00:00
|
|
|
periph_init_t *init;
|
|
|
|
|
char *driver_name;
|
2000-05-26 02:09:24 +00:00
|
|
|
TAILQ_HEAD(,cam_periph) units;
|
1998-09-15 06:33:23 +00:00
|
|
|
u_int generation;
|
2009-10-31 10:43:38 +00:00
|
|
|
u_int flags;
|
|
|
|
|
#define CAM_PERIPH_DRV_EARLY 0x01
|
2017-03-07 17:41:08 +00:00
|
|
|
periph_deinit_t *deinit;
|
1998-09-15 06:33:23 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
typedef enum {
|
2004-10-05 04:22:20 +00:00
|
|
|
CAM_PERIPH_BIO
|
1998-09-15 06:33:23 +00:00
|
|
|
} cam_periph_type;
|
|
|
|
|
|
2013-04-17 11:45:15 +00:00
|
|
|
/* Generically useful offsets into the peripheral private area */
|
1998-09-15 06:33:23 +00:00
|
|
|
#define ppriv_ptr0 periph_priv.entries[0].ptr
|
|
|
|
|
#define ppriv_ptr1 periph_priv.entries[1].ptr
|
|
|
|
|
#define ppriv_field0 periph_priv.entries[0].field
|
|
|
|
|
#define ppriv_field1 periph_priv.entries[1].field
|
|
|
|
|
|
|
|
|
|
typedef void periph_start_t (struct cam_periph *periph,
|
|
|
|
|
union ccb *start_ccb);
|
|
|
|
|
typedef cam_status periph_ctor_t (struct cam_periph *periph,
|
|
|
|
|
void *arg);
|
Fix a problem with the way we handled device invalidation when attaching
to a device failed.
In theory, the same steps that happen when we get an AC_LOST_DEVICE async
notification should have been taken when a driver fails to attach. In
practice, that wasn't the case.
This only affected the da, cd and ch drivers, but the fix affects all
peripheral drivers.
There were several possible problems:
- In the da driver, we didn't remove the peripheral's softc from the da
driver's linked list of softcs. Once the peripheral and softc got
removed, we'd get a kernel panic the next time the timeout routine
called dasendorderedtag().
- In the da, cd and possibly ch drivers, we didn't remove the
peripheral's devstat structure from the devstat queue. Once the
peripheral and softc were removed, this could cause a panic if anyone
tried to access device statistics. (one component of the linked list
wouldn't exist anymore)
- In the cd driver, we didn't take the peripheral off the changer run
queue if it was scheduled to run. In practice, it's highly unlikely,
and maybe impossible that the peripheral would have been on the
changer run queue at that stage of the probe process.
The fix is:
- Add a new peripheral callback function (the "oninvalidate" function)
that is called the first time cam_periph_invalidate() is called for a
peripheral.
- Create new foooninvalidate() routines for each peripheral driver. This
routine is always called at splsoftcam(), and contains all the stuff
that used to be in the AC_LOST_DEVICE case of the async callback
handler.
- Move the devstat cleanup call to the destructor/cleanup routines, since
some of the drivers do I/O in their close routines.
- Make sure that when we're flushing the buffer queue, we traverse it at
splbio().
- Add a check for the invalid flag in the pt driver's open routine.
Reviewed by: gibbs
1998-10-22 22:16:56 +00:00
|
|
|
typedef void periph_oninv_t (struct cam_periph *periph);
|
1998-09-15 06:33:23 +00:00
|
|
|
typedef void periph_dtor_t (struct cam_periph *periph);
|
|
|
|
|
struct cam_periph {
|
|
|
|
|
periph_start_t *periph_start;
|
Fix a problem with the way we handled device invalidation when attaching
to a device failed.
In theory, the same steps that happen when we get an AC_LOST_DEVICE async
notification should have been taken when a driver fails to attach. In
practice, that wasn't the case.
This only affected the da, cd and ch drivers, but the fix affects all
peripheral drivers.
There were several possible problems:
- In the da driver, we didn't remove the peripheral's softc from the da
driver's linked list of softcs. Once the peripheral and softc got
removed, we'd get a kernel panic the next time the timeout routine
called dasendorderedtag().
- In the da, cd and possibly ch drivers, we didn't remove the
peripheral's devstat structure from the devstat queue. Once the
peripheral and softc were removed, this could cause a panic if anyone
tried to access device statistics. (one component of the linked list
wouldn't exist anymore)
- In the cd driver, we didn't take the peripheral off the changer run
queue if it was scheduled to run. In practice, it's highly unlikely,
and maybe impossible that the peripheral would have been on the
changer run queue at that stage of the probe process.
The fix is:
- Add a new peripheral callback function (the "oninvalidate" function)
that is called the first time cam_periph_invalidate() is called for a
peripheral.
- Create new foooninvalidate() routines for each peripheral driver. This
routine is always called at splsoftcam(), and contains all the stuff
that used to be in the AC_LOST_DEVICE case of the async callback
handler.
- Move the devstat cleanup call to the destructor/cleanup routines, since
some of the drivers do I/O in their close routines.
- Make sure that when we're flushing the buffer queue, we traverse it at
splbio().
- Add a check for the invalid flag in the pt driver's open routine.
Reviewed by: gibbs
1998-10-22 22:16:56 +00:00
|
|
|
periph_oninv_t *periph_oninval;
|
1998-09-15 06:33:23 +00:00
|
|
|
periph_dtor_t *periph_dtor;
|
|
|
|
|
char *periph_name;
|
|
|
|
|
struct cam_path *path; /* Compiled path to device */
|
|
|
|
|
void *softc;
|
2007-04-15 08:49:19 +00:00
|
|
|
struct cam_sim *sim;
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t unit_number;
|
1998-09-15 06:33:23 +00:00
|
|
|
cam_periph_type type;
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t flags;
|
1998-09-15 06:33:23 +00:00
|
|
|
#define CAM_PERIPH_RUNNING 0x01
|
|
|
|
|
#define CAM_PERIPH_LOCKED 0x02
|
|
|
|
|
#define CAM_PERIPH_LOCK_WANTED 0x04
|
|
|
|
|
#define CAM_PERIPH_INVALID 0x08
|
|
|
|
|
#define CAM_PERIPH_NEW_DEV_FOUND 0x10
|
Fix a bug in the error recovery code. It was possible to have more than
one error recovery action oustanding for a given peripheral.
This is bad for several reasons. The first problem is that the error
recovery actions would likely be to fix the same problem. (e.g., we
queue 5 CCBs to a disk, and the first one comes back with 0x04,0x02. We
start error recovery, and the second one comes back with the same status.
Then the third one comes back, and so on. Each one causes the drive to get
nailed with a start unit, when we really only need one.)
The other problem is that we only have space to store one CCB while we're
doing error recovery. The subsequent error recovery actions that got
started were over-writing the CCBs from previous error recovery actions,
but we still tried to call the done routine N times for N error recovery
actions. Each call to dadone() was done with the same CCB, though. So on
the second one, we got a "biodone: buffer not busy" panic, since the buffer
in question had already been through biodone().
In any case, this fixes things so that any any given time, there's only one
error recovery action outstanding for any given peripheral driver.
Reviewed by: gibbs
Reported by: Philippe Regnauld <regnauld@deepo.prosa.dk>
[ Philippe wins the "bug finder of the week" award ]
1998-10-13 21:41:32 +00:00
|
|
|
#define CAM_PERIPH_RECOVERY_INPROG 0x20
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
#define CAM_PERIPH_RUN_TASK 0x40
|
Fix a race condition in CAM peripheral free handling, locking
in the CAM XPT bus traversal code, and a number of other periph level
issues.
cam_periph.h,
cam_periph.c: Modify cam_periph_acquire() to test the CAM_PERIPH_INVALID
flag prior to allowing a reference count to be gained
on a peripheral. Callers of this function will receive
CAM_REQ_CMP_ERR status in the situation of attempting to
reference an invalidated periph. This guarantees that
a peripheral scheduled for a deferred free will not
be accessed during its wait for destruction.
Panic during attempts to drop a reference count on
a peripheral that already has a zero reference count.
In cam_periph_list(), use a local sbuf with SBUF_FIXEDLEN
set so that mallocs do not occur while the xpt topology
lock is held, regardless of the allocation policy of the
passed in sbuf.
Add a new routine, cam_periph_release_locked_buses(),
that can be called when the caller already holds
the CAM topology lock.
Add some extra debugging for duplicate peripheral
allocations in cam_periph_alloc().
Treat CAM_DEV_NOT_THERE much the same as a selection
timeout (AC_LOST_DEVICE is emitted), but forgo retries.
cam_xpt.c: Revamp the way the EDT traversal code does locking
and reference counting. This was broken, since it
assumed that the EDT would not change during
traversal, but that assumption is no longer valid.
So, to prevent devices from going away while we
traverse the EDT, make sure we properly lock
everything and hold references on devices that
we are using.
The two peripheral driver traversal routines should
be examined. xptpdperiphtraverse() holds the
topology lock for the entire time it runs.
xptperiphtraverse() is now locked properly, but
only holds the topology lock while it is traversing
the list, and not while the traversal function is
running.
The bus locking code in xptbustraverse() should
also be revisited at a later time, since it is
complex and should probably be simplified.
scsi_da.c: Pay attention to the return value from cam_periph_acquire().
Return 0 always from daclose() even if the disk is now gone.
Add some rudimentary error injection support.
scsi_sg.c: Fix reference counting in the sg(4) driver.
The sg driver was calling cam_periph_release() on close,
but never called cam_periph_acquire() (which increments
the reference count) on open.
The periph code correctly complained that the sg(4)
driver was trying to decrement the refcount when it
was already 0.
Sponsored by: Spectra Logic
MFC after: 2 weeks
2012-01-12 00:41:48 +00:00
|
|
|
#define CAM_PERIPH_FREE 0x80
|
2013-10-15 17:59:41 +00:00
|
|
|
#define CAM_PERIPH_ANNOUNCED 0x100
|
2019-11-22 21:31:59 +00:00
|
|
|
#define CAM_PERIPH_RECOVERY_WAIT 0x200
|
|
|
|
|
#define CAM_PERIPH_RECOVERY_WAIT_FAILED 0x400
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
uint32_t scheduled_priority;
|
|
|
|
|
uint32_t immediate_priority;
|
|
|
|
|
int periph_allocating;
|
|
|
|
|
int periph_allocated;
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t refcount;
|
2000-05-26 02:09:24 +00:00
|
|
|
SLIST_HEAD(, ccb_hdr) ccb_list; /* For "immediate" requests */
|
|
|
|
|
SLIST_ENTRY(cam_periph) periph_links;
|
|
|
|
|
TAILQ_ENTRY(cam_periph) unit_links;
|
1998-09-15 06:33:23 +00:00
|
|
|
ac_callback_t *deferred_callback;
|
|
|
|
|
ac_code deferred_ac;
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
struct task periph_run_task;
|
cam: allocate CCBs from UMA for SCSI and ATA IO
This patch makes it possible for CAM to use small CCBs allocated
from an periph-specific UMA zone instead of the usual, huge ones.
The end result is that CCBs issued via da(4) take 544B (size of
ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB,
rounded up by malloc(9)). For ATA it's 272B. We waste less
memory, we avoid zeroing the unused 1kB, and it should be easier
to allocate those CCBs in low memory conditions. It should also
be possible to use uma_zone_reserve(9) to improve behaviour
in low memory conditions even further.
Note that this does not change the size, or the layout, of CCBs
as such. CCBs get allocated in various different ways, in particular
on the stack, and I don't want to redo all that. Instead, this
provides an opt-in mechanism for the periph to declare "my start()
callback is fine with receiving a CCB allocated from this UMA zone".
In other words, most of the code works exactly as it used to; the
change only happens to IOs issued by xpt_run_allockq(), which
is - conveniently - pretty much all that matters for performance.
The reason for doing it this way is that it's pretty small, localized
change, and can be implemented gradually and iteratively: take a
periph, make sure its start() callback only casts the CCBs it takes
to a particular type of CCB, for example ccb_scsiio, and that it only
casts CCBs returned by cam_periph_getccb() to that type, then add UMA
zone for that size, and declare it safe to XPT.
This is disabled by default. Set 'kern.cam.ada.enable_uma_ccbs=1'
and 'kern.cam.da.enable_uma_ccbs=1' tunables to enable it. Testing
is welcome; I will flip the default to enable in two weeks from now.
Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28674
2021-05-15 10:17:22 +00:00
|
|
|
uma_zone_t ccb_zone;
|
Delay GEOM disk_create() until CAM periph probe completes.
Before this patch CAM periph drivers called both disk_alloc() and
disk_create() same time on periph creation. But then prevented disks
from opening until the periph probe completion with cam_periph_hold().
As result, especially if disk misbehaves during the probe, GEOM event
thread, triggered to taste the disk, got blocked on open attempt,
potentially for a long time, unable to process other events.
This patch moves disk_create() call from periph creation to the end of
the probe. To allow disk_create() calls from non-sleepable CAM contexts
some of its duties requiring memory allocations are moved either back
to disk_alloc() or forward to g_disk_create(), so now disk_alloc() and
disk_add_alias() are the only disk methods that require sleeping. If
disk fails during the probe disk_create() may just be skipped, going
directly to disk_destroy(). Other method calls during that time are
just ignored. Since GEOM may now see the disks after CAM bus scan is
already completed, introduce per-periph boot hold functions. Enclosure
driver already had such mechanism, so just generalize it.
Reviewed by: imp
MFC after: 1 month
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D35784
2022-07-14 19:38:14 +00:00
|
|
|
struct root_hold_token periph_rootmount;
|
1998-09-15 06:33:23 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
|
|
#define CAM_PERIPH_MAXMAPS 2
|
|
|
|
|
|
|
|
|
|
struct cam_periph_map_info {
|
|
|
|
|
int num_bufs_used;
|
2019-03-28 20:41:02 +00:00
|
|
|
void *orig[CAM_PERIPH_MAXMAPS];
|
1998-09-15 06:33:23 +00:00
|
|
|
struct buf *bp[CAM_PERIPH_MAXMAPS];
|
|
|
|
|
};
|
|
|
|
|
|
Fix a problem with the way we handled device invalidation when attaching
to a device failed.
In theory, the same steps that happen when we get an AC_LOST_DEVICE async
notification should have been taken when a driver fails to attach. In
practice, that wasn't the case.
This only affected the da, cd and ch drivers, but the fix affects all
peripheral drivers.
There were several possible problems:
- In the da driver, we didn't remove the peripheral's softc from the da
driver's linked list of softcs. Once the peripheral and softc got
removed, we'd get a kernel panic the next time the timeout routine
called dasendorderedtag().
- In the da, cd and possibly ch drivers, we didn't remove the
peripheral's devstat structure from the devstat queue. Once the
peripheral and softc were removed, this could cause a panic if anyone
tried to access device statistics. (one component of the linked list
wouldn't exist anymore)
- In the cd driver, we didn't take the peripheral off the changer run
queue if it was scheduled to run. In practice, it's highly unlikely,
and maybe impossible that the peripheral would have been on the
changer run queue at that stage of the probe process.
The fix is:
- Add a new peripheral callback function (the "oninvalidate" function)
that is called the first time cam_periph_invalidate() is called for a
peripheral.
- Create new foooninvalidate() routines for each peripheral driver. This
routine is always called at splsoftcam(), and contains all the stuff
that used to be in the AC_LOST_DEVICE case of the async callback
handler.
- Move the devstat cleanup call to the destructor/cleanup routines, since
some of the drivers do I/O in their close routines.
- Make sure that when we're flushing the buffer queue, we traverse it at
splbio().
- Add a check for the invalid flag in the pt driver's open routine.
Reviewed by: gibbs
1998-10-22 22:16:56 +00:00
|
|
|
cam_status cam_periph_alloc(periph_ctor_t *periph_ctor,
|
|
|
|
|
periph_oninv_t *periph_oninvalidate,
|
|
|
|
|
periph_dtor_t *periph_dtor,
|
|
|
|
|
periph_start_t *periph_start,
|
2000-03-15 21:55:48 +00:00
|
|
|
char *name, cam_periph_type type, struct cam_path *,
|
|
|
|
|
ac_callback_t *, ac_code, void *arg);
|
1998-09-15 06:33:23 +00:00
|
|
|
struct cam_periph *cam_periph_find(struct cam_path *path, char *name);
|
Lay groundwork in CAM for recording and reporting physical path and
other device attributes stored in the CAM Existing Device Table (EDT).
This includes some infrastructure requried by the enclosure services
driver to export physical path information.
Make the CAM device advanced info interface accept store requests.
sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
- Replace scsi_get_sas_addr() with a scsi_get_devid() which takes
a callback that decides whether to accept a particular descriptor.
Provide callbacks for NAA IEEE Registered addresses and for SAS
addresses, replacing the old function. This is needed because
the old function doesn't work for an enclosure address for a SAS
device, which is not flagged as a SAS address, but is NAA IEEE
Registered. It may be worthwhile merging this interface with the
devid match interface.
- Add a few more defines for some device ID fields.
sbin/camcontrol/camcontrol.c:
- Update for the CCB_DEV_ADVINFO interface change.
cam/cam_xpt_internal.h:
- Add the new fields for the physical path string to the CAM EDT.
cam/cam_ccb.h:
- Rename CCB_GDEV_ADVINFO to simply CCB_DEV_ADVINFO, and the ccb
structure to ccb_dev_advinfo.
- Add a flag that changes this CCB's action to store, rather than
the default, retrieve.
- Add a new buffer type, CDAI_TYPE_PHYS_PATH, for the new CAM EDT
physpath field.
- Remove the never-implemented transport & proto flags.
cam/cam_xpt.c:
cam/cam_xpt.h:
- Add xpt_getattr(), which provides a wrapper for fetching a device's
attribute using the GEOM strings as key. This method currently
supports "GEOM::ident" and "GEOM::physpath".
Submitted by: will
Reviewed by : gibbs
Extend the XPT_DEV_MATCH api to allow a device search by device ID.
As far as the API is concerned, device ID is a binary blob to be
interpreted by the transport layer. The SCSI implementation assumes
it is an array of VPD device ID descriptors.
sys/cam/cam_ccb.h:
Create a new structure, device_id_match_pattern, and
update the XPT_DEV_MATCH datastructures and flags so
that this pattern type can be used.
sys/cam/cam_xpt.c:
- A single pattern matching on both inquiry data and device
ID is invalid. Report any violators.
- Pass device ID match requests through to the new routine
scsi_devid_match(). The direct call of a SCSI routine is
a layering violation, but no worse than the one a few
lines up that checks inquiry data. Defer cleaning this
up until our future, larger, rototilling of CAM.
- Zero out cam_ed and cam_et nodes on allocation. Prior to
this change, device_id_len and device_id were not inialized,
preventing proper detection of the presence of this
information.
sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add the scsi_match_devid() routine.
Add a helper function for extracting peripherial driver names
sys/cam/cam_periph.c:
sys/cam/cam_periph.h:
Add the cam_periph_list() method which fills an sbuf
with a comma delimited list of the peripheral instances
associated with a given CAM path.
Add a helper functions for SCSI commands used by the SES driver.
sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add structure definitions and csio filling functions for
the receive diagnostic results and send diagnostic commands.
Misc CAM XPT cleanups.
sys/cam/cam_xpt.c:
Broadcast AC_FOUND_DEVICE and AC_PATH_REGISTERED
events at the time async event handlers are attached
even when registering just for events on a partitular
SIM. Previously, you had to register for these
events on all SIMs in the system in order to get
the initial broadcast even though subsequent device
and path arrivals would be delivered.
sys/cam/cam_xpt.c:
Remove SIM mutex held asserts from path accessors.
CAM paths are reference counted and it is this
reference count, not the sim mutex, that garantees
they are stable.
Sponsored by: Spectra Logic Corporation
2011-06-14 14:53:17 +00:00
|
|
|
int cam_periph_list(struct cam_path *, struct sbuf *);
|
2018-02-06 06:42:25 +00:00
|
|
|
int cam_periph_acquire(struct cam_periph *periph);
|
2014-01-11 13:35:36 +00:00
|
|
|
void cam_periph_doacquire(struct cam_periph *periph);
|
1998-09-15 06:33:23 +00:00
|
|
|
void cam_periph_release(struct cam_periph *periph);
|
2008-12-19 14:31:40 +00:00
|
|
|
void cam_periph_release_locked(struct cam_periph *periph);
|
Fix a race condition in CAM peripheral free handling, locking
in the CAM XPT bus traversal code, and a number of other periph level
issues.
cam_periph.h,
cam_periph.c: Modify cam_periph_acquire() to test the CAM_PERIPH_INVALID
flag prior to allowing a reference count to be gained
on a peripheral. Callers of this function will receive
CAM_REQ_CMP_ERR status in the situation of attempting to
reference an invalidated periph. This guarantees that
a peripheral scheduled for a deferred free will not
be accessed during its wait for destruction.
Panic during attempts to drop a reference count on
a peripheral that already has a zero reference count.
In cam_periph_list(), use a local sbuf with SBUF_FIXEDLEN
set so that mallocs do not occur while the xpt topology
lock is held, regardless of the allocation policy of the
passed in sbuf.
Add a new routine, cam_periph_release_locked_buses(),
that can be called when the caller already holds
the CAM topology lock.
Add some extra debugging for duplicate peripheral
allocations in cam_periph_alloc().
Treat CAM_DEV_NOT_THERE much the same as a selection
timeout (AC_LOST_DEVICE is emitted), but forgo retries.
cam_xpt.c: Revamp the way the EDT traversal code does locking
and reference counting. This was broken, since it
assumed that the EDT would not change during
traversal, but that assumption is no longer valid.
So, to prevent devices from going away while we
traverse the EDT, make sure we properly lock
everything and hold references on devices that
we are using.
The two peripheral driver traversal routines should
be examined. xptpdperiphtraverse() holds the
topology lock for the entire time it runs.
xptperiphtraverse() is now locked properly, but
only holds the topology lock while it is traversing
the list, and not while the traversal function is
running.
The bus locking code in xptbustraverse() should
also be revisited at a later time, since it is
complex and should probably be simplified.
scsi_da.c: Pay attention to the return value from cam_periph_acquire().
Return 0 always from daclose() even if the disk is now gone.
Add some rudimentary error injection support.
scsi_sg.c: Fix reference counting in the sg(4) driver.
The sg driver was calling cam_periph_release() on close,
but never called cam_periph_acquire() (which increments
the reference count) on open.
The periph code correctly complained that the sg(4)
driver was trying to decrement the refcount when it
was already 0.
Sponsored by: Spectra Logic
MFC after: 2 weeks
2012-01-12 00:41:48 +00:00
|
|
|
void cam_periph_release_locked_buses(struct cam_periph *periph);
|
2007-04-15 08:49:19 +00:00
|
|
|
int cam_periph_hold(struct cam_periph *periph, int priority);
|
|
|
|
|
void cam_periph_unhold(struct cam_periph *periph);
|
Delay GEOM disk_create() until CAM periph probe completes.
Before this patch CAM periph drivers called both disk_alloc() and
disk_create() same time on periph creation. But then prevented disks
from opening until the periph probe completion with cam_periph_hold().
As result, especially if disk misbehaves during the probe, GEOM event
thread, triggered to taste the disk, got blocked on open attempt,
potentially for a long time, unable to process other events.
This patch moves disk_create() call from periph creation to the end of
the probe. To allow disk_create() calls from non-sleepable CAM contexts
some of its duties requiring memory allocations are moved either back
to disk_alloc() or forward to g_disk_create(), so now disk_alloc() and
disk_add_alias() are the only disk methods that require sleeping. If
disk fails during the probe disk_create() may just be skipped, going
directly to disk_destroy(). Other method calls during that time are
just ignored. Since GEOM may now see the disks after CAM bus scan is
already completed, introduce per-periph boot hold functions. Enclosure
driver already had such mechanism, so just generalize it.
Reviewed by: imp
MFC after: 1 month
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D35784
2022-07-14 19:38:14 +00:00
|
|
|
void cam_periph_hold_boot(struct cam_periph *periph);
|
|
|
|
|
void cam_periph_release_boot(struct cam_periph *periph);
|
1998-09-15 06:33:23 +00:00
|
|
|
void cam_periph_invalidate(struct cam_periph *periph);
|
|
|
|
|
int cam_periph_mapmem(union ccb *ccb,
|
2015-09-30 13:31:37 +00:00
|
|
|
struct cam_periph_map_info *mapinfo,
|
|
|
|
|
u_int maxmap);
|
2023-12-28 17:08:04 +00:00
|
|
|
int cam_periph_unmapmem(union ccb *ccb,
|
1998-09-15 06:33:23 +00:00
|
|
|
struct cam_periph_map_info *mapinfo);
|
|
|
|
|
union ccb *cam_periph_getccb(struct cam_periph *periph,
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t priority);
|
1998-09-15 06:33:23 +00:00
|
|
|
int cam_periph_runccb(union ccb *ccb,
|
|
|
|
|
int (*error_routine)(union ccb *ccb,
|
|
|
|
|
cam_flags camflags,
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t sense_flags),
|
|
|
|
|
cam_flags camflags, uint32_t sense_flags,
|
1998-09-15 06:33:23 +00:00
|
|
|
struct devstat *ds);
|
2009-06-22 14:43:48 +00:00
|
|
|
int cam_periph_ioctl(struct cam_periph *periph, u_long cmd,
|
1998-09-15 06:33:23 +00:00
|
|
|
caddr_t addr,
|
|
|
|
|
int (*error_routine)(union ccb *ccb,
|
|
|
|
|
cam_flags camflags,
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t sense_flags));
|
1999-05-22 21:58:47 +00:00
|
|
|
void cam_freeze_devq(struct cam_path *path);
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t cam_release_devq(struct cam_path *path, uint32_t relsim_flags,
|
|
|
|
|
uint32_t opening_reduction, uint32_t arg,
|
1998-09-15 06:33:23 +00:00
|
|
|
int getcount_only);
|
2023-07-25 03:17:41 +00:00
|
|
|
void cam_periph_async(struct cam_periph *periph, uint32_t code,
|
1999-05-22 21:58:47 +00:00
|
|
|
struct cam_path *path, void *arg);
|
|
|
|
|
void cam_periph_bus_settle(struct cam_periph *periph,
|
|
|
|
|
u_int bus_settle_ms);
|
|
|
|
|
void cam_periph_freeze_after_event(struct cam_periph *periph,
|
|
|
|
|
struct timeval* event_time,
|
|
|
|
|
u_int duration_ms);
|
1998-09-15 06:33:23 +00:00
|
|
|
int cam_periph_error(union ccb *ccb, cam_flags camflags,
|
2023-07-25 03:17:41 +00:00
|
|
|
uint32_t sense_flags);
|
2018-03-14 17:53:37 +00:00
|
|
|
int cam_periph_invalidate_sysctl(SYSCTL_HANDLER_ARGS);
|
1998-09-15 06:33:23 +00:00
|
|
|
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
static __inline struct mtx *
|
|
|
|
|
cam_periph_mtx(struct cam_periph *periph)
|
2007-04-19 22:46:26 +00:00
|
|
|
{
|
2018-01-09 00:00:55 +00:00
|
|
|
if (periph != NULL)
|
|
|
|
|
return (xpt_path_mtx(periph->path));
|
|
|
|
|
else
|
|
|
|
|
return (NULL);
|
2007-04-19 22:46:26 +00:00
|
|
|
}
|
|
|
|
|
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
#define cam_periph_owned(periph) \
|
|
|
|
|
mtx_owned(xpt_path_mtx((periph)->path))
|
2007-04-19 22:46:26 +00:00
|
|
|
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
#define cam_periph_lock(periph) \
|
|
|
|
|
mtx_lock(xpt_path_mtx((periph)->path))
|
2009-12-06 11:48:53 +00:00
|
|
|
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
#define cam_periph_unlock(periph) \
|
|
|
|
|
mtx_unlock(xpt_path_mtx((periph)->path))
|
|
|
|
|
|
|
|
|
|
#define cam_periph_assert(periph, what) \
|
|
|
|
|
mtx_assert(xpt_path_mtx((periph)->path), (what))
|
|
|
|
|
|
|
|
|
|
#define cam_periph_sleep(periph, chan, priority, wmesg, timo) \
|
|
|
|
|
xpt_path_sleep((periph)->path, (chan), (priority), (wmesg), (timo))
|
Lay groundwork in CAM for recording and reporting physical path and
other device attributes stored in the CAM Existing Device Table (EDT).
This includes some infrastructure requried by the enclosure services
driver to export physical path information.
Make the CAM device advanced info interface accept store requests.
sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
- Replace scsi_get_sas_addr() with a scsi_get_devid() which takes
a callback that decides whether to accept a particular descriptor.
Provide callbacks for NAA IEEE Registered addresses and for SAS
addresses, replacing the old function. This is needed because
the old function doesn't work for an enclosure address for a SAS
device, which is not flagged as a SAS address, but is NAA IEEE
Registered. It may be worthwhile merging this interface with the
devid match interface.
- Add a few more defines for some device ID fields.
sbin/camcontrol/camcontrol.c:
- Update for the CCB_DEV_ADVINFO interface change.
cam/cam_xpt_internal.h:
- Add the new fields for the physical path string to the CAM EDT.
cam/cam_ccb.h:
- Rename CCB_GDEV_ADVINFO to simply CCB_DEV_ADVINFO, and the ccb
structure to ccb_dev_advinfo.
- Add a flag that changes this CCB's action to store, rather than
the default, retrieve.
- Add a new buffer type, CDAI_TYPE_PHYS_PATH, for the new CAM EDT
physpath field.
- Remove the never-implemented transport & proto flags.
cam/cam_xpt.c:
cam/cam_xpt.h:
- Add xpt_getattr(), which provides a wrapper for fetching a device's
attribute using the GEOM strings as key. This method currently
supports "GEOM::ident" and "GEOM::physpath".
Submitted by: will
Reviewed by : gibbs
Extend the XPT_DEV_MATCH api to allow a device search by device ID.
As far as the API is concerned, device ID is a binary blob to be
interpreted by the transport layer. The SCSI implementation assumes
it is an array of VPD device ID descriptors.
sys/cam/cam_ccb.h:
Create a new structure, device_id_match_pattern, and
update the XPT_DEV_MATCH datastructures and flags so
that this pattern type can be used.
sys/cam/cam_xpt.c:
- A single pattern matching on both inquiry data and device
ID is invalid. Report any violators.
- Pass device ID match requests through to the new routine
scsi_devid_match(). The direct call of a SCSI routine is
a layering violation, but no worse than the one a few
lines up that checks inquiry data. Defer cleaning this
up until our future, larger, rototilling of CAM.
- Zero out cam_ed and cam_et nodes on allocation. Prior to
this change, device_id_len and device_id were not inialized,
preventing proper detection of the presence of this
information.
sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add the scsi_match_devid() routine.
Add a helper function for extracting peripherial driver names
sys/cam/cam_periph.c:
sys/cam/cam_periph.h:
Add the cam_periph_list() method which fills an sbuf
with a comma delimited list of the peripheral instances
associated with a given CAM path.
Add a helper functions for SCSI commands used by the SES driver.
sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add structure definitions and csio filling functions for
the receive diagnostic results and send diagnostic commands.
Misc CAM XPT cleanups.
sys/cam/cam_xpt.c:
Broadcast AC_FOUND_DEVICE and AC_PATH_REGISTERED
events at the time async event handlers are attached
even when registering just for events on a partitular
SIM. Previously, you had to register for these
events on all SIMs in the system in order to get
the initial broadcast even though subsequent device
and path arrivals would be delivered.
sys/cam/cam_xpt.c:
Remove SIM mutex held asserts from path accessors.
CAM paths are reference counted and it is this
reference count, not the sim mutex, that garantees
they are stable.
Sponsored by: Spectra Logic Corporation
2011-06-14 14:53:17 +00:00
|
|
|
|
2013-03-29 07:50:47 +00:00
|
|
|
static inline struct cam_periph *
|
|
|
|
|
cam_periph_acquire_first(struct periph_driver *driver)
|
|
|
|
|
{
|
|
|
|
|
struct cam_periph *periph;
|
|
|
|
|
|
|
|
|
|
xpt_lock_buses();
|
|
|
|
|
periph = TAILQ_FIRST(&driver->units);
|
|
|
|
|
while (periph != NULL && (periph->flags & CAM_PERIPH_INVALID) != 0)
|
|
|
|
|
periph = TAILQ_NEXT(periph, unit_links);
|
|
|
|
|
if (periph != NULL)
|
|
|
|
|
periph->refcount++;
|
|
|
|
|
xpt_unlock_buses();
|
|
|
|
|
return (periph);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline struct cam_periph *
|
|
|
|
|
cam_periph_acquire_next(struct cam_periph *pperiph)
|
|
|
|
|
{
|
|
|
|
|
struct cam_periph *periph = pperiph;
|
|
|
|
|
|
Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.
Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.
While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.
To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.
Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.
HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.
Sponsored by: iXsystems, Inc.
MFC after: 2 months
2013-10-21 12:00:26 +00:00
|
|
|
cam_periph_assert(pperiph, MA_NOTOWNED);
|
2013-03-29 07:50:47 +00:00
|
|
|
xpt_lock_buses();
|
|
|
|
|
do {
|
|
|
|
|
periph = TAILQ_NEXT(periph, unit_links);
|
|
|
|
|
} while (periph != NULL && (periph->flags & CAM_PERIPH_INVALID) != 0);
|
|
|
|
|
if (periph != NULL)
|
|
|
|
|
periph->refcount++;
|
|
|
|
|
xpt_unlock_buses();
|
|
|
|
|
cam_periph_release(pperiph);
|
|
|
|
|
return (periph);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#define CAM_PERIPH_FOREACH(periph, driver) \
|
|
|
|
|
for ((periph) = cam_periph_acquire_first(driver); \
|
|
|
|
|
(periph) != NULL; \
|
|
|
|
|
(periph) = cam_periph_acquire_next(periph))
|
|
|
|
|
|
2018-01-26 23:56:07 +00:00
|
|
|
#define CAM_PERIPH_PRINT(p, msg, args...) \
|
|
|
|
|
printf("%s%d:" msg, (periph)->periph_name, (periph)->unit_number, ##args)
|
|
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#endif /* _KERNEL */
|
1998-09-15 06:33:23 +00:00
|
|
|
#endif /* _CAM_CAM_PERIPH_H */
|