mirror of
https://gitlab.freedesktop.org/pipewire/pipewire
synced 2024-10-06 16:09:43 +00:00
doc: add documentation about the design of media & session management objects
This commit is contained in:
parent
b445bdea27
commit
04b1881211
331
doc/objects_design.md
Normal file
331
doc/objects_design.md
Normal file
|
@ -0,0 +1,331 @@
|
|||
# Objects Design
|
||||
|
||||
This document is a design reference on the various objects that exist
|
||||
in the PipeWire media and session management graphs, explaining what these
|
||||
objects are, how they are meant to be used and how they relate to other
|
||||
kinds of objects and concepts that exist in subsystems or other libraries.
|
||||
|
||||
## The media graph
|
||||
|
||||
The media graph represents and enables the media flow inside the PipeWire
|
||||
daemon and between the daemon and its clients. It consists of nodes, ports
|
||||
and links.
|
||||
|
||||
```
|
||||
+------------+ +------------+
|
||||
| | | |
|
||||
| +--------+ Link +--------+ |
|
||||
| Node | Port |--------| Port | Node |
|
||||
| +--------+ +--------+ |
|
||||
| | | |
|
||||
+------------+ +------------+
|
||||
```
|
||||
|
||||
### Node
|
||||
|
||||
A **node** is a media processing element. It consumes and/or produces buffers
|
||||
that contain data, such as audio or video.
|
||||
|
||||
A node may operate entirely inside the PipeWire daemon or it may be operating
|
||||
in a client process. In the second case, media is transferred to/from that
|
||||
client using the PipeWire protocol.
|
||||
|
||||
In an analogy to GStreamer, a _node_ is similar (but not equal) to a
|
||||
GStreamer _element_.
|
||||
|
||||
### Port
|
||||
|
||||
A **port** is attached on a **node** and provides an interface for input
|
||||
or output of media on the node. A node may have multiple ports.
|
||||
|
||||
A port always has a direction, input or output:
|
||||
* Input: it allows media input into the node (in other terms, it is a _sink_)
|
||||
* Output: it outputs media out of the node (in other terms, it is a _source_)
|
||||
|
||||
In an analogy to GStreamer, a _port_ is similar (but not equal) to a
|
||||
GStreamer _pad_.
|
||||
|
||||
### Link
|
||||
|
||||
A **link** connects 2 ports of opposite direction, making media flow from
|
||||
the output port to the input port.
|
||||
|
||||
## The session management graph
|
||||
|
||||
The session management graph is a virtual, higher-level representation of the
|
||||
media flow. It is created entirely by the session manager and it can affect
|
||||
the routing on the media graph only through the session manager's actions.
|
||||
|
||||
The session management graph is useful to abstract the complexity of the
|
||||
actual media flow both for the target user and for the policy management
|
||||
codebase.
|
||||
|
||||
```
|
||||
+---------------------+ +----------------------+
|
||||
| | | |
|
||||
| +----------------+ Endpoint Link +----------------+ |
|
||||
| Endpoint |Endpoint Stream |-----------------|Endpoint Stream | Endpoint |
|
||||
| +----------------+ +----------------+ |
|
||||
| | | |
|
||||
+---------------------+ +----------------------+
|
||||
```
|
||||
|
||||
### Endpoint
|
||||
|
||||
An **endpoint** is a session management object that provides a representation
|
||||
of user-conceivable places where media can be routed to/from.
|
||||
|
||||
Examples of endpoints associated with hardware on a desktop-like system:
|
||||
* Laptop speakers
|
||||
* USB webcam
|
||||
* Bluetooth headset microphone
|
||||
* Line out stereo jack port
|
||||
|
||||
Examples of endpoints associated with hardware in a car:
|
||||
* Speakers amplifier
|
||||
* Front right seat microphone array
|
||||
* Rear left seat headphones
|
||||
* Bluetooth phone voice gateway
|
||||
* Hardware FM radio device
|
||||
|
||||
Examples of endpoints associated with software:
|
||||
* Desktop screen capture source
|
||||
* Media player application
|
||||
* Camera application
|
||||
|
||||
In most cases an endpoint maps to a node on the media graph, but this is not
|
||||
always the case. An endpoint may be backed by several nodes or no nodes at all.
|
||||
Different endpoints may also be sharing nodes in some cases.
|
||||
|
||||
An endpoint that does not map to any node may be useful to represent hardware
|
||||
that the session manager needs to be able to control, but there is no way
|
||||
to route media to/from that hardware through the PipeWire media graph. For
|
||||
example, in a car we may have a CD player device that is directly wired to the
|
||||
speakers amplifier and therefore audio flows between them without passing
|
||||
through the controlling CPU. However, it is useful for the session manager to
|
||||
be able to represent the *CD player endpoint* and the _endpoint link_ between
|
||||
it and the amplifier, so that it can apply audio policy that takes into account
|
||||
whether the CD player is playing or not.
|
||||
|
||||
#### Target
|
||||
|
||||
An **endpoint** may be grouping together targets that can be reached by
|
||||
following the same route and they are mutually exclusive with each other.
|
||||
|
||||
For example, the speakers and the headphones jack on a laptop are usually
|
||||
mutually exclusive by hardware design (hardware mutes the speakers when the
|
||||
headphones are enabled) and they share the same ALSA PCM device, so audio still
|
||||
follows the same route to reach both.
|
||||
|
||||
In this case, a session manager may choose to group these two targets into the
|
||||
same endpoint, using a parameter on the _endpoint_ object to allow the user
|
||||
to choose the target (if the hardware allows configuring this at all).
|
||||
|
||||
### Endpoint Stream
|
||||
|
||||
An **endpoint stream** is attached to an **endpoint** and represents a logical
|
||||
path that can be taken to reach this endpoint, often associated with
|
||||
a _use case_.
|
||||
|
||||
For example, the "Speakers amplifier" endpoint in a car might have the
|
||||
following streams:
|
||||
* _Music_: a path to play music;
|
||||
the implementation will output this to all speakers, using the volume
|
||||
that has been configured for the "Music" use case
|
||||
* _Voice_: a path to play a voice message, such as a navigation message or
|
||||
feedback from a voice assistant; the implementation will output this
|
||||
to the front speakers only, lowering the volume of the music (if any)
|
||||
on these speakers at the same time
|
||||
* _Emergency_: a path to play an emergency situation sound (a beep,
|
||||
or equivalent); the implementation will output this on all speakers,
|
||||
increasing the volume to a factory-defined value if necessary (to ensure
|
||||
that it is audible) while muting audio from all other streams at the
|
||||
same time
|
||||
|
||||
In another example, a microphone that can be used for activating a voice
|
||||
assistant might have the following streams:
|
||||
* _Capture_: a path to capture directly from the microphone; this can be used
|
||||
by an application that listens for the assistant's wake-word in order
|
||||
to activate the full voice recognition engine
|
||||
* _CaptureDelayed_: a path to capture with a constant delay (meaning that
|
||||
starting capturing now will actually capture something that was spoken
|
||||
a little earlier); this can be used by the full voice recognition engine,
|
||||
allowing it to start after the wake-word has been spoken while capturing
|
||||
audio that also includes the wake-word
|
||||
|
||||
Endpoint streams may be mutually exclusive or they may used simultaneously,
|
||||
depending on the implementation.
|
||||
|
||||
Endpoint streams may be implemented in many ways:
|
||||
* By plugging additional nodes in the media graph that link to the device node
|
||||
(ex. a simple buffering node linked to an alsa source node could implement
|
||||
the _CaptureDelayed_ stream in the above microphone example)
|
||||
* By using a different device node (ex. different ALSA device on the same card)
|
||||
that has a special meaning for the hardware
|
||||
* By triggering switches on the hardware (ex. modify ALSA controls on the
|
||||
same device)
|
||||
|
||||
### Endpoint Link
|
||||
|
||||
An **endpoint link** connects 2 streams from 2 different endpoints, creating
|
||||
a logical representation of media flow between the endpoints.
|
||||
|
||||
An **endpoint link** may be implemented by creating one or more _links_ in the
|
||||
underlying media graph, or it may be implemented by configuring hardware
|
||||
resources to enable media flow, in case the flow does not pass through the
|
||||
media graph.
|
||||
|
||||
#### Constructing
|
||||
|
||||
Constructing an **endpoint link** is done by asking the _endpoint stream_
|
||||
objects to prepare it. First, the source stream is asked to provide linking
|
||||
information. When the information is retrieved, the sink stream is asked to
|
||||
use this information to prepare and to provide its own linking information.
|
||||
When this is done, the session manager is asked to create the link using the
|
||||
provided information.
|
||||
|
||||
This mechanism allows stream implementations:
|
||||
* to prepare for linking, adjusting hardware paths if necessary
|
||||
* to check for stream linking compatibility; not all streams can be connected
|
||||
to all others (ex. streams with media flow in the hardware cannot be linked
|
||||
to streams that are backed by nodes in the media graph)
|
||||
* to provide implementation-specific information for linking; in the standard
|
||||
case this is going to be a list of _ports_ to be linked in the media graph,
|
||||
but in a hardware-flow case it can be any kind of hardware-specific detail
|
||||
|
||||
## Other related objects
|
||||
|
||||
### Device
|
||||
|
||||
A **device** represents a handle to an underlying API that is used to create
|
||||
higher level objects, such as nodes, or other devices.
|
||||
|
||||
Well-known devices include:
|
||||
| Device API | Description |
|
||||
| :--- | :--- |
|
||||
| alsa.pcm.device | A handle to an ALSA card (ex. `hw:0`, `hw:1`, etc) |
|
||||
| alsa.seq.device | A handle to an ALSA Midi device |
|
||||
| v4l2.device | A handle to a V4L2 device (`/dev/video0`, `/dev/video1`, etc..) |
|
||||
| jack.device | A JACK client, allowing PipeWire to slave to JACK for audio input/output |
|
||||
|
||||
A device may have a _profile_, which allows the user to choose between
|
||||
multiple configurations that the device may be capable of having, or to simply
|
||||
turn the device _off_, which means that the handle is closed and not used
|
||||
by PipeWire.
|
||||
|
||||
### Session
|
||||
|
||||
The **session** represents the session manager and can be used to expose
|
||||
global properties or methods that affect the session management.
|
||||
|
||||
#### Default endpoints
|
||||
|
||||
The session is responsible for book-keeping the default device endpoints (one
|
||||
for each kind of device) that is to be used to link new clients when
|
||||
simulating a PulseAudio-like behavior, where the user can choose from the UI
|
||||
device preferences.
|
||||
|
||||
For example, a system may have both "Speakers" and "HDMI" endpoints on the
|
||||
"Audio Output" category and the user may be offered to make a choice within
|
||||
the UI to select which endpoint she wants to use by default for audio output.
|
||||
This preference is meant to be stored in the session object.
|
||||
|
||||
#### Multiple sessions
|
||||
|
||||
It is not currently defined whether it is allowed to have multiple sessions
|
||||
or not and how the system should behave if this happens.
|
||||
|
||||
## Mappings to underlying subsystem objects
|
||||
|
||||
### ALSA UCM
|
||||
|
||||
This is a ***proposal***
|
||||
|
||||
| ALSA / UCM | PipeWire |
|
||||
| :--- | :--- |
|
||||
| ALSA card | device |
|
||||
| UCM verb | device profile |
|
||||
| UCM device | endpoint (+ target, grouping conflicting devices into the same endpoint) |
|
||||
| UCM modifier | endpoint stream |
|
||||
| PCM stream | node |
|
||||
|
||||
In UCM mode, an ALSA card is represented as a PipeWire device, with the
|
||||
available UCM verbs listed as profiles of the device.
|
||||
|
||||
Activating a profile (i.e. a verb) will create the necessary nodes for the
|
||||
available PCM streams and at the same time it will also create one endpoint
|
||||
for each UCM device. Optionally, conflicting UCM devices can be grouped in
|
||||
the same endpoint, listing the conflicting options as targets of the endpoint.
|
||||
|
||||
The available UCM modifiers for each UCM device will be added as streams, plus
|
||||
one "default" stream for accessing the device with no modifiers.
|
||||
|
||||
### ALSA fallback
|
||||
|
||||
| ALSA | PipeWire |
|
||||
| :--- | :--- |
|
||||
| card | device |
|
||||
| PCM stream | node + endpoint |
|
||||
|
||||
In the case where UCM (or another similar mechanism) is not available,
|
||||
ALSA cards are represented as PipeWire devices with only 2 profiles: On/Off
|
||||
|
||||
When the On profile is activated, a node and an associated endpoint are created
|
||||
for every available PCM stream.
|
||||
|
||||
Endpoints in this case have only one "default" stream, unless they are extended
|
||||
by the session manager to have software-backed streams.
|
||||
|
||||
### V4L2
|
||||
|
||||
***FIXME***
|
||||
|
||||
| V4L2 | PipeWire |
|
||||
| :--- | :--- |
|
||||
| device | device + node |
|
||||
|
||||
## Relationship to other APIs
|
||||
|
||||
### PulseAudio
|
||||
|
||||
#### Mapping PipeWire objects for access by PulseAudio clients
|
||||
|
||||
| PipeWire | PulseAudio |
|
||||
| :--- | :--- |
|
||||
| device | card |
|
||||
| device profile | card profile |
|
||||
| endpoint (associated with a device) | sink / source |
|
||||
| endpoint (associated with a client) | sink-input / source-output |
|
||||
| endpoint target | port |
|
||||
| endpoint stream | N/A, pa clients will be limited to the default stream |
|
||||
|
||||
#### Mapping PulseAudio clients to PipeWire
|
||||
|
||||
| PulseAudio | PipeWire |
|
||||
| :--- | :--- |
|
||||
| stream | client + node + endpoint (no targets, 1 default stream) |
|
||||
|
||||
### Jack
|
||||
|
||||
Note: This section is about JACK clients connecting to PipeWire through the
|
||||
JACK compatibility library. The scenario where PipeWire connects to another
|
||||
JACK server as a client is out of scope here.
|
||||
|
||||
#### Mapping PipeWire objects for access by JACK clients
|
||||
|
||||
| PipeWire | JACK |
|
||||
| :--- | :--- |
|
||||
| node | client |
|
||||
| port | port |
|
||||
| device | N/A |
|
||||
| endpoint | N/A |
|
||||
|
||||
#### Mapping JACK clients to PipeWire
|
||||
|
||||
| JACK | PipeWire |
|
||||
| :--- | :--- |
|
||||
| client | client + node |
|
||||
| port | port |
|
||||
|
||||
JACK clients do not create endpoints. A session manager should be JACK aware
|
||||
in order to anticipate direct node linking
|
Loading…
Reference in a new issue