From 0346c616e27cd0793ec98d273e1298ec24c12337 Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Fri, 23 Jul 2021 11:05:59 +0200 Subject: [PATCH] doc: add midi doc Reorganize the docs a little. First a short intro, then list the use cases, then the responsabilities of the various components, then the implementation in various places. --- doc/pipewire-access.dox | 82 +++++++++++++++++++++----------- doc/pipewire-midi.dox | 102 ++++++++++++++++++++++++++++++++++++++++ doc/pipewire-portal.dox | 77 ++++++++++++++++-------------- 3 files changed, 198 insertions(+), 63 deletions(-) create mode 100644 doc/pipewire-midi.dox diff --git a/doc/pipewire-access.dox b/doc/pipewire-access.dox index c31b9bd2d..21664350c 100644 --- a/doc/pipewire-access.dox +++ b/doc/pipewire-access.dox @@ -2,6 +2,57 @@ This document explains how access control is designed implemented. +PipeWire implements per client permissions on the objects in the graph. + +Permissions include R (read), W (write), X (execute) and M (metadata). + +An object with R permissions is visible to the client. The client will +receive registry events for the object and can interact with it. + +Methods can be called on an object with X permissions. Some of those +methods will only query state, others will modify the object. + +An object with W permission can be modified. This is usually done with +a method that modifies the state of the object. The W permission usually +implies the X permission. + +An object with M permission can be used as the subject in metadata. + +# Use cases + +## new clients need their permissions configured + +A new client is not allowed to communicate with the PipeWire daemon until +it has been configured with permissions. + +## Flatpaks can't modify other stream/device volumes + +Flatpaks should not be able to modify the state of certain objects. + +Permissions of the relevant PipeWire objects should not have the W +permission to avoid this. + +## Flatpaks can't move other streams to different devices + +Streams are moved to another device by setting the "target.node" metadata +on the node id. By not setting the M bit on the other objects, this can +be avoided. + +## Some application should be restricted in what they can see + +Application should only be able to see the objects that they are allowed +to see. For example, a web browser that was given access to a camera should +not be able to see audio devices. + +## Manager applications + +Some applications might be shipped in a flatpak but really require full +access to the PipeWire graph. This includes moving streams +(setting metadata) and modifying properties (volume). + + +# Design + ## The PipeWire daemon When a new client connects to the PipeWire daemon and right after it @@ -84,32 +135,7 @@ Permissions of objects for a client can be changed at any time by the session manager. Removing the client core R permission will suspend the client completely. - -## Use cases - -### new clients need their permissions configured - -A new client that has been suspended by the PipeWire access module need -to be resumed at some point with the right permissions. - -### Flatpaks can't modify other stream/device volumes - -Permissions of the PipeWire objects should not have the W permission to -avoid this. - -### Flatpaks can't move other streams to different devices - -Streams are moved to another device by setting the "target.node" metadata -on the node id. By not setting the M bit on the other objects, this can -be avoided. - -### Manager applications - -Some applications might be shipped in a flatpak but really require full -access to the PipeWire graph. This includes moving streams -(setting metadata) and modifying properties (volume). - -These manager applications will set the PW_KEY_MEDIA_CATEGORY property +Manager applications will set the PW_KEY_MEDIA_CATEGORY property in the client object to "Manager". The session manager needs to do additional checks to determine if the @@ -119,7 +145,9 @@ permission store checks or ask the user if the application is allowed full access. -### pipewire-media-session +# Implementation + +## pipewire-media-session The example media session has an access-flatpak module that handles the clients that have a PW_KEY_ACCESS as "flatpak". Other clients are diff --git a/doc/pipewire-midi.dox b/doc/pipewire-midi.dox new file mode 100644 index 000000000..d524880e6 --- /dev/null +++ b/doc/pipewire-midi.dox @@ -0,0 +1,102 @@ +/** \page page_midi PipeWire MIDI + +This document explains how MIDI is implemented. + +# Use cases + +## MIDI devices are made available as processing nodes/ports + +Applications need to be able to see a port for each stream of a +MIDI device. + +## MIDI devices can be plugged and unplugged + +When devices are plugged and unplugged the associated nodes/ports +need to be created and removed. + +## Applications can connect to MIDI devices + +Applications can create ports that can connect to the MIDI ports +so that data can be provided to or consumed from them. + +## Some MIDI devices are sinks or sources for midi data + +It should be possible to create a MIDI sink or source that routes the +midi events to specific midi ports. + +One example of such a sink would be in front of a software midi +renderer. + +An example of a MIDI source would be after a virtual keyboard or +as a mix from many midi input devices. + +## Applications should autoconnect to MIDI sinks or sources + +An application should be able to be connected to a MIDI sink when +it wants to play midi data. + +An application should be able to connect to a MIDI source when it +wants to capture midi data. + +# Design + +## SPA + +MIDI devices/streams are implemented with an SPA Node with generic +control input and output Ports. These ports have a media type of +"application/control" and the data transported over these ports +are of type spa_pod_sequence with the spa_pod_control type set to +SPA_CONTROL_Midi. + +This means that every midi event is timestamped with the sample +offset against the current graph clock cycle to get sample accurate +midi events that can be aligned with the corresponding sample data. + +Since the MIDI events are embedded in the generic control stream, +they can be interleaved with other control message types, such as +property updates or OSC messages. + +## The PipeWire daemon + +Nothing special is implemented for MIDI. Negotiation of formats +happens between "application/control" media types and buffers are +negotiated in the same way as any generic format. + +## The session manager + +The session manager needs to create the MIDI nodes/ports for the available +devices. + +This can either be done as a single node with ports per device/stream +or as separate nodes created by a MIDI device monitor. + +The session manager needs to be aware of the various MIDI sinks and sources +in order to route MIDI streams to them from applications that want this. + + +# Implementation + +## pipewire-media-session + +PipeWire media session uses the SPA_NAME_API_ALSA_SEQ_BRIDGE plugin +for the midi features. This creates a single SPA Node with ports per +MIDI client/stream. + +The media session will check the permissions on /dev/snd/seq before +attempting to create this node. It will also use inotify to wait +until the sequencer device node is accessible. + +## JACK + +JACK assumes all "application/control" ports are midi ports. + +The control messages are converted to the JACK event format by +filtering out the SPA_CONTROL_Midi types. On output ports, the JACK +event stream is converted to control messages in a similar way. + +There is a 1 to 1 mapping between the JACK events and control +messages so there is no information loss or need for complicated +conversions. + + +*/ diff --git a/doc/pipewire-portal.dox b/doc/pipewire-portal.dox index 674381fc5..6a64a62ab 100644 --- a/doc/pipewire-portal.dox +++ b/doc/pipewire-portal.dox @@ -2,24 +2,55 @@ This document explains how clients from the portal are handled. -## The portal - The portal is a DBus service that exposes a couple of interfaces to request access to the PipeWire daemon to perform a certain set of functions. The portal makes it possible to use PipeWire without having to expose -the local PipeWire socket in the sandboxed application. PipeWire can -detect and enforce extra permission checks on the Portal session. +the local PipeWire socket in the sandboxed application. It is the +portal can connects to PipeWire on behalf of the client. The client +becomes a portal managed client. PipeWire can detect and enforce +extra permission checks on the portal managed clients. Once such portal is the Camera portal that provides a PipeWire session to stream video from a camera. -In the portal case, it is the portal itself that will make a connection -to the PipeWire daemon to configure the session. It then hands the -file descriptor of the PipeWire connection to the client. The client -can then use the file descriptor to interface with the PipeWire -session. +# Use cases + +## new portal managed clients need device permissions configured + +When a new client is detected, the available objects need to be +scanned and permissions configured for each of them. + +Only the devices belonging to the media_roles given by the +portal are considered. + +## new devices need to be made visible to portal managed clients + +Newly created objects are made visible to a client when the client +is allowed to interact with it. + +Only the devices belonging to the media_roles given by the +portal are considered. + +## permissions for a device need to be revoked + +The session manager listens to changes in the permissions of devices +and will remove the client permissions accordingly. + +Usually this is implemented by listening to the permission store +DBus object. The desktop environment might provide a configuration panel +where these permissions can be managed. + + +# Design + +## The portal + +The portal itself will make a connection to the PipeWire daemon to +configure the session. It then hands the file descriptor of the PipeWire +connection to the client. The client can then use the file descriptor +to interface with the PipeWire session. When the portal connects, it will set the following properties on the client object: @@ -70,33 +101,7 @@ allowed. The permission store can be used for this. Usually the portal also implements "org.freedesktop.impl.portal.PermissionStore" for this. - -## Use cases - -### new portal managed clients need device permissions configured - -When a new client is detected, the available objects are scanned and -permissions are configured for each of them. - -Only the devices belonging to the media_roles given by the -portal are considered. - -### new devices need to be made visible to portal managed clients - -Newly created objects are made visible to a client when the client -is allowed to interact with it. - -Only the devices belonging to the media_roles given by the -portal are considered. - -### permissions for a device need to be revoked - -The session manager listens to changes in the permissions of devices -and will remove the client permissions accordingly. - -Usually this is implemented by listening to the permission store -DBus object. The desktop environment might provide a configuration panel -where these permissions can be managed. +# Implementation ## pipewire-media-session