From 70ce1da29c93eb4b59634d04521a57679973c612 Mon Sep 17 00:00:00 2001 From: Wim Taymans Date: Thu, 11 Jun 2020 13:29:30 +0200 Subject: [PATCH] doc: add some more docs --- doc/architecture.md | 218 ++++++++++++++++++++++++++++++++++++++++++++ doc/overview.md | 22 +++++ 2 files changed, 240 insertions(+) create mode 100644 doc/architecture.md diff --git a/doc/architecture.md b/doc/architecture.md new file mode 100644 index 000000000..dec92710b --- /dev/null +++ b/doc/architecture.md @@ -0,0 +1,218 @@ +# Architecture + +There are 2 main components that make up the PipeWire library: + + 1) An implementation of a graph based media processing engine. + 2) An asynchronous IPC mechanism to manipulate and introspect + a graph in another process. + +There is usually a daemon that implements the master graph and +clients that operate on this graph. + +The IPC mechanism in PipeWire is inspired by wayland in that it +follows the same design principles of objects and methods/events +along with how this API is presented to the user. + +PipeWire has a plugin architecture that allows new features to +be added (or removed) by the user. Plugins can hook into many +aspects of PipeWire and change the behaviour or number of +features dynamically. + +## Principles + +The PipeWire API is, a object oriented asynchronous protocol. +All requests and replies are method invocations on some object. + +Objects are identified with a unique ID. Each object implements an +interface and requestes result in invocations of methods on the +interface. + +The protocol is message based. A message sent by a client to the +server is called a method. A message from the server to the client +is called an event. Unlike Wayland, these messages are not (yet) +described in an external protocol file but implemented directly in +a protocol plugin. Protocol plugins can be added to add new +objects or even protocols when required. + +Messages are encoded with [SPA PODs][spa/pod.md], which make it +possible to encode complex objects with right types. + +Events from the server can be a reply to a method or can be emited +when the server state changes. + +Upon connecting to a server, it will broadcast its state. Clients +should listen for these state changes and cache them. There is no +need (or mechanism) to query the state of the server. + +The server also have a registry object that, when listening to, +will broadcast the presence of global objects and any changes in +their state. + +State about objects can be obtained by binding to them and listening +for state changes. + +## Versioning + +All interfaces have a version number. The maximum supported version +number of an interface is advertized in the registry global event. + +A client asks for a specific version of an interface when it binds +to them. It is the task of the server to adapt to the version of the +client. + +Interfaces increase their version number when new methods or events +are added. Methods or events should never be removed or changed for +simplicity. + +## Proxies and resources + +When a client connect to a PipeWire daemon, a new `struct pw_proxy` +object is created with ID 0. The `struct pw_core` interface is +assigned to the proxy. + +On the server side there is an equivalent `struct pw_resource` with +ID 0. Whenever the client sends a message on the proxy (by calling +a method on the interface of the proxy) it will transparently result +in a callback on the resource with the same ID. + +Likewise if the server sends a message (an event) on a resource, it +will result in an event on the client proxy with the same ID. + +PipeWire will notify a client when a resource ID (and thus also proxy +ID) becomes unused. The client is responsible for destroying the +proxy when it no longer wants to use it. + + +## Interfaces + +### `struct pw_loop` + +An abstraction for a `poll(2)` loop. It is usually part of one of: + +* `struct pw_main_loop`: a helper that can run and stop a `pw_loop`. + +* `struct pw_thread_loop`: a helper that can run and stop a `pw_loop` + in a different thread. It also has some helper + functions for various thread related synchronization + issues. + +* `struct pw_data_loop`: a helper that can run and stop a `pw_loop` + in a real-time thread along with some useful helper + functions. + + +### `struct pw_context` + +The main context for PipeWire resources. It keeps track of the mainloop, +loaded modules, the processing graph and proxies to remote PipeWire +instances. + +An application has to select an implementation of a `struct pw_loop` +when creating a context. + +The context has methods to create the various objects you can use to +build a server or client application. + + +### `struct pw_core` + +A proxy to a remote PipeWire instance. This is used to send messages +to a remote PipeWire daemon and to receive events from it. + +A core proxy can be used to receive errors from the remote daemon +or to perform a roundtrip message to flush out pending requests. + +Other core methods and events are used internally for the object +life cycle management internally. + +### `struct pw_registry` + +A proxy to a PipeWire registry object. It emits events about the +available objects on the server and can be used to bind to those +objects in order call methods or receive events from them. + +### `struct pw_module` + +A proxy to a loadable module. Modules implement functionality such +as provide new objects or policy. + +### `struct pw_factory` + +A proxy to an object that can create other objects. + +### `struct pw_device` + +A proxy to a device object. Device objects model a physical hardware +or software device in the system and can create other objects +such as nodes or other devices. + +### `struct pw_node` + +A Proxy to a processing node in the graph. Nodes can have input and +output ports and the ports can be linked together to form a graph. + +### `struct pw_port` + +A Proxy to an input or output port of a node. They can be linked +together to form a processing graph. + +### `struct pw_link` + +A proxy to a link between in output and input port. A link negotiates +a format and buffers between ports. A port can be linked to many other +ports and PipeWire will manage mixing and duplicating the buffers. + + +## High level helper objects + +Some high level objects are implemented to make it easier to interface +with a PipeWire graph. + +### `struct pw_filter` + +A `struct pw_filter` allows you implement a processing filter that can +be added to a PipeWire graph. It is comparable to a JACK client. + +### `struct pw_stream` + +a `struct pw_stream` makes it easy to implement a playback or capture +client for the graph. It takes care of format conversion and buffer +sizes. It is comparable to Core Audio AudioQueue or a PulseAudio +stream. + + +## Security + +With the default native protocol, clients connect to PipeWire using +a named socket. This results in a client socket that is used to +send messages. + +For sandboxed clients, it is possible to get the client socket via +other ways, like using the portal. In that case, a portal will +do the connection for the client and then hands the connection socket +to the client. + +All objects in PipeWire have per client permission bits, currently +READ, WRITE and EXECUTE. A client can not see an objects unless it +has READ permissions. Similarly, a client can only execute methods +on an object when the EXECUTE bit is set and to modify the state of +an object, the client needs WRITE permissions. + +A client (the portal after it makes a connection) can drop permissions +on an object. Once dropped, it can never reacquire the permission. + +Clients with WRITE/EXECUTE permissions on another client can +add and remove permissions for the client at will. + +Clients that need permissions assigned to them can be started in +blocked mode and resume when perrmissions are assigned to them by +a session manager or portal, for example. + +PipeWire uses memfd (`memfd_create(2)`) or DMA-BUF for sharing media +and data between clients. Clients can thus not look at other clients +data unless they can see the objects and connect to them. + +## Implementation + +PipeWire also exposes an API to implement the server side objects in +a graph. diff --git a/doc/overview.md b/doc/overview.md index 7f15da538..d80df938f 100644 --- a/doc/overview.md +++ b/doc/overview.md @@ -17,6 +17,28 @@ The framework is used to build a modular daemon that can be configured to: * a central hub where video can be made available for other applications such as the gnome-shell screencast API. +## Motivation + +Linux has no unified framework for exchanging multimedia content between +applications or even devices. Im most cases, developers realized that +a user-space daemon is needed to make this possible: + +* For video content, we typically rely on the compositor to render our + data. +* For video capture, we usually go directly to the hardware devices, with + all security implications and inflexible routing that this brings. +* For consumer audio, we use PulseAudio to manage and mix multiple streams + from clients +* For Pro audio, we use JACK to manage the graph of nodes. + +None of these solutions (with perhaps to some extend Wayland), however, +were designed to support the security features that are required when +daeling with flatpaks or other containerized applications. PipeWire +aims to solve this problem and provides a unified framework to run both +consumer and Pro audio as well as video capture and processing in a +secure way. + + ## Components Currently PipeWire ships with the following components: