mirror of
https://gitlab.freedesktop.org/pipewire/pipewire
synced 2024-09-06 00:40:34 +00:00
de53315f6e
Implements the basics of the PORT_UPDATE control message Add the ports to the proxy node with whe PORT_UPDATE control message. Let the proxy node check the events and create dbus objects based on added/removed ports.
510 lines
19 KiB
Plaintext
510 lines
19 KiB
Plaintext
Pinos
|
|
-----
|
|
|
|
The idea is to make a DBus service where you can provide
|
|
and consume media to/from.
|
|
|
|
Some of the requirements are:
|
|
|
|
- must be efficient for raw video using fd passing
|
|
- must be able to provide/consume/process media from any process
|
|
- streaming media only (no seeking)
|
|
- policy to restrict access to devices and streams
|
|
|
|
Although an initial goal, the design is not limited to raw video
|
|
only and should be able to handle compressed video and other
|
|
streamable media as well.
|
|
|
|
The design is in some part inspired by pulseaudio, hence its original
|
|
name. Increasinly we also seem to add functionality of jack.
|
|
|
|
|
|
Objects
|
|
-------
|
|
|
|
Daemon1: the main pinos daemon
|
|
/org/pinos/server
|
|
Client1: a connected client, the result object from call
|
|
Daemon1.ConnectClient
|
|
/org/pinos/client*
|
|
Device1: a physical device on the machine, devices can provide
|
|
processing nodes
|
|
/org/pinos/device*
|
|
Node1: a processing node, this can be a source, sink or transform
|
|
element.
|
|
/org/pinos/node*
|
|
Port1: a port on a Node1, ports can be input or output ports
|
|
/org/pinos/node*/port*
|
|
Channel1: a communication channel between a client and port
|
|
/org/pinos/client/channel*
|
|
Link1: a link between 2 ports
|
|
/org/pinos/link*
|
|
|
|
|
|
DBus protocol
|
|
-------------
|
|
|
|
The main daemon is registered on the session bus with name: org.pinos
|
|
|
|
Various Node1 objects are registered in the server based on the available
|
|
sources or sinks of content. Source1 has properties and has format descriptions of
|
|
what it can provide.
|
|
|
|
First a client needs to register with pinos by calling
|
|
org.pinos.Daemon1.ConnectClient(). This creates a new Client1 object that
|
|
the client must use for further communication.
|
|
|
|
A client can then do org.pinos.Client1.CreateChannel() to create a
|
|
new Channel to retrieve/send data from/to a node. It can specify a node/port
|
|
explicitly or let the server choose a port. The client must provide a list
|
|
of formats it can handle along with extra properties that can help with
|
|
selecting an appropriate port.
|
|
|
|
A client can then call org.pinos.Channel1.Start() to negotiate the final
|
|
media format and start the data transfer. A new fd is returned to the client
|
|
along with the negotiated format and properties.
|
|
|
|
All further media transport is then performed on the fd. The client will read
|
|
from the fd to get data and metadata from the server. The wire format is
|
|
generic and extensible and allows for inline serialized events such as
|
|
property changes and format changes.
|
|
|
|
|
|
fd management
|
|
-------------
|
|
|
|
Pinos shares data between clients by using fd passing. Sometimes the memory
|
|
referenced by the fd needs to be reused. It is important that all pinos
|
|
clients lost their reference to the fd before it can be reused.
|
|
|
|
What follows are some scenarios of how the lifecycle of the fds are
|
|
managed.
|
|
|
|
* server side
|
|
|
|
v4l2src pinossocketsink
|
|
| |
|
|
| |
|
|
make buffer |--------->|
|
|
| | (1)
|
|
| | (2) ----->
|
|
| | (3)
|
|
|... | ...
|
|
| |
|
|
| | (4) <-----
|
|
|<---------|
|
|
|
|
|
|
(1) pinossocketsink generates the pinos message from the v4l2 input
|
|
buffer. It is assumed in the next steps that the sink
|
|
receives fd-memory from v4l2src and that the memory is only
|
|
freed again when no clients are looking at the fd.
|
|
(2) pinossocketsink sends the buffer to N Pinos clients
|
|
(3) for each client that is sent a buffer, pinossocketsink uses the
|
|
fdmanager object to map the fd-index that was
|
|
sent, to the buffer and client. The buffer is reffed and kept
|
|
alive for as long as the client is using the buffer.
|
|
(4) when a message is received from a client, pinossocketsink
|
|
parses the message and instructs the fdmanager to release
|
|
the fd-index again. When all clients release the fd, the buffer
|
|
will be unreffed and v4l2src can reuse the memory.
|
|
|
|
* client consumer
|
|
|
|
pinossrc xvimagesink
|
|
| |
|
|
-------> (1)|------------------->| (2)
|
|
| |
|
|
| (3) |
|
|
|<-------------------|
|
|
<------- (4)| |
|
|
| |
|
|
|
|
|
|
(1) pinossrc receives a buffer from Pinos and wraps the fd with data
|
|
in an fd-memory. It remembers the fd-index as qdata on the memory.
|
|
It has a special DestroyNotify on the qdata.
|
|
(2) xvimagesink renders the buffer and frees the buffer.
|
|
(3) freeing the buffer causes the qdata DestoyNotify to be called.
|
|
(4) pinossrc constructs a release-fd message and sends it to Pinos
|
|
|
|
* client producer
|
|
|
|
|
|
videotestsrc pinossink
|
|
| |
|
|
(1)|------------------->|
|
|
| | (2) ----->
|
|
| |
|
|
| (4) | (3) <-----
|
|
|<-------------------|
|
|
|
|
|
|
(1) videotestsrc produces a buffer
|
|
(2) pinossink makes a PinosBuffer from the input buffer. It will also
|
|
keep the buffer in a hash table indexed by the fd-index.
|
|
(3) pinossink receives an fd-release message from Pinos. It removes
|
|
the fd-index from the hashtable and
|
|
the hashtable and the buffer is unreffed
|
|
(4) the fd is returned to videotestsrc for reuse.
|
|
|
|
|
|
* client producer, server side
|
|
|
|
socketsrc pinossocketsink
|
|
| |
|
|
------> (1) | (2)|
|
|
|----------->|
|
|
| | (3) -------->
|
|
| | ....
|
|
| (4)|
|
|
| | ....
|
|
| |
|
|
| (6)| (5) <--------
|
|
<------- (7)|<-----------|
|
|
| |
|
|
|
|
|
|
(1) pinos buffer arrives from a client. socketsrc wraps the
|
|
fd
|
|
(2) pinossocketsink sets a weak-ref on the buffer to know when it is
|
|
freed.
|
|
(3) pinossocketsink sends the buffer to the clients
|
|
(4) for each buffer that is sent, the sink uses the fdmanager to map the
|
|
fd-index to a buffer and a client. it keeps a ref on the buffer
|
|
(5) release-fd is received from a client
|
|
(6) pinossocketsink removes the fd-index from the fdmanager. If all
|
|
clients released the fd, the buffer will be freeds, triggering
|
|
the DestroyNotify. This will then trigger an event with a release-fd
|
|
message to the source.
|
|
(7) the source sends the release-fd message to Pinos
|
|
|
|
|
|
* client remove
|
|
|
|
When a client disconnects from pinos, it must have released all fd-indexes
|
|
that it received. Pinos will force a release and will reuse the fd-indexes
|
|
when the client disconnects.
|
|
|
|
|
|
|
|
Wire
|
|
----
|
|
|
|
Fixed header
|
|
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Version |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Flags |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Length |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
Version : 4 bytes : message version
|
|
Flags : 4 bytes : extra flags
|
|
Length : 4 bytes : total message length
|
|
|
|
|
|
Followed by 1 or more type-length-data sections
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Type | Len ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Data .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<type> : 1 byte
|
|
<length> : variable length, 7 bits, high bit is continuation marker
|
|
<data> : <length> bytes, see below for contents based on <type>
|
|
|
|
Types:
|
|
|
|
1: continuation section
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Size ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
Rest of the commands can be found in the shared memory region at
|
|
@offset and @size. A shared memory region is negotiated when the client
|
|
connects to the server.
|
|
|
|
<offset> : 8 bytes : offset
|
|
<size> : 8 bytes : size
|
|
|
|
2: header
|
|
|
|
Header for payload
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| flags |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| seq |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| PTS ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| DTS-offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
|
|
<flags> : 4 bytes : buffer flags
|
|
<seq> : 4 bytes : sequence number
|
|
<pts> : 8 bytes : presentation time
|
|
<dts-offset> : 8 bytes : dts-offset
|
|
|
|
3: fd-payload section
|
|
|
|
Used to send a block of data between client and server. The type of fd and
|
|
the possible operations on it are negotiated when the client connects.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| size ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| fd-index |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<id> : 4 bytes : id of the fd-payload
|
|
<offset> : 8 bytes : offset
|
|
<size> : 8 bytes : size
|
|
<fd-index> : 4 bytes : index of fd
|
|
|
|
4: release fd-payload
|
|
|
|
Release a fd-payload with <id>
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<id> : 4 bytes : the id number of the released fd-payload
|
|
|
|
5: format change
|
|
|
|
Perform an in-band format change. The following data blocks will be in this
|
|
new format.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id | format ..... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<format-id> : 1 byte : format id
|
|
<format> : 0-terminated : contains serialized format
|
|
|
|
6: property changes
|
|
|
|
Notify a property change.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| key .... .. |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| value .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<key> : 0-terminated : key
|
|
<value> : 0-terminated : value
|
|
... : more key/values to fill length, 0 key or
|
|
message length is end
|
|
|
|
|
|
7: refresh request
|
|
|
|
Request a new refresh point
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| last-id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| request-type |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| PTS ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<last-id> : the last seen id
|
|
<request-type> : the type of request
|
|
0 = keyframe
|
|
1 = keyframe+full headers
|
|
<PTS> : the timestamp when the refresh should be,
|
|
0 for as soon as possible
|
|
|
|
|
|
|
|
communication channel
|
|
---------------------
|
|
|
|
|
|
+-----+ +----+ +----+
|
|
| | | S | | |
|
|
| ----- ----- |
|
|
+-----+ +----+ +----+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+----+
|
|
| |
|
|
| C |
|
|
+----+
|
|
|
|
|
|
Client Proxy
|
|
| INIT
|
|
node-update |
|
|
-------------------------------------->|
|
|
port-update |
|
|
-------------------------------------->|
|
|
state-change CONFIGURE | CONFIGURE
|
|
-------------------------------------->|
|
|
|<--- enum-ports
|
|
|<--- enum-formats
|
|
|<--- add-port
|
|
|<--- remove-port
|
|
set-property |<--- set-property
|
|
<--------------------------------------|
|
|
set-format |<--- set-format
|
|
<--------------------------------------|
|
|
|
|
|
port-update |
|
|
-------------------------------------->|
|
|
state-change READY | READY
|
|
-------------------------------------->|
|
|
|<--- port memory requirements
|
|
add-mem |<--- use-buffers
|
|
<--------------------------------------|
|
|
remove-mem |
|
|
<--------------------------------------|
|
|
add-buffer |
|
|
<--------------------------------------|
|
|
remove-buffer |
|
|
<--------------------------------------|
|
|
|
|
|
pause |<--- stop
|
|
<--------------------------------------|
|
|
state-change PAUSED | PAUSED
|
|
-------------------------------------->|
|
|
|
|
|
play |<--- start
|
|
<--------------------------------------|
|
|
state-change STREAMING | STREAMING
|
|
-------------------------------------->|
|
|
|
|
|
need-input |
|
|
<------------------------------------->|
|
|
have-output |
|
|
<------------------------------------->|
|
|
process-buffer |
|
|
<------------------------------------->|
|
|
reuse-buffer |
|
|
<------------------------------------->|
|
|
|
|
|
|
|
|
|
|
1) Update config C->S INIT
|
|
|
|
node-update
|
|
port-update
|
|
state change CONFIGURE
|
|
|
|
2) Set formats S->C CONFIGURE
|
|
|
|
set-property
|
|
enumerate ports
|
|
add-port
|
|
remove-port
|
|
enumerate formats
|
|
set-format
|
|
|
|
3) Buffer requirements update C->S
|
|
|
|
Update port status
|
|
state change READY if enough formats are set
|
|
|
|
4) Start S->C READY
|
|
|
|
read port memory requirements
|
|
add_mem
|
|
remove_mem
|
|
add_buffer
|
|
remove_buffer
|
|
command START/PAUSE
|
|
|
|
5) Pause S->C PAUSED
|
|
|
|
state change STREAMING
|
|
set-format to NULL -> state change to CONFIGURE
|
|
|
|
5) data transfer C->S STREAMING
|
|
|
|
need-input
|
|
have-output
|
|
|
|
process_buffer
|
|
reuse_buffer
|
|
state change PAUSED
|
|
|
|
6) data transfer S->C
|
|
|
|
process_buffer
|
|
reuse_buffer
|
|
|
|
7) format change C->S
|
|
|
|
port-update
|
|
state change CONFIGURE
|
|
|
|
8) format change S->C
|
|
|
|
Send set-format change on ports -> READY if new memory requirements
|
|
-> PAUSED/STREAMING if all ok
|
|
|
|
9) format-change to NULL
|
|
|
|
state change CONFIGURE
|
|
|
|
10) ERROR
|