mirror of
https://gitlab.freedesktop.org/pipewire/pipewire
synced 2024-10-14 20:02:38 +00:00
d5e333ac4b
Add a new refresh message to request a keyframe from the pinos server. pinospay: pass the refresh-request message upstream pinossink: turn refresh-request messages into events pinossrc: turn a keyframe event into a refresh-request message
357 lines
14 KiB
Plaintext
357 lines
14 KiB
Plaintext
Pinos
|
|
-----
|
|
|
|
The idea is to make a DBus service where you can provide
|
|
and consume media to/from.
|
|
|
|
Some of the requirements are:
|
|
|
|
- must be efficient for raw video using fd passing
|
|
- must be able to provide media from any process
|
|
- streaming media only (no seeking)
|
|
- policy to restrict access to devices and streams
|
|
|
|
Although an initial goal, the design is not limited to raw video
|
|
only and should be able to handle compressed video and other
|
|
streamable media as well.
|
|
|
|
The design is in some part inspired by pulseaudio, hence its original
|
|
name. We however are not concerned with playback of any of the media,
|
|
this should be handled by a separate consumer rendering the media to
|
|
a specific output device.
|
|
|
|
|
|
DBus protocol
|
|
-------------
|
|
|
|
The main daemon is registered on the session bus with name: org.pinos
|
|
|
|
Various Source1 objects are registered in the server based on the available
|
|
sources of content. Source1 has properties and has format descriptions of
|
|
what it can provide.
|
|
|
|
First a client needs to register with pinos by calling
|
|
org.pinos.Daemon1.ConnectClient(). This creates a new Client1 object that
|
|
the client must use for further communication.
|
|
|
|
A client can then do org.pinos.Client1.CreateSourceOutput() to create a
|
|
new SourceOutput1 to retrieve data from a source. It can specify a source
|
|
explicitly or let the server choose a source. The client must provide a list
|
|
of formats it can handle along with extra properties that can help with
|
|
selecting an appropriate source.
|
|
|
|
A client can then call org.pinos.SourceOutput1.Start() to negotiate the final
|
|
media format and start the data transfer. A new fd is returned to the client
|
|
along with the negotiated format and properties.
|
|
|
|
All further media transport is then performed on the fd. The client will read
|
|
from the fd to get data and metadata from the server. The wire format is
|
|
generic and extensible and allows for inline serialized events such as
|
|
property changes and format changes.
|
|
|
|
|
|
fd management
|
|
-------------
|
|
|
|
Pinos shares data between clients by using fd passing. Sometimes the memory
|
|
referenced by the fd needs to be reused. It is important that all pinos
|
|
clients lost their reference to the fd before it can be reused.
|
|
|
|
What follows are some scenarios of how the lifecycle of the fds are
|
|
managed.
|
|
|
|
* server side
|
|
|
|
v4l2src pinospay multisocketsink
|
|
| | |
|
|
| | |
|
|
make buffer |--------->| |
|
|
| | (1) |
|
|
| |------------>| (2) ----->
|
|
| | |
|
|
| |<------------|
|
|
| (4)| ............| (3)
|
|
| | |
|
|
|... | ... |
|
|
| | |
|
|
| |<------------| (5) <-----
|
|
| (6)| |
|
|
| | |
|
|
|
|
|
|
(1) pinospay generates the pinos message for the v4l2 input
|
|
buffer. It is assumed in the next steps that the payloader
|
|
receives fd-memory from v4l2src and that the memory is only
|
|
freed again when no clients are looking at the fd.
|
|
(2) multisocketsink sends the buffer to N Pinos clients
|
|
(3) for each client that is sent a buffer, multisocketsink sends an
|
|
event with the client object and buffer in it.
|
|
(4) pinospay uses the fdmanager object to map the fd-index that was
|
|
sent, to the buffer and client. The buffer is reffed and kept
|
|
alive for as long as the client is using the buffer.
|
|
(5) when a message is received from a client, multisocketsink sends an
|
|
event upstream.
|
|
(6) pinospay parses the message and instructs the fdmanager to release
|
|
the fd-index again. When all clients release the fd, the buffer
|
|
will be unreffed and v4l2src can reuse the memory.
|
|
|
|
* client consumer
|
|
|
|
pinossrc xvimagesink
|
|
| |
|
|
-------> (1)|------------------->| (2)
|
|
| |
|
|
| (3) |
|
|
|<-------------------|
|
|
<------- (4)| |
|
|
| |
|
|
|
|
|
|
(1) pinossrc receives a buffer from Pinos and wraps the fd with data
|
|
in an fd-memory. It remembers the fd-index as qdata on the memory.
|
|
It has a special DestroyNotify on the qdata.
|
|
(2) xvimagesink renders the buffer and frees the buffer.
|
|
(3) freeing the buffer causes the qdata DestoyNotify to be called.
|
|
(4) pinossrc constructs a release-fd message and sends it to Pinos
|
|
|
|
* client producer
|
|
|
|
|
|
videotestsrc pinossink
|
|
| |
|
|
(1)|------------------->|
|
|
| | (2) ----->
|
|
| |
|
|
| (4) | (3) <-----
|
|
|<-------------------|
|
|
|
|
|
|
(1) videotestsrc produces a buffer
|
|
(2) pinossink makes a PinosBuffer from the input buffer. It will also
|
|
keep the buffer in a hash table indexed by the fd-index.
|
|
(3) pinossink receives an fd-release message from Pinos. It removes
|
|
the fd-index from the hashtable and
|
|
the hashtable and the buffer is unreffed
|
|
(4) the fd is returned to videotestsrc for reuse.
|
|
|
|
|
|
* client producer, server side
|
|
|
|
socketsrc pinospay multisocketsink
|
|
| | |
|
|
------> (1) | (2)| |
|
|
|----------->| |
|
|
| |------------>| (3) -------->
|
|
| | | ....
|
|
| (5)|<------------| (4)
|
|
| | ...... |
|
|
| | |
|
|
| (7)|<------------| (6) <--------
|
|
<------- (8)|<-----------| |
|
|
| | |
|
|
|
|
|
|
(1) pinos buffer arrives from a client. socketsrc wraps the
|
|
fd
|
|
(2) pinospay sets a weak-ref on the buffer to know when it is
|
|
freed.
|
|
(3) multisocketsink sends the buffer to the clients
|
|
(4) for each buffer that is sent, an event is sent to the payloader
|
|
(5) the payloader uses the fdmanager to map the fd-index to a buffer
|
|
and a client. it keeps a ref on the buffer
|
|
(6) release-fd is received from a client
|
|
(7) pinospay removes the fd-index from the fdmanager. If all
|
|
clients released the fd, the buffer will be freeds, triggering
|
|
the DestroyNotify. This will then trigger an event with a release-fd
|
|
message to the source.
|
|
(8) the source sends the release-fd message to Pinos
|
|
|
|
|
|
* client remove
|
|
|
|
When a client disconnects from pinos, it must have released all fd-indexes
|
|
that it received. Pinos will force a release and will reuse the fd-indexes
|
|
when the client disconnects.
|
|
|
|
|
|
|
|
Wire
|
|
----
|
|
|
|
Fixed header
|
|
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Version |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Length |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
Version : 4 bytes : message version
|
|
Length : 4 bytes : total message length
|
|
|
|
|
|
Followed by 1 or more type-length-data sections
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Type | Len ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Data .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<type> : 1 byte
|
|
<length> : variable length, 7 bits, high bit is continuation marker
|
|
<data> : <length> bytes, see below for contents based on <type>
|
|
|
|
Types:
|
|
|
|
1: continuation section
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| Size ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
Rest of the commands can be found in the shared memory region at
|
|
@offset and @size. A shared memory region is negotiated when the client
|
|
connects to the server.
|
|
|
|
<offset> : 8 bytes : offset
|
|
<size> : 8 bytes : size
|
|
|
|
2: header
|
|
|
|
Header for payload
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| flags |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| seq |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| PTS ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| DTS-offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
|
|
<flags> : 4 bytes : buffer flags
|
|
<seq> : 4 bytes : sequence number
|
|
<pts> : 8 bytes : presentation time
|
|
<dts-offset> : 8 bytes : dts-offset
|
|
|
|
3: fd-payload section
|
|
|
|
Used to send a block of data between client and server. The type of fd and
|
|
the possible operations on it are negotiated when the client connects.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| offset ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| size ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| fd-index |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<id> : 4 bytes : id of the fd-payload
|
|
<offset> : 8 bytes : offset
|
|
<size> : 8 bytes : size
|
|
<fd-index> : 4 bytes : index of fd
|
|
|
|
4: release fd-payload
|
|
|
|
Release a fd-payload with <id>
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<id> : 4 bytes : the id number of the released fd-payload
|
|
|
|
5: format change
|
|
|
|
Perform an in-band format change. The following data blocks will be in this
|
|
new format.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| id | format ..... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<format-id> : 1 byte : format id
|
|
<format> : 0-terminated : contains serialized format
|
|
|
|
6: property changes
|
|
|
|
Notify a property change.
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| key .... .. |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| value .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| ...... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<key> : 0-terminated : key
|
|
<value> : 0-terminated : value
|
|
... : more key/values to fill length, 0 key or
|
|
message length is end
|
|
|
|
|
|
7: refresh request
|
|
|
|
Request a new refresh point
|
|
|
|
0 1 2 3
|
|
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| last-id |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| request-type |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| PTS ... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
| .... |
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
|
<last-id> : the last seen id
|
|
<request-type> : the type of request
|
|
0 = keyframe
|
|
1 = keyframe+full headers
|
|
<PTS> : the timestamp when the refresh should be,
|
|
0 for as soon as possible
|