mirror of
https://github.com/git/git
synced 2024-10-30 04:01:21 +00:00
a58088abe2
Signed-off-by: Justin Lebar <jlebar@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
554 lines
21 KiB
Text
554 lines
21 KiB
Text
Packfile transfer protocols
|
|
===========================
|
|
|
|
Git supports transferring data in packfiles over the ssh://, git:// and
|
|
file:// transports. There exist two sets of protocols, one for pushing
|
|
data from a client to a server and another for fetching data from a
|
|
server to a client. All three transports (ssh, git, file) use the same
|
|
protocol to transfer data.
|
|
|
|
The processes invoked in the canonical Git implementation are 'upload-pack'
|
|
on the server side and 'fetch-pack' on the client side for fetching data;
|
|
then 'receive-pack' on the server and 'send-pack' on the client for pushing
|
|
data. The protocol functions to have a server tell a client what is
|
|
currently on the server, then for the two to negotiate the smallest amount
|
|
of data to send in order to fully update one or the other.
|
|
|
|
Transports
|
|
----------
|
|
There are three transports over which the packfile protocol is
|
|
initiated. The Git transport is a simple, unauthenticated server that
|
|
takes the command (almost always 'upload-pack', though Git
|
|
servers can be configured to be globally writable, in which 'receive-
|
|
pack' initiation is also allowed) with which the client wishes to
|
|
communicate and executes it and connects it to the requesting
|
|
process.
|
|
|
|
In the SSH transport, the client just runs the 'upload-pack'
|
|
or 'receive-pack' process on the server over the SSH protocol and then
|
|
communicates with that invoked process over the SSH connection.
|
|
|
|
The file:// transport runs the 'upload-pack' or 'receive-pack'
|
|
process locally and communicates with it over a pipe.
|
|
|
|
Git Transport
|
|
-------------
|
|
|
|
The Git transport starts off by sending the command and repository
|
|
on the wire using the pkt-line format, followed by a NUL byte and a
|
|
hostname parameter, terminated by a NUL byte.
|
|
|
|
0032git-upload-pack /project.git\0host=myserver.com\0
|
|
|
|
--
|
|
git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]
|
|
request-command = "git-upload-pack" / "git-receive-pack" /
|
|
"git-upload-archive" ; case sensitive
|
|
pathname = *( %x01-ff ) ; exclude NUL
|
|
host-parameter = "host=" hostname [ ":" port ]
|
|
--
|
|
|
|
Only host-parameter is allowed in the git-proto-request. Clients
|
|
MUST NOT attempt to send additional parameters. It is used for the
|
|
git-daemon name based virtual hosting. See --interpolated-path
|
|
option to git daemon, with the %H/%CH format characters.
|
|
|
|
Basically what the Git client is doing to connect to an 'upload-pack'
|
|
process on the server side over the Git protocol is this:
|
|
|
|
$ echo -e -n \
|
|
"0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" |
|
|
nc -v example.com 9418
|
|
|
|
If the server refuses the request for some reasons, it could abort
|
|
gracefully with an error message.
|
|
|
|
----
|
|
error-line = PKT-LINE("ERR" SP explanation-text)
|
|
----
|
|
|
|
|
|
SSH Transport
|
|
-------------
|
|
|
|
Initiating the upload-pack or receive-pack processes over SSH is
|
|
executing the binary on the server via SSH remote execution.
|
|
It is basically equivalent to running this:
|
|
|
|
$ ssh git.example.com "git-upload-pack '/project.git'"
|
|
|
|
For a server to support Git pushing and pulling for a given user over
|
|
SSH, that user needs to be able to execute one or both of those
|
|
commands via the SSH shell that they are provided on login. On some
|
|
systems, that shell access is limited to only being able to run those
|
|
two commands, or even just one of them.
|
|
|
|
In an ssh:// format URI, it's absolute in the URI, so the '/' after
|
|
the host name (or port number) is sent as an argument, which is then
|
|
read by the remote git-upload-pack exactly as is, so it's effectively
|
|
an absolute path in the remote filesystem.
|
|
|
|
git clone ssh://user@example.com/project.git
|
|
|
|
|
v
|
|
ssh user@example.com "git-upload-pack '/project.git'"
|
|
|
|
In a "user@host:path" format URI, its relative to the user's home
|
|
directory, because the Git client will run:
|
|
|
|
git clone user@example.com:project.git
|
|
|
|
|
v
|
|
ssh user@example.com "git-upload-pack 'project.git'"
|
|
|
|
The exception is if a '~' is used, in which case
|
|
we execute it without the leading '/'.
|
|
|
|
ssh://user@example.com/~alice/project.git,
|
|
|
|
|
v
|
|
ssh user@example.com "git-upload-pack '~alice/project.git'"
|
|
|
|
A few things to remember here:
|
|
|
|
- The "command name" is spelled with dash (e.g. git-upload-pack), but
|
|
this can be overridden by the client;
|
|
|
|
- The repository path is always quoted with single quotes.
|
|
|
|
Fetching Data From a Server
|
|
---------------------------
|
|
|
|
When one Git repository wants to get data that a second repository
|
|
has, the first can 'fetch' from the second. This operation determines
|
|
what data the server has that the client does not then streams that
|
|
data down to the client in packfile format.
|
|
|
|
|
|
Reference Discovery
|
|
-------------------
|
|
|
|
When the client initially connects the server will immediately respond
|
|
with a listing of each reference it has (all branches and tags) along
|
|
with the object name that each reference currently points to.
|
|
|
|
$ echo -e -n "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" |
|
|
nc -v example.com 9418
|
|
00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\0multi_ack thin-pack
|
|
side-band side-band-64k ofs-delta shallow no-progress include-tag
|
|
00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration
|
|
003f7217a7c7e582c46cec22a130adf4b9d7d950fba0 refs/heads/master
|
|
003cb88d2441cac0977faf98efc80305012112238d9d refs/tags/v0.9
|
|
003c525128480b96c89e6418b1e40909bf6c5b2d580f refs/tags/v1.0
|
|
003fe92df48743b7bc7d26bcaabfddde0a1e20cae47c refs/tags/v1.0^{}
|
|
0000
|
|
|
|
Server SHOULD terminate each non-flush line using LF ("\n") terminator;
|
|
client MUST NOT complain if there is no terminator.
|
|
|
|
The returned response is a pkt-line stream describing each ref and
|
|
its current value. The stream MUST be sorted by name according to
|
|
the C locale ordering.
|
|
|
|
If HEAD is a valid ref, HEAD MUST appear as the first advertised
|
|
ref. If HEAD is not a valid ref, HEAD MUST NOT appear in the
|
|
advertisement list at all, but other refs may still appear.
|
|
|
|
The stream MUST include capability declarations behind a NUL on the
|
|
first ref. The peeled value of a ref (that is "ref^{}") MUST be
|
|
immediately after the ref itself, if presented. A conforming server
|
|
MUST peel the ref if it's an annotated tag.
|
|
|
|
----
|
|
advertised-refs = (no-refs / list-of-refs)
|
|
*shallow
|
|
flush-pkt
|
|
|
|
no-refs = PKT-LINE(zero-id SP "capabilities^{}"
|
|
NUL capability-list LF)
|
|
|
|
list-of-refs = first-ref *other-ref
|
|
first-ref = PKT-LINE(obj-id SP refname
|
|
NUL capability-list LF)
|
|
|
|
other-ref = PKT-LINE(other-tip / other-peeled)
|
|
other-tip = obj-id SP refname LF
|
|
other-peeled = obj-id SP refname "^{}" LF
|
|
|
|
shallow = PKT-LINE("shallow" SP obj-id)
|
|
|
|
capability-list = capability *(SP capability)
|
|
capability = 1*(LC_ALPHA / DIGIT / "-" / "_")
|
|
LC_ALPHA = %x61-7A
|
|
----
|
|
|
|
Server and client MUST use lowercase for obj-id, both MUST treat obj-id
|
|
as case-insensitive.
|
|
|
|
See protocol-capabilities.txt for a list of allowed server capabilities
|
|
and descriptions.
|
|
|
|
Packfile Negotiation
|
|
--------------------
|
|
After reference and capabilities discovery, the client can decide to
|
|
terminate the connection by sending a flush-pkt, telling the server it can
|
|
now gracefully terminate, and disconnect, when it does not need any pack
|
|
data. This can happen with the ls-remote command, and also can happen when
|
|
the client already is up-to-date.
|
|
|
|
Otherwise, it enters the negotiation phase, where the client and
|
|
server determine what the minimal packfile necessary for transport is,
|
|
by telling the server what objects it wants, its shallow objects
|
|
(if any), and the maximum commit depth it wants (if any). The client
|
|
will also send a list of the capabilities it wants to be in effect,
|
|
out of what the server said it could do with the first 'want' line.
|
|
|
|
----
|
|
upload-request = want-list
|
|
*shallow-line
|
|
*1depth-request
|
|
flush-pkt
|
|
|
|
want-list = first-want
|
|
*additional-want
|
|
|
|
shallow-line = PKT_LINE("shallow" SP obj-id)
|
|
|
|
depth-request = PKT_LINE("deepen" SP depth)
|
|
|
|
first-want = PKT-LINE("want" SP obj-id SP capability-list LF)
|
|
additional-want = PKT-LINE("want" SP obj-id LF)
|
|
|
|
depth = 1*DIGIT
|
|
----
|
|
|
|
Clients MUST send all the obj-ids it wants from the reference
|
|
discovery phase as 'want' lines. Clients MUST send at least one
|
|
'want' command in the request body. Clients MUST NOT mention an
|
|
obj-id in a 'want' command which did not appear in the response
|
|
obtained through ref discovery.
|
|
|
|
The client MUST write all obj-ids which it only has shallow copies
|
|
of (meaning that it does not have the parents of a commit) as
|
|
'shallow' lines so that the server is aware of the limitations of
|
|
the client's history.
|
|
|
|
The client now sends the maximum commit history depth it wants for
|
|
this transaction, which is the number of commits it wants from the
|
|
tip of the history, if any, as a 'deepen' line. A depth of 0 is the
|
|
same as not making a depth request. The client does not want to receive
|
|
any commits beyond this depth, nor does it want objects needed only to
|
|
complete those commits. Commits whose parents are not received as a
|
|
result are defined as shallow and marked as such in the server. This
|
|
information is sent back to the client in the next step.
|
|
|
|
Once all the 'want's and 'shallow's (and optional 'deepen') are
|
|
transferred, clients MUST send a flush-pkt, to tell the server side
|
|
that it is done sending the list.
|
|
|
|
Otherwise, if the client sent a positive depth request, the server
|
|
will determine which commits will and will not be shallow and
|
|
send this information to the client. If the client did not request
|
|
a positive depth, this step is skipped.
|
|
|
|
----
|
|
shallow-update = *shallow-line
|
|
*unshallow-line
|
|
flush-pkt
|
|
|
|
shallow-line = PKT-LINE("shallow" SP obj-id)
|
|
|
|
unshallow-line = PKT-LINE("unshallow" SP obj-id)
|
|
----
|
|
|
|
If the client has requested a positive depth, the server will compute
|
|
the set of commits which are no deeper than the desired depth. The set
|
|
of commits start at the client's wants.
|
|
|
|
The server writes 'shallow' lines for each
|
|
commit whose parents will not be sent as a result. The server writes
|
|
an 'unshallow' line for each commit which the client has indicated is
|
|
shallow, but is no longer shallow at the currently requested depth
|
|
(that is, its parents will now be sent). The server MUST NOT mark
|
|
as unshallow anything which the client has not indicated was shallow.
|
|
|
|
Now the client will send a list of the obj-ids it has using 'have'
|
|
lines, so the server can make a packfile that only contains the objects
|
|
that the client needs. In multi_ack mode, the canonical implementation
|
|
will send up to 32 of these at a time, then will send a flush-pkt. The
|
|
canonical implementation will skip ahead and send the next 32 immediately,
|
|
so that there is always a block of 32 "in-flight on the wire" at a time.
|
|
|
|
----
|
|
upload-haves = have-list
|
|
compute-end
|
|
|
|
have-list = *have-line
|
|
have-line = PKT-LINE("have" SP obj-id LF)
|
|
compute-end = flush-pkt / PKT-LINE("done")
|
|
----
|
|
|
|
If the server reads 'have' lines, it then will respond by ACKing any
|
|
of the obj-ids the client said it had that the server also has. The
|
|
server will ACK obj-ids differently depending on which ack mode is
|
|
chosen by the client.
|
|
|
|
In multi_ack mode:
|
|
|
|
* the server will respond with 'ACK obj-id continue' for any common
|
|
commits.
|
|
|
|
* once the server has found an acceptable common base commit and is
|
|
ready to make a packfile, it will blindly ACK all 'have' obj-ids
|
|
back to the client.
|
|
|
|
* the server will then send a 'NACK' and then wait for another response
|
|
from the client - either a 'done' or another list of 'have' lines.
|
|
|
|
In multi_ack_detailed mode:
|
|
|
|
* the server will differentiate the ACKs where it is signaling
|
|
that it is ready to send data with 'ACK obj-id ready' lines, and
|
|
signals the identified common commits with 'ACK obj-id common' lines.
|
|
|
|
Without either multi_ack or multi_ack_detailed:
|
|
|
|
* upload-pack sends "ACK obj-id" on the first common object it finds.
|
|
After that it says nothing until the client gives it a "done".
|
|
|
|
* upload-pack sends "NAK" on a flush-pkt if no common object
|
|
has been found yet. If one has been found, and thus an ACK
|
|
was already sent, it's silent on the flush-pkt.
|
|
|
|
After the client has gotten enough ACK responses that it can determine
|
|
that the server has enough information to send an efficient packfile
|
|
(in the canonical implementation, this is determined when it has received
|
|
enough ACKs that it can color everything left in the --date-order queue
|
|
as common with the server, or the --date-order queue is empty), or the
|
|
client determines that it wants to give up (in the canonical implementation,
|
|
this is determined when the client sends 256 'have' lines without getting
|
|
any of them ACKed by the server - meaning there is nothing in common and
|
|
the server should just send all of its objects), then the client will send
|
|
a 'done' command. The 'done' command signals to the server that the client
|
|
is ready to receive its packfile data.
|
|
|
|
However, the 256 limit *only* turns on in the canonical client
|
|
implementation if we have received at least one "ACK %s continue"
|
|
during a prior round. This helps to ensure that at least one common
|
|
ancestor is found before we give up entirely.
|
|
|
|
Once the 'done' line is read from the client, the server will either
|
|
send a final 'ACK obj-id' or it will send a 'NAK'. 'obj-id' is the object
|
|
name of the last commit determined to be common. The server only sends
|
|
ACK after 'done' if there is at least one common base and multi_ack or
|
|
multi_ack_detailed is enabled. The server always sends NAK after 'done'
|
|
if there is no common base found.
|
|
|
|
Then the server will start sending its packfile data.
|
|
|
|
----
|
|
server-response = *ack_multi ack / nak
|
|
ack_multi = PKT-LINE("ACK" SP obj-id ack_status LF)
|
|
ack_status = "continue" / "common" / "ready"
|
|
ack = PKT-LINE("ACK SP obj-id LF)
|
|
nak = PKT-LINE("NAK" LF)
|
|
----
|
|
|
|
A simple clone may look like this (with no 'have' lines):
|
|
|
|
----
|
|
C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack \
|
|
side-band-64k ofs-delta\n
|
|
C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n
|
|
C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n
|
|
C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n
|
|
C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n
|
|
C: 0000
|
|
C: 0009done\n
|
|
|
|
S: 0008NAK\n
|
|
S: [PACKFILE]
|
|
----
|
|
|
|
An incremental update (fetch) response might look like this:
|
|
|
|
----
|
|
C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack \
|
|
side-band-64k ofs-delta\n
|
|
C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n
|
|
C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n
|
|
C: 0000
|
|
C: 0032have 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n
|
|
C: [30 more have lines]
|
|
C: 0032have 74730d410fcb6603ace96f1dc55ea6196122532d\n
|
|
C: 0000
|
|
|
|
S: 003aACK 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01 continue\n
|
|
S: 003aACK 74730d410fcb6603ace96f1dc55ea6196122532d continue\n
|
|
S: 0008NAK\n
|
|
|
|
C: 0009done\n
|
|
|
|
S: 0031ACK 74730d410fcb6603ace96f1dc55ea6196122532d\n
|
|
S: [PACKFILE]
|
|
----
|
|
|
|
|
|
Packfile Data
|
|
-------------
|
|
|
|
Now that the client and server have finished negotiation about what
|
|
the minimal amount of data that needs to be sent to the client is, the server
|
|
will construct and send the required data in packfile format.
|
|
|
|
See pack-format.txt for what the packfile itself actually looks like.
|
|
|
|
If 'side-band' or 'side-band-64k' capabilities have been specified by
|
|
the client, the server will send the packfile data multiplexed.
|
|
|
|
Each packet starting with the packet-line length of the amount of data
|
|
that follows, followed by a single byte specifying the sideband the
|
|
following data is coming in on.
|
|
|
|
In 'side-band' mode, it will send up to 999 data bytes plus 1 control
|
|
code, for a total of up to 1000 bytes in a pkt-line. In 'side-band-64k'
|
|
mode it will send up to 65519 data bytes plus 1 control code, for a
|
|
total of up to 65520 bytes in a pkt-line.
|
|
|
|
The sideband byte will be a '1', '2' or a '3'. Sideband '1' will contain
|
|
packfile data, sideband '2' will be used for progress information that the
|
|
client will generally print to stderr and sideband '3' is used for error
|
|
information.
|
|
|
|
If no 'side-band' capability was specified, the server will stream the
|
|
entire packfile without multiplexing.
|
|
|
|
|
|
Pushing Data To a Server
|
|
------------------------
|
|
|
|
Pushing data to a server will invoke the 'receive-pack' process on the
|
|
server, which will allow the client to tell it which references it should
|
|
update and then send all the data the server will need for those new
|
|
references to be complete. Once all the data is received and validated,
|
|
the server will then update its references to what the client specified.
|
|
|
|
Authentication
|
|
--------------
|
|
|
|
The protocol itself contains no authentication mechanisms. That is to be
|
|
handled by the transport, such as SSH, before the 'receive-pack' process is
|
|
invoked. If 'receive-pack' is configured over the Git transport, those
|
|
repositories will be writable by anyone who can access that port (9418) as
|
|
that transport is unauthenticated.
|
|
|
|
Reference Discovery
|
|
-------------------
|
|
|
|
The reference discovery phase is done nearly the same way as it is in the
|
|
fetching protocol. Each reference obj-id and name on the server is sent
|
|
in packet-line format to the client, followed by a flush-pkt. The only
|
|
real difference is that the capability listing is different - the only
|
|
possible values are 'report-status', 'delete-refs' and 'ofs-delta'.
|
|
|
|
Reference Update Request and Packfile Transfer
|
|
----------------------------------------------
|
|
|
|
Once the client knows what references the server is at, it can send a
|
|
list of reference update requests. For each reference on the server
|
|
that it wants to update, it sends a line listing the obj-id currently on
|
|
the server, the obj-id the client would like to update it to and the name
|
|
of the reference.
|
|
|
|
This list is followed by a flush-pkt and then the packfile that should
|
|
contain all the objects that the server will need to complete the new
|
|
references.
|
|
|
|
----
|
|
update-request = *shallow command-list [pack-file]
|
|
|
|
shallow = PKT-LINE("shallow" SP obj-id)
|
|
|
|
command-list = PKT-LINE(command NUL capability-list LF)
|
|
*PKT-LINE(command LF)
|
|
flush-pkt
|
|
|
|
command = create / delete / update
|
|
create = zero-id SP new-id SP name
|
|
delete = old-id SP zero-id SP name
|
|
update = old-id SP new-id SP name
|
|
|
|
old-id = obj-id
|
|
new-id = obj-id
|
|
|
|
pack-file = "PACK" 28*(OCTET)
|
|
----
|
|
|
|
If the receiving end does not support delete-refs, the sending end MUST
|
|
NOT ask for delete command.
|
|
|
|
The pack-file MUST NOT be sent if the only command used is 'delete'.
|
|
|
|
A pack-file MUST be sent if either create or update command is used,
|
|
even if the server already has all the necessary objects. In this
|
|
case the client MUST send an empty pack-file. The only time this
|
|
is likely to happen is if the client is creating
|
|
a new branch or a tag that points to an existing obj-id.
|
|
|
|
The server will receive the packfile, unpack it, then validate each
|
|
reference that is being updated that it hasn't changed while the request
|
|
was being processed (the obj-id is still the same as the old-id), and
|
|
it will run any update hooks to make sure that the update is acceptable.
|
|
If all of that is fine, the server will then update the references.
|
|
|
|
Report Status
|
|
-------------
|
|
|
|
After receiving the pack data from the sender, the receiver sends a
|
|
report if 'report-status' capability is in effect.
|
|
It is a short listing of what happened in that update. It will first
|
|
list the status of the packfile unpacking as either 'unpack ok' or
|
|
'unpack [error]'. Then it will list the status for each of the references
|
|
that it tried to update. Each line is either 'ok [refname]' if the
|
|
update was successful, or 'ng [refname] [error]' if the update was not.
|
|
|
|
----
|
|
report-status = unpack-status
|
|
1*(command-status)
|
|
flush-pkt
|
|
|
|
unpack-status = PKT-LINE("unpack" SP unpack-result LF)
|
|
unpack-result = "ok" / error-msg
|
|
|
|
command-status = command-ok / command-fail
|
|
command-ok = PKT-LINE("ok" SP refname LF)
|
|
command-fail = PKT-LINE("ng" SP refname SP error-msg LF)
|
|
|
|
error-msg = 1*(OCTECT) ; where not "ok"
|
|
----
|
|
|
|
Updates can be unsuccessful for a number of reasons. The reference can have
|
|
changed since the reference discovery phase was originally sent, meaning
|
|
someone pushed in the meantime. The reference being pushed could be a
|
|
non-fast-forward reference and the update hooks or configuration could be
|
|
set to not allow that, etc. Also, some references can be updated while others
|
|
can be rejected.
|
|
|
|
An example client/server communication might look like this:
|
|
|
|
----
|
|
S: 007c74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/local\0report-status delete-refs ofs-delta\n
|
|
S: 003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\n
|
|
S: 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\n
|
|
S: 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/team\n
|
|
S: 0000
|
|
|
|
C: 003e7d1665144a3a975c05f1f43902ddaf084e784dbe 74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/debug\n
|
|
C: 003e74730d410fcb6603ace96f1dc55ea6196122532d 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/master\n
|
|
C: 0000
|
|
C: [PACKDATA]
|
|
|
|
S: 000eunpack ok\n
|
|
S: 0018ok refs/heads/debug\n
|
|
S: 002ang refs/heads/master non-fast-forward\n
|
|
----
|