Update "git p4" to work with Python 3.
* yz/p4-py3:
ci: use python3 in linux-gcc and osx-gcc and python2 elsewhere
git-p4: use python3's input() everywhere
git-p4: simplify regex pattern generation for parsing diff-tree
git-p4: use dict.items() iteration for python3 compatibility
git-p4: use functools.reduce instead of reduce
git-p4: fix freezing while waiting for fast-import progress
git-p4: use marshal format version 2 when sending to p4
git-p4: open .gitp4-usercache.txt in text mode
git-p4: convert path to unicode before processing them
git-p4: encode/decode communication with git for python3
git-p4: encode/decode communication with p4 for python3
git-p4: remove string type aliasing
git-p4: change the expansion test from basestring to list
git-p4: make python2.7 the oldest supported version
Check we can talk to the remote host before starting the git-fastimport
subchild.
Otherwise we fail to connect, and then exit, leaving git-fastimport
still running since we did not wait() for it.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
After an error, git-p4 calls die(). This just exits, and leaves child
processes still running.
Instead of calling die(), raise an exception and catch it where the
child process(es) (git-fastimport) are created.
This was analyzed in detail here:
https://public-inbox.org/git/20190227094926.GE19739@szeder.dev/
This change does not address the particular issue of p4CmdList()
invoking a subchild and not waiting for it on error.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This makes it easier to try/catch around this block of code to ensure
cleanup following p4 failures is handled properly.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pylint is incredibly useful for finding bugs, but git-p4 has never used
it, so there are a lot of warnings that while important, don't actually
result in bugs.
Let's turn those off for now, so we can get some useful output.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently when there is a P4 error, git-p4 calls die() which just exits.
This then leaves the git-fast-import process still running, and can even
leave p4 itself still running.
As a result, git-p4 fails to exit cleanly. This is a particular problem
for people running the unit tests in regression.
Use this exception to report errors upwards, cleaning up as the error
propagates.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Ensure that we can safely call self.closeStreams() multiple times, and
can also call it even if there is no git fast-import stream at all.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Python3 deprecates raw_input() from 2.7 and replaced it with input().
Since we do not need 2.7's input() semantics, `raw_input()` is aliased
to `input()` for easy forward compatability.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It is not clear why a generator was used to create the regex used to
parse git-diff-tree output; I assume an early implementation required
it, but is not part of the mainline change.
Simply use a lazily initialized global instead.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Python3 uses dict.items() instead of .iteritems() to provide
iteratoration over dict. Although items() is technically less efficient
for python2.7 (allocates a new list instead of simply iterating), the
amount of data involved is very small and the penalty negligible.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
For python3, reduce() has been moved to functools.reduce(). This is
also available in python2.7.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As part of its importing process, git-p4 sends a `checkpoint` followed
immediately by `progress` to fast-import to force synchronization.
Due to buffering, it is possible for the `progress` command to not be
flushed before git-p4 begins to wait for the corresponding response.
This causes the script to freeze completely, and is consistently
observable at least on python-3.6.9.
Make sure this command sequence is completely flushed before waiting.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
p4 does not appear to understand marshal format version 3 and above.
Version 2 was the latest supported by python-2.7.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Opening .gitp4-usercache.txt in text mode makes python 3 happy without
explicitly adding encoding and decoding.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
P4 allows essentially arbitrary encoding for path data while we would
perfer to be dealing only with unicode strings. Since path data need to
survive round-trip back to p4, this patch implements the general policy
that we store path data as-is, but decode them to unicode before doing
any non-trivial processing.
A new `decode_path()` method is provided that generally does the correct
conversion, taking into account `git-p4.pathEncoding` configuration.
For python2.7, path strings will be left as-is if it only contains ASCII
characters.
For python3, decoding is always done so that we have str objects.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Under python3, calls to write() on the stream to `git fast-import` must
be encoded. This patch wraps the IO object such that this encoding is
done transparently.
Conversely, any text data read from subprocesses must also be decoded
before running through the rest of the pipeline.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The marshalled dict in the response given on STDOUT by p4 uses `str` for
keys and string values. When run using python3, these values are
deserialized as `bytes`, leading to a whole host of problems as the rest
of the code assumes `str` is used throughout.
This patch changes the deserialization behaviour such that, as much as
possible, text output from p4 is decoded to native unicode strings.
Exceptions are made for the field `data` as it is usually arbitrary
binary data. `depotFile[0-9]*`, `path`, and `clientFile` are also exempt
as they contain path strings not encoded with UTF-8, and must survive
round-trip back to p4.
Conversely, text data being piped to p4 must always be encoded when
running under python3.
encode_text_stream() and decode_text_stream() were added to make these
transformations more convenient.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Now that python2.7 is the minimum required version and we no longer use
the basestring type, it is not necessary to use type aliasing to ensure
python3 compatibility.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Python 3 handles strings differently than Python 2.7. Since Python 2
is reaching it's end of life, a series of changes are being submitted to
enable python 3.5 and following support. The current code fails basic
tests under python 3.5.
Some codepaths can represent a command line the program
internally prepares to execute either as a single string
(i.e. each token properly quoted, concatenated with $IFS) or
as a list of argv[] elements, and there are 9 places where
we say "if X is isinstance(_, basestring), then do this
thing to handle X as a command line in a single string; if
not, X is a command line in a list form".
This does not work well with Python 3, as there is no
basestring (everything is Unicode now), and even with Python
2, it was not an ideal way to tell the two cases apart,
because an internally formed command line could have been in
a single Unicode string.
Flip the check to say "if X is not a list, then handle X as
a command line in a single string; otherwise treat it as a
command line in a list form".
This will get rid of references to 'basestring', to migrate
the code ready for Python 3.
Thanks-to: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Python2.6 and earlier have been end-of-life'd for many years now, and
we actually already use 2.7-only features in the code. Make the version
check reflect current realities.
This also removes the need to explicitly define CalledProcessError if
it's not available.
Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Miscellaneous small UX improvements on "git-p4".
* bk/p4-misc-usability:
git-p4: show detailed help when parsing options fail
git-p4: yes/no prompts should sanitize user text
When a user provides invalid parameters to git-p4, the program
reports the failure but does not provide the correct command syntax.
Add an exception handler to the command-line argument parser to display
the command's specific command line parameter syntax when an exception
is thrown. Rethrow the exception so the current behavior is retained.
Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When prompting the user interactively for direction, the tests are
not forgiving of user input format.
For example, the first query asks for a yes/no response. If the user
enters the full word "yes" or "no" or enters a capital "Y" the test
will fail.
Create a new function, prompt(prompt_text) where
* prompt_text is the text prompt for the user
* returns a single character where valid return values are
found by inspecting prompt_text for single characters
surrounded by square brackets
This new function must prompt the user for input and sanitize it by
converting the response to a lower case string, trimming leading and
trailing spaces, and checking if the first character is in the list
of choices. If it is, return the first letter.
Change the current references to raw_input() to use this new function.
Since the method requires the returned text to be one of the available
choices, remove the loop from the calling code that handles response
verification.
Thanks-to: Denton Liu <liu.denton@gmail.com>
Signed-off-by: Ben Keene <seraphire@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git lfs" allows users to specify the custom storage location with
the configuration variable `lfs.storage`, but when interacting with
GitLFS pointers, "git p4" always uses the hardcoded default that is
the `.git/lfs/` directory, without paying attention to the
configuration.
Use the value configured in `lfs.storage`, if exists, as all the
"git" operations do, for consistency.
Signed-off-by: r.burenkov <panzercheg@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Avoid double-open exceptions on Windows platform when
calculating for lfs compressed size threshold
(git-p4.largeFileCompressedThreshold) comparisons.
Take new approach using the NamedTemporaryFile()
file-like object as input to the ZipFile() which
auto-deletes after implicit close leaving with scope.
Original code had double-open exception on Windows
platform because file still open from NamedTemporaryFile()
using generated filename instead of object.
Thanks to Andrey for patiently suggesting several
iterations on this change for avoiding exceptions!
Also print error details after resulting IOError to make
debugging cause of exception less mysterious when it has
nothing to do with "git version recent enough."
Signed-off-by: Philip.McGraw <Philip.McGraw@bentley.com>
Reviewed-by: Andrey Mazo <ahippo@yandex.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git p4" update.
* am/p4-branches-excludes:
git-p4: respect excluded paths when detecting branches
git-p4: add failing test for "git-p4: respect excluded paths when detecting branches"
git-p4: don't exclude other files with same prefix
git-p4: add failing test for "don't exclude other files with same prefix"
git-p4: don't groom exclude path list on every commit
git-p4: match branches case insensitively if configured
git-p4: add failing test for "git-p4: match branches case insensitively if configured"
git-p4: detect/prevent infinite loop in gitCommitByP4Change()
The command line to invoke a "git cat-file" command from inside
"git p4" was not properly quoted to protect a caret and running a
broken command on Windows, which has been corrected.
* mm/p4-unshelve-windows-fix:
p4 unshelve: fix "Not a valid object name HEAD0" on Windows
git p4 unshelve was failing with these errors:
fatal: Not a valid object name HEAD0
Command failed: git cat-file commit HEAD^0
(git version 2.21.0.windows.1, python 2.7.16)
The pOpen call used by git-p4 to invoke the git command can take either a
string or an array as a first argument. The array form is preferred
because platform-specific escaping of special characters will be
handled automatically.(https://docs.python.org/2/library/subprocess.html)
The extractLogMessageFromGitCommit method was, however, using the string
form and so the caret (^) character in the HEAD^0 argument was not being
escaped on Windows. The caret happens to be the escape character, which
is why the git command was receiving HEAD0.
The behaviour can be confirmed by typing ECHO HEAD^0 at the command-
prompt, which emits HEAD0.
The solution is simply to use the array format of passing the command to
fOpen, which is recommended and used in other parts of this code anyway.
Signed-off-by: Mike Mueller <mike.mueller@moodys.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When unshelving a changelist, git-p4 tries to work out the appropriate
parent commit in a given branch (default: HEAD). To do this, it looks
at the state of any pre-existing files in the target Perforce branch,
omitting files added in the shelved changelist. Currently, only files
added (or move targets) are classed as new. However, files integrated
from other branches (i.e. a 'branch' action) also need to be considered
as added, for this purpose.
Signed-off-by: Simon Williams <simon@no-dns-yet.org.uk>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, excluded paths are only handled in the following cases:
* no branch detection;
* branch detection with using clientspec.
However, excluded paths are not respected in case of
branch detection without using clientspec.
Fix this by consulting the list of excluded paths
when splitting files across branches.
Signed-off-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Make sure not to exclude files unintentionally
if exclude paths are specified without a trailing /.
I.e., don't exclude "//depot/file_dont_exclude" if run with "-//depot/file".
Do this by ensuring that paths without a trailing "/" are only matched completely.
Also, abort path search on the first match as a micro-optimization.
Signed-off-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Currently, `cloneExclude` array is being groomed (by removing trailing "...")
on every changeset.
(since `extractFilesFromCommit()` is called on every imported changeset)
As a micro-optimization, do it once while parsing arguments.
Also, prepend "/" and remove trailing "..." at the same time.
Signed-off-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-p4 knows how to handle case insensitivity in file paths
if core.ignorecase is set.
However, when determining a branch for a file,
it still does a case-sensitive prefix match.
This may result in some file changes to be lost on import.
For example, given the following commits
1. add //depot/main/file1
2. add //depot/DirA/file2
3. add //depot/dira/file3
4. add //depot/DirA/file4
and "branchList = main:DirA" branch mapping,
commit 3 will be lost.
So, do branch search case insensitively if running with core.ignorecase set.
Teach splitFilesIntoBranches() to use the p4PathStartsWith() function
for path prefix matches instead of always case-sensitive match.
Signed-off-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git p4" failed to update a shelved change when there were moved
files, which has been corrected.
* ld/git-p4-shelve-update-fix:
git-p4: handle update of moved/copied files when updating a shelve
git-p4: add failing test for shelved CL update involving move/copy
Perforce requires a complete list of files being operated on. If
git is updating an existing shelved changelist, then any files
which are moved or copied were not being added to this list.
Signed-off-by: Luke Diamand <luke@diamand.org>
Acked-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In a perforce setup where login is not required, communication fails
because p4_check_access does not understand the response from the p4
client. Fixed by detecting and ignoring the "info" response.
Signed-off-by: Peter Osterlund <peterosterlund2@gmail.com>
Acked-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The previous git-p4 unshelve support would check for changes
in Perforce to the files being unshelved since the original
shelve, and would complain if any were found.
This was to ensure that the user wouldn't end up with both the
shelved change delta, and some deltas from other changes in their
git commit.
e.g. given fileA:
the
quick
brown
fox
change1: s/the/The/ <- p4 shelve this change
change2: s/fox/Fox/ <- p4 submit this change
git p4 unshelve 1 <- FAIL
This change teaches the P4Unshelve class to always create a parent
commit which matches the P4 tree (for the files being unshelved) at
the point prior to the P4 shelve being created (which is reported
in the p4 description for a shelved changelist).
That then means git-p4 can always create a git commit matching the
P4 shelve that was originally created, without any extra deltas.
The user might still need to use the --origin option though - there
is no way for git-p4 to work out the versions of all of the other
*unchanged* files in the shelve, since this information is not recorded
by Perforce.
Additionally this fixes handling of shelved 'move' operations.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The branch detection code looks for branches under refs/remotes/p4/...
and can end up getting confused if there are unshelved changes in
there as well. This happens in the function p4BranchesInGit().
Instead, put the unshelved changes into refs/remotes/p4-unshelved/<N>.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If deleting or moving a file, sometimes P4 doesn't report the file size.
The code handles this just fine but some logging crashes. Stop this
happening.
There was some earlier discussion on the list about this:
https://public-inbox.org/git/xmqq1sqpp1vv.fsf@gitster.mtv.corp.google.com/
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `p4-pre-submit` hook is executed before git-p4 submits code.
If the hook exits with non-zero value, submit process not start.
Signed-off-by: Chen Bin <chenbin.sh@gmail.com>
Reviewed-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Replace calls to print ... with the function form, print(...), to
allow use with python3 as well as python2.x.
Converted using 2to3 (and some hand-editing).
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In Python3, basestring no longer exists, so use this workaround.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Backticks around a variable are a deprecated alias for repr().
This has been removed in python3, so just use the string
representation instead, which is equivalent.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Python3 does not have the dict.has_key() function, so replace all
such calls with "k in dict". This will still work with python2.6
and python2.7.
Converted using 2to3 (plus some hand-editing)
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The <> string inequality operator (which doesn't seem to be even
documented) no longer exists in python3. Replace with !=.
This still works with python2.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git p4" updates.
* ld/git-p4-updates:
git-p4: auto-size the block
git-p4: narrow the scope of exceptions caught when parsing an int
git-p4: raise exceptions from p4CmdList based on error from p4 server
git-p4: better error reporting when p4 fails
git-p4: add option to disable syncing of p4/master with p4
git-p4: disable-rebase: allow setting this via configuration
git-p4: add options --commit and --disable-rebase
git-p4 originally would fetch changes in one query. On large repos this
could fail because of the limits that Perforce imposes on the number of
items returned and the number of queries in the database.
To fix this, git-p4 learned to query changes in blocks of 512 changes,
However, this can be very slow - if you have a few million changes,
with each chunk taking about a second, it can be an hour or so.
Although it's possible to tune this value manually with the
"--changes-block-size" option, it's far from obvious to ordinary users
that this is what needs doing.
This change alters the block size dynamically by looking for the
specific error messages returned from the Perforce server, and reducing
the block size if the error is seen, either to the limit reported by the
server, or to half the current block size.
That means we can start out with a very large block size, and then let
it automatically drop down to a value that works without error, while
still failing correctly if some other error occurs.
Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
