git/tree-diff.c
Linus Torvalds 6dd4b66fde Fix diffcore-break total breakage
Ok, so on the kernel list, some people noticed that "git log --follow"
doesn't work too well with some files in the x86 merge, because a lot of
files got renamed in very special ways.

In particular, there was a pattern of doing single commits with renames
that looked basically like

 - rename "filename.h" -> "filename_64.h"
 - create new "filename.c" that includes "filename_32.h" or
   "filename_64.h" depending on whether we're 32-bit or 64-bit.

which was preparatory for smushing the two trees together.

Now, there's two issues here:

 - "filename.c" *remained*. Yes, it was a rename, but there was a new file
   created with the old name in the same commit. This was important,
   because we wanted each commit to compile properly, so that it was
   bisectable, so splitting the rename into one commit and the "create
   helper file" into another was *not* an option.

   So we need to break associations where the contents change too much.
   Fine. We have the -B flag for that. When we break things up, then the
   rename detection will be able to figure out whether there are better
   alternatives.

 - "git log --follow" didn't with with -B.

Now, the second case was really simple: we use a different "diffopt"
structure for the rename detection than the basic one (which we use for
showing the diffs). So that second case is trivially fixed by a trivial
one-liner that just copies the break_opt values from the "real" diffopts
to the one used for rename following. So now "git log -B --follow" works
fine:

	diff --git a/tree-diff.c b/tree-diff.c
	index 26bdbdd..7c261fd 100644
	--- a/tree-diff.c
	+++ b/tree-diff.c
	@@ -319,6 +319,7 @@ static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, co
	 	diff_opts.detect_rename = DIFF_DETECT_RENAME;
	 	diff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;
	 	diff_opts.single_follow = opt->paths[0];
	+	diff_opts.break_opt = opt->break_opt;
	 	paths[0] = NULL;
	 	diff_tree_setup_paths(paths, &diff_opts);
	 	if (diff_setup_done(&diff_opts) < 0)

however, the end result does *not* work. Because our diffcore-break.c
logic is totally bogus!

In particular:

 - it used to do

	if (base_size < MINIMUM_BREAK_SIZE)
		return 0; /* we do not break too small filepair */

   which basically says "don't bother to break small files". But that
   "base_size" is the *smaller* of the two sizes, which means that if some
   large file was rewritten into one that just includes another file, we
   would look at the (small) result, and decide that it's smaller than the
   break size, so it cannot be worth it to break it up! Even if the other
   side was ten times bigger and looked *nothing* like the samell file!

   That's clearly bogus. I replaced "base_size" with "max_size", so that
   we compare the *bigger* of the filepair with the break size.

 - It calculated a "merge_score", which was the score needed to merge it
   back together if nothing else wanted it. But even if it was *so*
   different that we would never want to merge it back, we wouldn't
   consider it a break! That makes no sense. So I added

	if (*merge_score_p > break_score)
		return 1;

   to make it clear that if we wouldn't want to merge it at the end, it
   was *definitely* a break.

 - It compared the whole "extent of damage", counting all inserts and
   deletes, but it based this score on the "base_size", and generated the
   damage score with

	delta_size = src_removed + literal_added;
	damage_score = delta_size * MAX_SCORE / base_size;

   but that makes no sense either, since quite often, this will result in
   a number that is *bigger* than MAX_SCORE! Why? Because base_size is
   (again) the smaller of the two files we compare, and when you start out
   from a small file and add a lot (or start out from a large file and
   remove a lot), the base_size is going to be much smaller than the
   damage!

   Again, the fix was to replace "base_size" with "max_size", at which
   point the damage actually becomes a sane percentage of the whole.

With these changes in place, not only does "git log -B --follow" work for
the case that triggered this in the first place, ie now

	git log -B --follow arch/x86/kernel/vmlinux_64.lds.S

actually gives reasonable results. But I also wanted to verify it in
general, by doing a full-history

	git log --stat -B -C

on my kernel tree with the old code and the new code.

There's some tweaking to be done, but generally, the new code generates
much better results wrt breaking up files (and then finding better rename
candidates). Here's a few examples of the "--stat" output:

 - This:
	include/asm-x86/Kbuild        |    2 -
	include/asm-x86/debugreg.h    |   79 +++++++++++++++++++++++++++++++++++------
	include/asm-x86/debugreg_32.h |   64 ---------------------------------
	include/asm-x86/debugreg_64.h |   65 ---------------------------------
	4 files changed, 68 insertions(+), 142 deletions(-)

      Becomes:

	include/asm-x86/Kbuild                        |    2 -
	include/asm-x86/{debugreg_64.h => debugreg.h} |    9 +++-
	include/asm-x86/debugreg_32.h                 |   64 -------------------------
	3 files changed, 7 insertions(+), 68 deletions(-)

 - This:
	include/asm-x86/bug.h    |   41 +++++++++++++++++++++++++++++++++++++++--
	include/asm-x86/bug_32.h |   37 -------------------------------------
	include/asm-x86/bug_64.h |   34 ----------------------------------
	3 files changed, 39 insertions(+), 73 deletions(-)

      Becomes

	include/asm-x86/{bug_64.h => bug.h} |   20 +++++++++++++-----
	include/asm-x86/bug_32.h            |   37 -----------------------------------
	2 files changed, 14 insertions(+), 43 deletions(-)

Now, in some other cases, it does actually turn a rename into a real
"delete+create" pair, and then the diff is usually bigger, so truth in
advertizing: it doesn't always generate a nicer diff. But for what -B was
meant for, I think this is a big improvement, and I suspect those cases
where it generates a bigger diff are tweakable.

So I think this diff fixes a real bug, but we might still want to tweak
the default values and perhaps the exact rules for when a break happens.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-21 01:59:42 -04:00

444 lines
11 KiB
C

/*
* Helper functions for tree diff generation
*/
#include "cache.h"
#include "diff.h"
#include "diffcore.h"
#include "tree.h"
static char *malloc_base(const char *base, int baselen, const char *path, int pathlen)
{
char *newbase = xmalloc(baselen + pathlen + 2);
memcpy(newbase, base, baselen);
memcpy(newbase + baselen, path, pathlen);
memcpy(newbase + baselen + pathlen, "/", 2);
return newbase;
}
static void show_entry(struct diff_options *opt, const char *prefix, struct tree_desc *desc,
const char *base, int baselen);
static int compare_tree_entry(struct tree_desc *t1, struct tree_desc *t2, const char *base, int baselen, struct diff_options *opt)
{
unsigned mode1, mode2;
const char *path1, *path2;
const unsigned char *sha1, *sha2;
int cmp, pathlen1, pathlen2;
sha1 = tree_entry_extract(t1, &path1, &mode1);
sha2 = tree_entry_extract(t2, &path2, &mode2);
pathlen1 = tree_entry_len(path1, sha1);
pathlen2 = tree_entry_len(path2, sha2);
cmp = base_name_compare(path1, pathlen1, mode1, path2, pathlen2, mode2);
if (cmp < 0) {
show_entry(opt, "-", t1, base, baselen);
return -1;
}
if (cmp > 0) {
show_entry(opt, "+", t2, base, baselen);
return 1;
}
if (!opt->find_copies_harder && !hashcmp(sha1, sha2) && mode1 == mode2)
return 0;
/*
* If the filemode has changed to/from a directory from/to a regular
* file, we need to consider it a remove and an add.
*/
if (S_ISDIR(mode1) != S_ISDIR(mode2)) {
show_entry(opt, "-", t1, base, baselen);
show_entry(opt, "+", t2, base, baselen);
return 0;
}
if (opt->recursive && S_ISDIR(mode1)) {
int retval;
char *newbase = malloc_base(base, baselen, path1, pathlen1);
if (opt->tree_in_recursive)
opt->change(opt, mode1, mode2,
sha1, sha2, base, path1);
retval = diff_tree_sha1(sha1, sha2, newbase, opt);
free(newbase);
return retval;
}
opt->change(opt, mode1, mode2, sha1, sha2, base, path1);
return 0;
}
/*
* Is a tree entry interesting given the pathspec we have?
*
* Return:
* - 2 for "yes, and all subsequent entries will be"
* - 1 for yes
* - zero for no
* - negative for "no, and no subsequent entries will be either"
*/
static int tree_entry_interesting(struct tree_desc *desc, const char *base, int baselen, struct diff_options *opt)
{
const char *path;
const unsigned char *sha1;
unsigned mode;
int i;
int pathlen;
int never_interesting = -1;
if (!opt->nr_paths)
return 1;
sha1 = tree_entry_extract(desc, &path, &mode);
pathlen = tree_entry_len(path, sha1);
for (i = 0; i < opt->nr_paths; i++) {
const char *match = opt->paths[i];
int matchlen = opt->pathlens[i];
int m = -1; /* signals that we haven't called strncmp() */
if (baselen >= matchlen) {
/* If it doesn't match, move along... */
if (strncmp(base, match, matchlen))
continue;
/*
* The base is a subdirectory of a path which
* was specified, so all of them are interesting.
*/
return 2;
}
/* Does the base match? */
if (strncmp(base, match, baselen))
continue;
match += baselen;
matchlen -= baselen;
if (never_interesting) {
/*
* We have not seen any match that sorts later
* than the current path.
*/
/*
* Does match sort strictly earlier than path
* with their common parts?
*/
m = strncmp(match, path,
(matchlen < pathlen) ? matchlen : pathlen);
if (m < 0)
continue;
/*
* If we come here even once, that means there is at
* least one pathspec that would sort equal to or
* later than the path we are currently looking at.
* In other words, if we have never reached this point
* after iterating all pathspecs, it means all
* pathspecs are either outside of base, or inside the
* base but sorts strictly earlier than the current
* one. In either case, they will never match the
* subsequent entries. In such a case, we initialized
* the variable to -1 and that is what will be
* returned, allowing the caller to terminate early.
*/
never_interesting = 0;
}
if (pathlen > matchlen)
continue;
if (matchlen > pathlen) {
if (match[pathlen] != '/')
continue;
if (!S_ISDIR(mode))
continue;
}
if (m == -1)
/*
* we cheated and did not do strncmp(), so we do
* that here.
*/
m = strncmp(match, path, pathlen);
/*
* If common part matched earlier then it is a hit,
* because we rejected the case where path is not a
* leading directory and is shorter than match.
*/
if (!m)
return 1;
}
return never_interesting; /* No matches */
}
/* A whole sub-tree went away or appeared */
static void show_tree(struct diff_options *opt, const char *prefix, struct tree_desc *desc, const char *base, int baselen)
{
int all_interesting = 0;
while (desc->size) {
int show;
if (all_interesting)
show = 1;
else {
show = tree_entry_interesting(desc, base, baselen,
opt);
if (show == 2)
all_interesting = 1;
}
if (show < 0)
break;
if (show)
show_entry(opt, prefix, desc, base, baselen);
update_tree_entry(desc);
}
}
/* A file entry went away or appeared */
static void show_entry(struct diff_options *opt, const char *prefix, struct tree_desc *desc,
const char *base, int baselen)
{
unsigned mode;
const char *path;
const unsigned char *sha1 = tree_entry_extract(desc, &path, &mode);
if (opt->recursive && S_ISDIR(mode)) {
enum object_type type;
int pathlen = tree_entry_len(path, sha1);
char *newbase = malloc_base(base, baselen, path, pathlen);
struct tree_desc inner;
void *tree;
unsigned long size;
tree = read_sha1_file(sha1, &type, &size);
if (!tree || type != OBJ_TREE)
die("corrupt tree sha %s", sha1_to_hex(sha1));
init_tree_desc(&inner, tree, size);
show_tree(opt, prefix, &inner, newbase, baselen + 1 + pathlen);
free(tree);
free(newbase);
} else {
opt->add_remove(opt, prefix[0], mode, sha1, base, path);
}
}
static void skip_uninteresting(struct tree_desc *t, const char *base, int baselen, struct diff_options *opt)
{
int all_interesting = 0;
while (t->size) {
int show;
if (all_interesting)
show = 1;
else {
show = tree_entry_interesting(t, base, baselen, opt);
if (show == 2)
all_interesting = 1;
}
if (!show) {
update_tree_entry(t);
continue;
}
/* Skip it all? */
if (show < 0)
t->size = 0;
return;
}
}
int diff_tree(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
{
int baselen = strlen(base);
for (;;) {
if (opt->quiet && opt->has_changes)
break;
if (opt->nr_paths) {
skip_uninteresting(t1, base, baselen, opt);
skip_uninteresting(t2, base, baselen, opt);
}
if (!t1->size) {
if (!t2->size)
break;
show_entry(opt, "+", t2, base, baselen);
update_tree_entry(t2);
continue;
}
if (!t2->size) {
show_entry(opt, "-", t1, base, baselen);
update_tree_entry(t1);
continue;
}
switch (compare_tree_entry(t1, t2, base, baselen, opt)) {
case -1:
update_tree_entry(t1);
continue;
case 0:
update_tree_entry(t1);
/* Fallthrough */
case 1:
update_tree_entry(t2);
continue;
}
die("git-diff-tree: internal error");
}
return 0;
}
/*
* Does it look like the resulting diff might be due to a rename?
* - single entry
* - not a valid previous file
*/
static inline int diff_might_be_rename(void)
{
return diff_queued_diff.nr == 1 &&
!DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
}
static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
{
struct diff_options diff_opts;
struct diff_queue_struct *q = &diff_queued_diff;
struct diff_filepair *choice;
const char *paths[1];
int i;
/* Remove the file creation entry from the diff queue, and remember it */
choice = q->queue[0];
q->nr = 0;
diff_setup(&diff_opts);
diff_opts.recursive = 1;
diff_opts.detect_rename = DIFF_DETECT_RENAME;
diff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;
diff_opts.single_follow = opt->paths[0];
diff_opts.break_opt = opt->break_opt;
paths[0] = NULL;
diff_tree_setup_paths(paths, &diff_opts);
if (diff_setup_done(&diff_opts) < 0)
die("unable to set up diff options to follow renames");
diff_tree(t1, t2, base, &diff_opts);
diffcore_std(&diff_opts);
/* Go through the new set of filepairing, and see if we find a more interesting one */
for (i = 0; i < q->nr; i++) {
struct diff_filepair *p = q->queue[i];
/*
* Found a source? Not only do we use that for the new
* diff_queued_diff, we will also use that as the path in
* the future!
*/
if ((p->status == 'R' || p->status == 'C') && !strcmp(p->two->path, opt->paths[0])) {
/* Switch the file-pairs around */
q->queue[i] = choice;
choice = p;
/* Update the path we use from now on.. */
opt->paths[0] = xstrdup(p->one->path);
diff_tree_setup_paths(opt->paths, opt);
break;
}
}
/*
* Then, discard all the non-relevane file pairs...
*/
for (i = 0; i < q->nr; i++) {
struct diff_filepair *p = q->queue[i];
diff_free_filepair(p);
}
/*
* .. and re-instate the one we want (which might be either the
* original one, or the rename/copy we found)
*/
q->queue[0] = choice;
q->nr = 1;
}
int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
{
void *tree1, *tree2;
struct tree_desc t1, t2;
unsigned long size1, size2;
int retval;
tree1 = read_object_with_reference(old, tree_type, &size1, NULL);
if (!tree1)
die("unable to read source tree (%s)", sha1_to_hex(old));
tree2 = read_object_with_reference(new, tree_type, &size2, NULL);
if (!tree2)
die("unable to read destination tree (%s)", sha1_to_hex(new));
init_tree_desc(&t1, tree1, size1);
init_tree_desc(&t2, tree2, size2);
retval = diff_tree(&t1, &t2, base, opt);
if (opt->follow_renames && diff_might_be_rename()) {
init_tree_desc(&t1, tree1, size1);
init_tree_desc(&t2, tree2, size2);
try_to_follow_renames(&t1, &t2, base, opt);
}
free(tree1);
free(tree2);
return retval;
}
int diff_root_tree_sha1(const unsigned char *new, const char *base, struct diff_options *opt)
{
int retval;
void *tree;
unsigned long size;
struct tree_desc empty, real;
tree = read_object_with_reference(new, tree_type, &size, NULL);
if (!tree)
die("unable to read root tree (%s)", sha1_to_hex(new));
init_tree_desc(&real, tree, size);
init_tree_desc(&empty, "", 0);
retval = diff_tree(&empty, &real, base, opt);
free(tree);
return retval;
}
static int count_paths(const char **paths)
{
int i = 0;
while (*paths++)
i++;
return i;
}
void diff_tree_release_paths(struct diff_options *opt)
{
free(opt->pathlens);
}
void diff_tree_setup_paths(const char **p, struct diff_options *opt)
{
opt->nr_paths = 0;
opt->pathlens = NULL;
opt->paths = NULL;
if (p) {
int i;
opt->paths = p;
opt->nr_paths = count_paths(p);
if (opt->nr_paths == 0) {
opt->pathlens = NULL;
return;
}
opt->pathlens = xmalloc(opt->nr_paths * sizeof(int));
for (i=0; i < opt->nr_paths; i++)
opt->pathlens[i] = strlen(p[i]);
}
}