Topic: Advanced git customization
Audience: Technical (e.g., developers)
TL;DR Getting started with a custom merge driver
Git is a command line tool for version control. It’s default merge drivers are good for most cases, but sometimes you have unique cases where you need to merge differently. Let’s look at how the existing git merge driver, look at example use cases that you may want to merge differently, and then look at how to write your own merge driver.
Definitions
These are a few definitions we’ll need for this article with links to the git documentation for further reading.
- merge strategy: a merge strategy is used for determining which commits to compare between differing git branches. The merge strategy helps determine which is the parent or base commit.
- merge driver: the low-level method for determining which lines or which parts of a file differ between two commits.
Built-in git tools and defaults
Merge Strategies
Git has default merge strategies determining which commits to compare when running a merge. These strategies are examples of the git merge -s <strategy> command and do not refer to the type of git branching strategy. Generally, the defaults work well, and we will not talk about modifying them in this article. Some of the available merge strategies are the following:
- Resolve
- Recursive (default for 3-way merge)
- Octopus (default when passing in more than one additional branch)
- …misc others (see further reading for this section below)
Further reading on git merge strategies:
Merge Drivers
Git has built-in merge drivers for determining differences between commits. Merge drivers can be manually specified in a .gitattributes file per the documentation. Here are the default drivers:
- text: a 3-way merge modification for text files. This is the default method for most files and uses delimiters like “<<<<<<” and “======” to allow the user to sort out conflicts.
- binary: used for binary files and defaults to letting the user sort out conflicts. Some file types are automatically detected as binary. This can be set in
.gitattributeswith the following:foo*.txt binary - union: uses a 3-way file level merge for text files but takes lines from both versions. The user should verify the result.
Here’s a sample using a text driver.
Starting file:
def main():
print('This is the first line.')
print('This is the another line.')
main()File after modifications:
def main():
print('This is the first line.')
print('This line was added later.')
print('This is the another line.')
main()Git diff result using the default text driver:
diff --git a/foo.txt b/foo.txt
index d579b2f..6af485a 100644
--- a/foo.txt
+++ b/foo.txt
@@ -1,5 +1,6 @@
def main():
print('This is the first line.')
+ print('This line was added later.')
print('This is the another line.')
main()In this example, the merge driver simply adds the line in place.
Further reading on default merge drivers:
Customizing Git Merge Drivers
When the default drivers don’t work (json example)
Default merge strategies and drivers work most of the time, but there are instances when they don’t work for the file types that you’re trying to merge. One such example is with multiple people working on .json files.
One such example is with multiple people working on .json files
Let’s look at a scenario when two people are working on the same json file. I set up a git scenario that looks like this. In this example, personA worked on the json file at the same time as personB.
> git log --pretty --oneline --decorate --all --graph
* 33e46b9 (personB) Person B | * 2760f16 (personA) Person A |/ * d71658b (HEAD -> master) Add foo.json * 6916d8d test Base file foo.json (master branch) looks like this:
[
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
}
]Person A’s changes for foo.json (on branch personA):
[
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
},
{
"id": 2,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
}
]Person B’s changes for foo.json (on branch personB):
[
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
},
{
"id": 3,
"foobar": "test3",
"tmp": "test3"
}
]We can see that person A and person B both added an entry into the json file. They logically look fine from a json standpoint: they have separate IDs and the json data looks valid. Let’s try merging personA into master.
> git merge --no-ff personA
Merge made by the 'recursive' strategy.
foo.json | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
> cat .\foo.json [
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
},
{
"id": 2,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
}
]
> git log --pretty --oneline --decorate --all --graph
* 888525e (HEAD -> master) Merge branch 'personA' |\ | * 2760f16 (personA) Person A |/ | * 33e46b9 (personB) Person B |/ * d71658b Add foo.json * 6916d8d test Sweet! We were able to cleanly add the work from person A in personA into master with a merge. (NOTE: We chose --no-ff so that the merge commit would make the history clear). Also note that the personB branch is still separate from master and has not been merged in at this time. Let’s try adding person B’s work in personB branch into master.
First before merging, let’s look at the diff while we’re still on master branch. It looks like it’s going to be messy:
> git diff personB
diff --git a/foo.json b/foo.json
index c54e779..7db125c 100644
--- a/foo.json
+++ b/foo.json
@@ -6,8 +6,9 @@
"tmp": "test2"
},
{
- "id": 3,
- "foobar": "test3",
- "tmp": "test3"
+ "id": 2,
+ "foo": "bar",
+ "foobar": "test",
+ "tmp": "test2"
}
-]
\ No newline at end of file
+]
Let’s try merging and see what happens:
> git merge personB Auto-merging foo.json
CONFLICT (content): Merge conflict in foo.json
Automatic merge failed; fix conflicts and then commit the result.
> cat .\foo.json [
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
},
{
<<<<<<< HEAD
"id": 2,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
=======
"id": 3,
"foobar": "test3",
"tmp": "test3"
>>>>>>> personB
}
]Uh oh. Looks like we’ve got a merge conflict to manually sort out! The resolution in this instance would be to manually modify the merge conflict file and then continue the merge. You can imagine how frustrating this might be with larger json files. The reason this conflict occurred was because the default git text merge driver just did a line-by-line comparison and was not aware of the logical structure of the file.
…the default git text merge driver just did a line-by-line comparison and was not aware of the logical structure of the file.
There are other examples of when the default driver may not work besides json, but regardless of the issue, if you’re able to write a merge driver that is logically aware of the contents of the file then you can make it significantly easier to merge.
…write a merge driver that is logically aware of the contents of the file…
Custom merge drivers
Node.js custom JSON merge driver
Jonatan Pedersen does an excellent job with a json merge driver for Node.js, and you can see the merge driver at this link. You can use this example out-of-the-box for Node.js following his readme from that link.
In his example, you need to update the following files. I’ll show my paths for the example.
.gitattributes:
*.json merge=json.gitconfig:
[merge "json"]
name = custom merge driver for json files
driver = C:\\Users\\Greg Micek\\Documents\\test\\node_modules\\.bin/git-json-merge %A %O %BGit uses the .gitattributes file to determine how to handle specific files. In this instance, git treats all *.json files with the merge driver named “json.” In .gitconfig, we define the driver location with driver = <path>, and we define the name of the driver with [merge "json"]. We could have named the driver whatever we wanted, so [merge "custom_json"] would have been fine too as long as .gitattributes matched with merge=custom_json.
The merge() function from here works with three different commits for a 3-way merge. The git merge strategy is what chooses the three different commits, and the .gitattributes tells git to use the custom merge driver “json.” The three commits results in three different files that git passes in via the command line defined in our driver from .gitconfig:
- %A: our commit from the current branch
- %O: original commit or the base commit from which our current branch and the other branch diverged
- %B: the commit for the other branch
Git passes the file locations for these 3 temporary files in as a command-line argument to git-json-merge as defined in .gitconfig. The temporary file location is a temporary space within the .git/ subdirectory, and git expects the final merge answer to be stored in %A. As long as our merge driver stores the final result in the %A file location, git will consider that merged.
Jonatan Pedersen’s solution takes in the 3 files, reads them as json data structures, and merges them as a json data array. The solution is aware of how json structures work and merges with xdiff rather than git’s default line-by-line comparison.
Let’s look at the same example using this custom driver starting before the 3-way merge and just before we had the conflict with the default method:
> git log --pretty --oneline --decorate --all --graph
* 406fe30 (HEAD -> master) Merge branch 'personA' |\ | * 2760f16 (personA) Person A |/ | * 33e46b9 (personB) Person B |/ * d71658b Add foo.json * 6916d8d test
> git merge personB
Auto-merging foo.json
Merge made by the 'recursive' strategy.
foo.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
> git log --pretty --oneline --decorate --all --graph
* aeb0bf2 (HEAD -> master) Merge branch 'personB' |\ | * 33e46b9 (personB) Person B * | 406fe30 Merge branch 'personA' |\ \ | |/ |/| | * 2760f16 (personA) Person A |/ * d71658b Add foo.json * 6916d8d test Looks like we were able to cleanly merge, and we created a merge commit with aeb0bf2. Let’s look at the contents of our foo.json now:
> cat .\foo.json [
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
},
{
"id": 1,
"foo": "bar",
"foobar": "test",
"tmp": "test2"
}
]Even though we cleanly merged, the contents of our foo.json are not exactly what we’d expect. I’d expect to see array entries with ids 1, 2, and 3. This result is likely the merge driver is looking at the differences, and so I would want to debug the merge driver in .bin/git-json-merge and the resulting code in the npm module a bit further. Regardless, you can see that git accepted the results in the %A file because it treated the custom merge driver as the authoritative figure on the actual 3-way merge!
If we look at the commit in git, we see it stored like this:
> git show commit e0a65d75f839cd646fc6e79e68fa451d7479a2c0 Merge: 406fe30 33e46b9 Author: Greg Micek <omitted@omittedcom> Date: Sun Dec 15 10:11:46 2019 -0600 Merge branch 'personB' diff --cc foo.json index 7db125c,c54e779..2fc1196 --- a/foo.json +++ b/foo.json @@@ -6,9 -6,8 +6,9 @@@ "tmp": "test2" }, { - "id": 2, - "id": 3, - "foobar": "test3", - "tmp": "test3" ++ "id": 1, + "foo": "bar", + "foobar": "test", + "tmp": "test2" } - ] + ] Git used the merge driver to determine the resulting file and then stored a line-by-line diff from the previous file. This is good and just means that the merge driver is only used at the time of doing the merge but not used when looking at diff histories or individual commits. The merge driver treated the contents of the json file as a logical json entry and merged with that in mind.
Your own custom merge driver
You can write a git merge driver in any programming language as long as the system running it is configured to run that language. Just remember that other developers will likely be using your codebase, so you should pick a custom driver that will work out-of-the-box for other developers. This often ends up being a driver in the same programming language as the project (e.g., a Node.js custom merge driver for a Node.js development project).
The Node.js driver functioned on my machine because I already had Node.js installed and because the merge driver begins with #!/usr/bin/env node or with #!/bin/sh on the first line to tell the OS how to execute the file.
can write a git merge driver in any programming language
We could have written a merge driver in BASH directly or in C and compiled it for this operating system. Any of these work as long as the resulting file with the correct merge is stored in the %A path location. Just keep three things in mind when writing your own driver:
- Set your merge driver with
git config merge.json.driver "<path_to_driver> %A %O %B"andgit config merge.json.name "<details>"where you can change “json” out for the name of your driver. - Set your .gitattributes file with the appropriate files and for the driver and your driver name.
- Deal with the merge knowing that %A is the file from your current branch, %O is the file from where the branches diverged, and %B is the file from the other branch. Store the results in file %A.