cmd/go: add notary simulation and GONOVERIFY support

As an experiment to better understand the impact of
having an authoritative source of truth for module hashes
before the real notary is available, this CL adds the basic
notary authorization checks using a partial whitelist of
known go.sum values for popular modules.

In addition to the temporary whitelist, this CL adds code
implementing $GONOVERIFY, a new 'go help modules-auth',
and clearer error messages for verification mismatches.

See #25530 for notary proposal.
Filed #30601 to remove whitelist when notary lands.

Change-Id: Ibcb6ac39c5e60455edf003d8c20af6932aeb7e88
Reviewed-on: https://go-review.googlesource.com/c/go/+/165380
Reviewed-by: Bryan C. Mills <bcmills@google.com>
This commit is contained in:
Russ Cox 2019-02-26 00:16:07 -05:00
parent a6436a5655
commit fe954ea1e2
11 changed files with 4039 additions and 33 deletions

View file

@ -48,6 +48,7 @@
// modules modules, module versions, and more
// module-get module-aware go get
// packages package lists and patterns
// module-auth module authentication using go.sum
// testflag testing flags
// testfunc testing functions
//
@ -2414,19 +2415,9 @@
//
// Module downloading and verification
//
// The go command maintains, in the main module's root directory alongside
// go.mod, a file named go.sum containing the expected cryptographic checksums
// of the content of specific module versions. Each time a dependency is
// used, its checksum is added to go.sum if missing or else required to match
// the existing entry in go.sum.
//
// The go command maintains a cache of downloaded packages and computes
// and records the cryptographic checksum of each package at download time.
// In normal operation, the go command checks these pre-computed checksums
// against the main module's go.sum file, instead of recomputing them on
// each command invocation. The 'go mod verify' command checks that
// the cached copies of module downloads still match both their recorded
// checksums and the entries in go.sum.
// The go command checks downloads against known checksums,
// to detect unexpected changes in the content of any specific module
// version from one day to the next. See 'go help module-auth' for details.
//
// The go command can fetch modules from a proxy instead of connecting
// to source control systems directly, according to the setting of the GOPROXY
@ -2643,6 +2634,87 @@
// by the go tool, as are directories named "testdata".
//
//
// Module authentication using go.sum
//
// The go command tries to authenticate every downloaded module,
// checking that the bits downloaded for a specific module version today
// match bits downloaded yesterday. This ensures repeatable builds
// and detects introduction of unexpected changes, malicious or not.
//
// In each module's root, alongside go.mod, the go command maintains
// a file named go.sum containing the cryptographic checksums of the
// module's dependencies.
//
// The form of each line in go.sum is three fields:
//
// <module> <version>[/go.mod] <hash>
//
// Each known module version results in two lines in the go.sum file.
// The first line gives the hash of the module version's file tree.
// The second line appends "/go.mod" to the version and gives the hash
// of only the module version's (possibly synthesized) go.mod file.
// The go.mod-only hash allows downloading and authenticating a
// module version's go.mod file, which is needed to compute the
// dependency graph, without also downloading all the module's source code.
//
// The hash begins with an algorithm prefix of the form "h<N>:".
// The only defined algorithm prefix is "h1:", which uses SHA-256.
//
// Module authentication failures
//
// The go command maintains a cache of downloaded packages and computes
// and records the cryptographic checksum of each package at download time.
// In normal operation, the go command checks the main module's go.sum file
// against these precomputed checksums instead of recomputing them on
// each command invocation. The 'go mod verify' command checks that
// the cached copies of module downloads still match both their recorded
// checksums and the entries in go.sum.
//
// In day-to-day development, the checksum of a given module version
// should never change. Each time a dependency is used by a given main
// module, the go command checks its local cached copy, freshly
// downloaded or not, against the main module's go.sum. If the checksums
// don't match, the go command reports the mismatch as a security error
// and refuses to run the build. When this happens, proceed with caution:
// code changing unexpectedly means today's build will not match
// yesterday's, and the unexpected change may not be beneficial.
//
// If the go command reports a mismatch in go.sum, the downloaded code
// for the reported module version does not match the one used in a
// previous build of the main module. It is important at that point
// to find out what the right checksum should be, to decide whether
// go.sum is wrong or the downloaded code is wrong. Usually go.sum is right:
// you want to use the same code you used yesterday.
//
// If a downloaded module is not yet included in go.sum and it is a publicly
// available module, the go command consults the Go notary server to fetch
// the expected go.sum lines. If the downloaded code does not match those
// lines, the go command reports the mismatch and exits. Note that the
// notary is not consulted for module versions already listed in go.sum.
//
// The GONOVERIFY environment variable is a comma-separated list of
// patterns (in the syntax of Go's path.Match) of module path prefixes
// that should not be verified using the notary. For example,
//
// GONOVERIFY=*.corp.example.com,rsc.io/private
//
// disables notary verification for modules with path prefixes matching
// either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
// and "rsc.io/private/quux".
//
// As a special case, if GONOVERIFY is set to "off", or if "go get" was invoked
// with the -insecure flag, the notary is never consulted, but note that this
// defeats the security provided by the notary. A better course of action is
// to set a narrower GONOVERIFY and, in the case of go.sum mismatches,
// investigate why the code downloaded code differs from what was
// downloaded yesterday.
//
// NOTE: Early in the Go 1.13 dev cycle, the notary is being simulated by
// a whitelist of known hashes for popular Go modules, to expose any
// problems arising from knowing the expected hashes.
// TODO(rsc): This note should be removed once the real notary is used instead. See #30601.
//
//
// Testing flags
//
// The 'go test' command takes both flags that apply to 'go test' itself

View file

@ -400,25 +400,62 @@ func checkOneSum(mod module.Version, h string) {
defer goSum.mu.Unlock()
if initGoSum() {
checkOneSumLocked(mod, h)
} else if useNotary(mod) {
checkNotarySum(mod, h)
}
}
func checkOneSumLocked(mod module.Version, h string) {
goSum.checked[modSum{mod, h}] = true
for _, vh := range goSum.m[mod] {
checkGoSum := func() bool {
for _, vh := range goSum.m[mod] {
if h == vh {
return true
}
if strings.HasPrefix(vh, "h1:") {
base.Fatalf("verifying %s@%s: checksum mismatch\n\tdownloaded: %v\n\tgo.sum: %v"+goSumMismatch, mod.Path, mod.Version, h, vh)
}
}
return false
}
if checkGoSum() {
return
}
if useNotary(mod) {
goSum.mu.Unlock()
checkNotarySum(mod, h) // dies if h is wrong
goSum.mu.Lock()
// Because we dropped the lock, a racing goroutine
// may have already added this entry to go.sum.
// Check again.
if checkGoSum() {
return
}
}
if len(goSum.m[mod]) > 0 {
fmt.Fprintf(os.Stderr, "warning: verifying %s@%s: unknown hashes in go.sum: %v; adding %v"+hashVersionMismatch, mod.Path, mod.Version, strings.Join(goSum.m[mod], ", "), h)
}
goSum.m[mod] = append(goSum.m[mod], h)
goSum.dirty = true
}
// checkNotarySum checks the mod, h pair against the Go notary.
// It calls base.Fatalf if the hash is to be rejected.
func checkNotarySum(mod module.Version, h string) {
hashes := notaryHashes(mod)
for _, vh := range hashes {
if h == vh {
return
}
if strings.HasPrefix(vh, "h1:") {
base.Fatalf("verifying %s@%s: checksum mismatch\n\tdownloaded: %v\n\tgo.sum: %v", mod.Path, mod.Version, h, vh)
base.Fatalf("verifying %s@%s: checksum mismatch\n\tdownloaded: %v\n\tnotary: %v"+notarySumMismatch, mod.Path, mod.Version, h, vh)
}
}
if len(goSum.m[mod]) > 0 {
fmt.Fprintf(os.Stderr, "warning: verifying %s@%s: unknown hashes in go.sum: %v; adding %v", mod.Path, mod.Version, strings.Join(goSum.m[mod], ", "), h)
}
goSum.m[mod] = append(goSum.m[mod], h)
goSum.dirty = true
}
// Sum returns the checksum for the downloaded copy of the given module,
@ -539,3 +576,117 @@ func TrimGoSum(keep map[module.Version]bool) {
}
}
}
const goSumMismatch = `
SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.
For more information, see 'go help module-auth'.
`
const notarySumMismatch = `
SECURITY ERROR
This download does NOT match the expected download known to the notary.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.
For more information, see 'go help module-auth'.
`
const hashVersionMismatch = `
SECURITY WARNING
This download is listed in go.sum, but using an unknown hash algorithm.
The download cannot be verified.
For more information, see 'go help module-auth'.
`
var HelpSum = &base.Command{
UsageLine: "module-auth",
Short: "module authentication using go.sum",
Long: `
The go command tries to authenticate every downloaded module,
checking that the bits downloaded for a specific module version today
match bits downloaded yesterday. This ensures repeatable builds
and detects introduction of unexpected changes, malicious or not.
In each module's root, alongside go.mod, the go command maintains
a file named go.sum containing the cryptographic checksums of the
module's dependencies.
The form of each line in go.sum is three fields:
<module> <version>[/go.mod] <hash>
Each known module version results in two lines in the go.sum file.
The first line gives the hash of the module version's file tree.
The second line appends "/go.mod" to the version and gives the hash
of only the module version's (possibly synthesized) go.mod file.
The go.mod-only hash allows downloading and authenticating a
module version's go.mod file, which is needed to compute the
dependency graph, without also downloading all the module's source code.
The hash begins with an algorithm prefix of the form "h<N>:".
The only defined algorithm prefix is "h1:", which uses SHA-256.
Module authentication failures
The go command maintains a cache of downloaded packages and computes
and records the cryptographic checksum of each package at download time.
In normal operation, the go command checks the main module's go.sum file
against these precomputed checksums instead of recomputing them on
each command invocation. The 'go mod verify' command checks that
the cached copies of module downloads still match both their recorded
checksums and the entries in go.sum.
In day-to-day development, the checksum of a given module version
should never change. Each time a dependency is used by a given main
module, the go command checks its local cached copy, freshly
downloaded or not, against the main module's go.sum. If the checksums
don't match, the go command reports the mismatch as a security error
and refuses to run the build. When this happens, proceed with caution:
code changing unexpectedly means today's build will not match
yesterday's, and the unexpected change may not be beneficial.
If the go command reports a mismatch in go.sum, the downloaded code
for the reported module version does not match the one used in a
previous build of the main module. It is important at that point
to find out what the right checksum should be, to decide whether
go.sum is wrong or the downloaded code is wrong. Usually go.sum is right:
you want to use the same code you used yesterday.
If a downloaded module is not yet included in go.sum and it is a publicly
available module, the go command consults the Go notary server to fetch
the expected go.sum lines. If the downloaded code does not match those
lines, the go command reports the mismatch and exits. Note that the
notary is not consulted for module versions already listed in go.sum.
The GONOVERIFY environment variable is a comma-separated list of
patterns (in the syntax of Go's path.Match) of module path prefixes
that should not be verified using the notary. For example,
GONOVERIFY=*.corp.example.com,rsc.io/private
disables notary verification for modules with path prefixes matching
either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
and "rsc.io/private/quux".
As a special case, if GONOVERIFY is set to "off", or if "go get" was invoked
with the -insecure flag, the notary is never consulted, but note that this
defeats the security provided by the notary. A better course of action is
to set a narrower GONOVERIFY and, in the case of go.sum mismatches,
investigate why the code downloaded code differs from what was
downloaded yesterday.
NOTE: Early in the Go 1.13 dev cycle, the notary is being simulated by
a whitelist of known hashes for popular Go modules, to expose any
problems arising from knowing the expected hashes.
TODO(rsc): This note should be removed once the real notary is used instead. See #30601.
`,
}

View file

@ -0,0 +1,156 @@
// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package modfetch
import (
"fmt"
"os"
pathpkg "path"
"strings"
"cmd/go/internal/base"
"cmd/go/internal/get"
"cmd/go/internal/module"
)
// notaryShouldVerify reports whether the notary should be used for path,
// given the GONOVERIFY setting.
func notaryShouldVerify(path, GONOVERIFY string) (bool, error) {
if GONOVERIFY == "off" {
return false, nil
}
for GONOVERIFY != "" {
var pattern string
i := strings.Index(GONOVERIFY, ",")
if i < 0 {
pattern, GONOVERIFY = GONOVERIFY, ""
} else {
pattern, GONOVERIFY = GONOVERIFY[:i], GONOVERIFY[i+1:]
}
if pattern == "" {
continue
}
n := strings.Count(pattern, "/") + 1
prefix := path
for i := 0; i < len(prefix); i++ {
if prefix[i] == '/' {
n--
if n == 0 {
prefix = prefix[:i]
break
}
}
}
if n > 1 {
continue
}
matched, err := pathpkg.Match(pattern, prefix)
if err != nil {
// Note that path.Match does not guarantee to detect
// pattern errors. It usually depends on whether the
// given text (prefix in this case) matches enough of
// the pattern to reach the error. So this will only
// trigger on malformed patterns that are "close enough" to prefix.
return false, fmt.Errorf("malformed GONOVERIFY pattern: %s", pattern)
}
if matched {
return false, nil
}
}
return true, nil
}
// useNotary reports whether to use the notary for the given module.
func useNotary(mod module.Version) bool {
if get.Insecure {
return false
}
wantNotary, err := notaryShouldVerify(mod.Path, os.Getenv("GONOVERIFY"))
if err != nil {
base.Fatalf("%v", err)
}
// TODO(rsc): return wantNotary. See #30601.
//
// This code must be deleted when goSumPin is deleted.
// goSumPin is only a partial notary simulation, so we don't return true from
// useNotary when we don't have an entry for that module.
// This differs from the real notary, which will be authoritative
// for everything it is asked for. When goSumPin is removed,
// this function body should end here with "return wantNotary".
_ = goSumPin // read TODO above if goSumPin is gone
return wantNotary && notaryHashes(mod) != nil
}
// notaryHashes fetches hashes for mod from the notary.
// The caller must have checked that useNotary(mod) is true.
func notaryHashes(mod module.Version) []string {
// For testing, hard-code this result.
if mod.Path == "rsc.io/badsum" {
switch mod.Version {
case "v1.0.0":
return []string{"h1:6/o+QJfe6mFSNuegDihphabcvR94anXQk/qq7Enr19U="}
case "v1.0.0/go.mod":
return []string{"h1:avOsLUJaHavllihBU9qCTW37z64ypkZjqZg8O16JLVY="}
case "v1.0.1":
return []string{"h1:S7G9Ikksx7htnFivDrUOv8xI0kIdAf15gLt97Gy//Zk="}
case "v1.0.1/go.mod":
return []string{"h1:avOsLUJaHavllihBU9qCTW37z64ypkZjqZg8O16JLVY="}
}
}
// Until the notary is ready, simulate contacting the notary by
// looking in the known hash list goSumPin in pin.go.
// Entries not listed in goSumPin are treated as "not for the notary",
// but once the real notary is added, they should be treated as
// "failed to verify".
//
// TODO(rsc): Once the notary is ready, this function should be
// rewritten to use it. See #30601.
i := strings.Index(goSumPin, "\n"+mod.Path+"\n")
if i < 0 {
return nil
}
wantGoSum := false
if strings.HasSuffix(mod.Version, "/go.mod") {
wantGoSum = true
mod.Version = strings.TrimSuffix(mod.Version, "/go.mod")
}
versions := goSumPin[i+1+len(mod.Path)+1:]
var lastSum, lastGoSum string
for {
i := strings.Index(versions, "\n")
if i < 0 {
break
}
line := versions[:i]
versions = versions[i+1:]
if !strings.HasPrefix(line, " ") {
break
}
f := strings.Fields(line)
if len(f) < 3 {
break
}
if f[1] == "-" {
f[1] = lastSum
} else {
lastSum = f[1]
}
if f[2] == "-" {
f[2] = lastGoSum
} else {
lastGoSum = f[2]
}
if f[0] == mod.Version {
if wantGoSum {
return []string{f[2]}
}
return []string{f[1]}
}
}
return nil
}

View file

@ -0,0 +1,47 @@
// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package modfetch
import (
"testing"
)
var notaryShouldVerifyTests = []struct {
modPath string
GONOVERIFY string
result int // -1 = bad GONOVERIFY, 0 = wantNotary=false, 1 = wantNotary=true
}{
{"anything", "off", 0},
{"anything", "", 1},
{"anything", ",", 1},
{"anything", ",foo,", 1},
{"anything", "[malformed", -1},
{"anything", "malformed[", 1},
{"my.corp.example.com", "*.[c]orp.*", 0},
{"my.corp.example.com/foo", "*.c[^a]rp.*", 0},
{"my.corp.example.com", "*.corp.*,bar.com", 0},
{"my.corp.example.com/foo", "*.corp.*,bar.com", 0},
{"my.corp.example.com", "bar.com,*.corp.*", 0},
{"my.corp.example.com/foo", "bar.com,*.corp.*", 0},
{"bar.com", "*.corp.*", 1},
{"bar.com/foo", "*.corp.*", 1},
{"bar.com", "*.corp.*,bar.com", 0},
{"bar.com/foo", "*.corp.*,bar.com", 0},
{"bar.com", "bar.com,*.corp.*", 0},
{"bar.com/foo", "bar.com,*.corp.*", 0},
}
func TestNotaryShouldVerify(t *testing.T) {
for _, tt := range notaryShouldVerifyTests {
wantNotary, err := notaryShouldVerify(tt.modPath, tt.GONOVERIFY)
if wantNotary != (tt.result > 0) || (err != nil) != (tt.result < 0) {
wantErr := "nil"
if tt.result < 0 {
wantErr = "non-nil error"
}
t.Errorf("notaryShouldVerify(%q, %q) = %v, %v, want %v, %s", tt.modPath, tt.GONOVERIFY, wantNotary, err, tt.result > 0, wantErr)
}
}
}

File diff suppressed because it is too large Load diff

View file

@ -337,19 +337,9 @@ module file trees.
Module downloading and verification
The go command maintains, in the main module's root directory alongside
go.mod, a file named go.sum containing the expected cryptographic checksums
of the content of specific module versions. Each time a dependency is
used, its checksum is added to go.sum if missing or else required to match
the existing entry in go.sum.
The go command maintains a cache of downloaded packages and computes
and records the cryptographic checksum of each package at download time.
In normal operation, the go command checks these pre-computed checksums
against the main module's go.sum file, instead of recomputing them on
each command invocation. The 'go mod verify' command checks that
the cached copies of module downloads still match both their recorded
checksums and the entries in go.sum.
The go command checks downloads against known checksums,
to detect unexpected changes in the content of any specific module
version from one day to the next. See 'go help module-auth' for details.
The go command can fetch modules from a proxy instead of connecting
to source control systems directly, according to the setting of the GOPROXY

View file

@ -72,6 +72,7 @@ func init() {
modload.HelpModules,
modget.HelpModuleGet,
help.HelpPackages,
modfetch.HelpSum,
test.HelpTestflag,
test.HelpTestfunc,
}

View file

@ -109,6 +109,7 @@ func (ts *testScript) setup() {
"GOPATH=" + filepath.Join(ts.workdir, "gopath"),
"GOPROXY=" + proxyURL,
"GOROOT=" + testGOROOT,
"GONOVERIFY=*",
tempEnvName() + "=" + filepath.Join(ts.workdir, "tmp"),
"devnull=" + os.DevNull,
"goversion=" + goVersion(ts),

View file

@ -0,0 +1,14 @@
rsc.io/badsum@v1.0.0
This module would match the hard-coded hash for rsc.io/badsum v1.0.0
in modfetch/notary.go if not for the "break hash" line.
-- .mod --
module "rsc.io/badsum"
-- .info --
{"Version":"v1.0.0","Time":"2018-02-14T00:45:20Z"}
-- go.mod --
module "rsc.io/badsum"
-- badsum.go --
package badsum
// break hash

View file

@ -0,0 +1,14 @@
rsc.io/badsum@v1.0.1
This module would match the hard-coded hash for rsc.io/badsum v1.0.1/go.mod
in modfetch/notary.go if not for the "break hash" line.
-- .mod --
module "rsc.io/badsum"
# break hash
-- .info --
{"Version":"v1.0.1","Time":"2018-02-14T00:45:20Z"}
-- go.mod --
module "rsc.io/badsum"
-- badsum.go --
package badsum

View file

@ -0,0 +1,73 @@
env GO111MODULE=on
env GONOVERIFY= # default in test scripts is *
# notary should reject rsc.io/badsum v1.0.0 with good go.mod, bad tree
cp go.mod.orig go.mod
! go get rsc.io/badsum@v1.0.0
stderr 'verifying rsc.io/badsum@v1.0.0: checksum mismatch'
stderr 'notary: +h1:6/o\+QJfe6mFSNuegDihphabcvR94anXQk/qq7Enr19U='
stderr 'SECURITY ERROR'
stderr 'NOT match the expected download known to the notary'
stderr 'go help module-auth'
grep 'rsc.io/badsum v1.0.0/go.mod' go.sum
! grep 'rsc.io/badsum v1.0.0 ' go.sum
rm go.sum
# notary should reject rsc.io/badsum v1.0.1 with bad go.mod, good tree
cp go.mod.orig go.mod
! go get rsc.io/badsum@v1.0.1
stderr 'verifying rsc.io/badsum@v1.0.1/go.mod: checksum mismatch'
stderr 'notary: +h1:avOsLUJaHavllihBU9qCTW37z64ypkZjqZg8O16JLVY='
stderr 'SECURITY ERROR'
stderr 'NOT match the expected download known to the notary'
stderr 'go help module-auth'
! exists go.sum # failed at go.mod, did not get to the tree
# notary checks should run even without explicit go.mod
rm go.mod
rm go.sum
! go get rsc.io/badsum@v1.0.0
stderr 'verifying rsc.io/badsum@v1.0.0: checksum mismatch'
stderr 'notary: +h1:6/o\+QJfe6mFSNuegDihphabcvR94anXQk/qq7Enr19U='
stderr 'SECURITY ERROR'
stderr 'NOT match the expected download known to the notary'
stderr 'go help module-auth'
! go get rsc.io/badsum@v1.0.1
stderr 'verifying rsc.io/badsum@v1.0.1/go.mod: checksum mismatch'
stderr 'notary: +h1:avOsLUJaHavllihBU9qCTW37z64ypkZjqZg8O16JLVY='
stderr 'SECURITY ERROR'
stderr 'NOT match the expected download known to the notary'
stderr 'go help module-auth'
! exists go.mod
! exists go.sum
# go get -insecure should skip notary without go.mod
go get -insecure rsc.io/badsum@v1.0.0
go get -insecure rsc.io/badsum@v1.0.1
! exists go.mod
! exists go.sum
# GONOVERIFY should skip notary too
env GONOVERIFY=rsc.i[aeiou]
go get rsc.io/badsum@v1.0.0
go get rsc.io/badsum@v1.0.1
! exists go.mod
! exists go.sum
env GONOVERIFY=
# go get -insecure should skip notary with go.mod and update go.sum
cp go.mod.orig go.mod
go get -insecure rsc.io/badsum@v1.0.0
go get -insecure rsc.io/badsum@v1.0.1
# go.sum should override notary
cp go.mod.orig go.mod
cp go.sum.wrong go.sum
go get rsc.io/badsum@v1.0.0
go get rsc.io/badsum@v1.0.1
-- go.mod.orig --
module m
-- go.sum.wrong --
rsc.io/badsum v1.0.0 h1:kZaLRhqz4LgsHPQddRMr+124lTgJfm28AxghGw3vLB0=
rsc.io/badsum v1.0.1/go.mod h1:xwKWaN8OFR80Kg5/vdt+V2b+2P5kaLH+wWo5CL+pwHs=