Commit graph

6739 commits

Author SHA1 Message Date
Praveen raj Mani 0cc2ed04f5
humanize timeToFirstByte and timeToResponse upto nanoseconds (#9641) 2020-05-19 18:34:02 -07:00
Andreas Auernhammer b11adfa5cd
simplify the KMS guide and remove unnecessary sections (#9629)
This commit simplifies the KMS configuration guide by
adding a get started section that uses our KES play instance
at `https://play.min.io:7373`.

Further, it removes sections that we don't recommend for production
anyways (MASTER_KEY).
2020-05-19 18:33:11 -07:00
Anis Elleuch 9baeda781a
fix storage info output with unordered endpoints arguments (#9610)
Shuffling arguments that we pass to MinIO server are supported. However,
when that happens, Prometheus returns wrong information about disks usage
and online/offline status.

The commit fixes the issue by avoiding relying on xl.endpoints since
it is not ordered.
2020-05-19 14:27:20 -07:00
Harshavardhana bd032d13ff
migrate all bucket metadata into a single file (#9586)
this is a major overhaul by migrating off all
bucket metadata related configs into a single
object '.metadata.bin' this allows us for faster
bootups across 1000's of buckets and as well
as keeps the code simple enough for future
work and additions.

Additionally also fixes #9396, #9394
2020-05-19 13:53:54 -07:00
Harshavardhana 730775bb4e
update all the outdated browser deps (#9640) 2020-05-19 09:32:14 -07:00
Harshavardhana d31eaddba3
fix: avoid double body reads in SelectObject call (#9638)
Bonus fix handle encryption headers in response
properly for both notification and response to
the client.
2020-05-19 02:01:08 -07:00
poornas 3202f78f0f
Fix cache metadata update for range GET (#9636)
This was inadvertently deleting cached ranges
because HTTPRangeSpec was not being passed down

fixes #9597
2020-05-18 18:33:43 -07:00
Harshavardhana 6de410a0aa
fix: possiblity of double write lockers on same resource (#9616)
To avoid this issue with refCounter refactor the code
such that

- locker() always increases refCount upon success
- unlocker() always decrements refCount upon success
  (as a special case removes the resource if the
  refCount is zero)

By these two assumptions we are able to see that we
are never granted two write lockers in any situation.

Thanks to @vcabbage for writing a nice reproducer.
2020-05-18 17:33:35 -07:00
darkdragon-001 a3f41c7049
Improve browser README (#9628)
- Add instructions to download node dependencies
- Add instructions to allow development from IPs other than localhost
2020-05-18 11:35:57 -07:00
Klaus Post 1847f17f50
Set Deployment ID before starting handlers (#9635)
Global handler ID is added to response headers, so initialize it before the server starts.

Fixes #9634
2020-05-18 11:35:05 -07:00
Harshavardhana 1bc32215b9
enable full linter across the codebase (#9620)
enable linter using golangci-lint across
codebase to run a bunch of linters together,
we shall enable new linters as we fix more
things the codebase.

This PR fixes the first stage of this
cleanup.
2020-05-18 09:59:45 -07:00
Anis Elleuch 96009975d6
relax validation when loading lifecycle document from the backend (#9612) 2020-05-18 08:33:43 -07:00
Harshavardhana de9b391db3
fix: Disable presigned without appropriate policy (#9621)
Fixes #9590
2020-05-17 23:38:52 -07:00
kannappanr a62572fb86
Check for address flags in all positions (#9615)
Fixes #9599
2020-05-17 08:46:23 -07:00
poornas 011a2c0b78
Add docs for bucket quota feature (#9503)
This PR also adds a check to not enforce
bucket quota for server-side metadata copy
of an object onto itself.
2020-05-16 19:27:33 -07:00
Harshavardhana daf4418cbb
add 50m for windows tests (#9614) 2020-05-15 21:07:27 -07:00
Minio Trusted 0f8df3d340 Update yaml files to latest version RELEASE.2020-05-16T01-33-21Z 2020-05-16 01:40:31 +00:00
Harshavardhana 9cac385aec add comment on exported error 2020-05-15 18:17:54 -07:00
Harshavardhana 814ddc0923
add missing admin actions, enhance AccountUsageInfo (#9607) 2020-05-15 18:16:45 -07:00
Harshavardhana 247795dd36
add github workflow for windows (#9611)
bye, bye travis
2020-05-15 15:54:39 -07:00
Anis Elleuch dfadf70a7f
mint: Add more SQL tests (#9540) 2020-05-15 11:20:57 -07:00
Harshavardhana d348ec0f6c
avoid double listObjectParts calls improves performance (#9606)
this PR is to avoid double calls across multiple calls
in APIs

- CopyObjectPart
- PutObjectPart
2020-05-15 08:06:45 -07:00
Harshavardhana b730bd1396
fix: possible race in FS local lockMap (#9598) 2020-05-14 23:59:07 -07:00
Klaus Post 56e0c6adf8
Track if bloom filter is dirty (#9601)
Only save bloom filter on cycles and updates.

Fixes #9600
2020-05-14 21:46:36 -07:00
Anis Elleuch f44a960dcd
tests: Fix one multi-delete test failure in Windows CI (#9602)
There is a disparency of behavior under Linux & Windows about
the returned error when trying to rename a non existant path.

err := os.Rename("/path/does/not/exist", "/tmp/copy")

Linux:
  isSysErrNotDir(err) = false
  os.IsNotExist(err) = true

Windows:
  isSysErrNotDir(err) = true
  os.IsNotExist(err) = true

ENOTDIR in Linux is returned when the destination path
of the rename call contains a file in one of the middle
segments of the path (e.g. /tmp/file/dst, where /tmp/file
is an actual file not a directory)

However, as shown above, Windows has more scenarios when
it returns ENOTDIR. For example, when the source path contains
an inexistant directory in its path.

In that case, we want errFileNotFound returned and not
errFileAccessDenied, so this commit will add a further check to close
the disparency between Windows & Linux.
2020-05-14 18:09:30 -07:00
kannappanr 6c1bbf918d
do not add quotes around etag, if already present (#9603) 2020-05-14 17:43:54 -07:00
Anis Elleuch 48e614b167
honor lifecycle expiration with tag rule (#9604) 2020-05-14 16:21:03 -07:00
poornas fe8d33452b
Allow writes for bucket exceeding FIFO quota (#9575)
the quota will be enforced while
deleting oldest entries in FIFO manner.
2020-05-14 15:18:24 -07:00
Andreas Auernhammer c19ece6921
update KMS guide to reflect latest KES changes (#9591)
This commit updates the two client env. variables:
```
KES_CLIENT_TLS_KEY_FILE
KES_CLIENT_TLS_CERT_FILE
```

The KES CLI client expects the client key and certificate
as `KES_CLIENT_KEY` resp. `KES_CLIENT_CERT`.
2020-05-14 14:02:43 -07:00
Klaus Post 216fa57b88
merge nested hash readers (#9582)
The `ioutil.NopCloser(reader)` was hiding nested hash readers.

We make it an `io.Closer` so it can be attached without wrapping 
and allows for nesting, by merging the requests.
2020-05-14 14:01:31 -07:00
Dzmitry Pasiukevich a9558ae248
Simplify cast of string to rune slice in wildcard matching (#9577) 2020-05-14 08:20:13 -07:00
Klaus Post ee9077db7d
fix: windows tests for all cases (#9594)
Replaces #9299
2020-05-13 23:55:38 -07:00
Harshavardhana 9c85928740
add formatting message for zones in ordinals (#9596)
Unlike the message
> Formatting 2 zone, 1 set(s), 6 drives per set.

It is more readable as ordinal
> Formatting 2nd zone, 1 set(s), 6 drives per set.
2020-05-13 20:25:29 -07:00
kannappanr af0309371e
mint tests s3cmd remove copy object test (#9595)
test is failing in Azure gateway as a mapping from 
x-amz-storage-class to Azure storage tier is failing 
and there is no way to not send storage class in s3cmd.
2020-05-13 19:55:34 -07:00
ebozduman ead3c186a6
Fixes broken image preview for anon user (#9584) 2020-05-13 13:26:12 -07:00
Harshavardhana 6ac48a65cb
fix: use unused cacheMetrics code in prometheus (#9588)
remove all other unusued/deadcode
2020-05-13 08:15:26 -07:00
poornas 2ecf5ba1de
fix panic when checking if es/nats event target is active (#9587) 2020-05-13 06:37:22 -07:00
Krishna Srinivas 94f1a1dea3
add option for O_SYNC writes for standalone FS backend (#9581) 2020-05-12 19:24:59 -07:00
Anis Elleuch c045ae15e7
fix: avoid undoing bucket creation and return the first err instead (#9578) 2020-05-12 15:20:42 -07:00
Harshavardhana 1756b7c6ff
fix: LDAP derivative accounts parentUser validation is not needed (#9573)
* fix: LDAP derivative accounts parentUser validation is not needed

fixes #9435

* Update cmd/iam.go

Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>

Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
2020-05-12 09:21:08 -07:00
Klaus Post e25ace2151
Forward RPC errors from crawler (#9569)
The `keepHTTPResponseAlive` would cause errors to be 
returned with status OK.

- Add '32' as a filler byte until a response is ready
- '0' to indicate the response is ready to be consumed
- '1' to indicate response has an error which needs
to be returned to the caller

Clear out 'file not found' errors from dir walker, since it may be 
in a folder that has been deleted since it was scanned.
2020-05-11 20:41:38 -07:00
poornas a8e5a86fa0
Remove brittle tests for cache (#9570) 2020-05-11 15:41:10 -07:00
Harshavardhana f8edc233ab
support multiple policies for temporary users (#9550) 2020-05-11 13:04:11 -07:00
Harshavardhana 337c2a7cb4
add audit logging for all admin calls (#9568)
- add ServiceRestart/ServiceStop actions
- audit log appropriately in all admin handlers

fixes #9522
2020-05-11 10:34:08 -07:00
Harshavardhana 2d735144b9 fix: distributed docs image path 2020-05-11 09:33:55 -07:00
Harshavardhana 0113035237
update distributed setup guide (#9566) 2020-05-11 09:19:10 -07:00
Anis Elleuch 52a1d248b2
policy: Do not return an error for invalid value during parsing (#9442)
s3:HardwareInfo was removed recently. Users having that admin action
stored in the backend will have an issue starting the server.

To fix this, we need to avoid returning an error in Marshal/Unmarshal
when they encounter an invalid action and validate only in specific
location.

Currently the validation is done and in ParseConfig().
2020-05-10 10:55:28 -07:00
Harshavardhana b5ed42c845
ignore policy/group missing errors appropriately (#9559) 2020-05-09 13:59:12 -07:00
Klaus Post d9e7cadacf
Update reed+solomon (#9562)
Only create encoder when strictly needed.
2020-05-09 09:54:20 -07:00
Harshavardhana 36e88cbd50
fix mint tests in awscli to ignore NotImplemented properly (#9561) 2020-05-08 22:32:10 -07:00