Commit graph

5925 commits

Author SHA1 Message Date
Nitish Tiwari
1cd801b2e9 Fix DeleteObjects() to remove renamed objects inside (#8072) 2019-08-14 11:15:25 -07:00
Aditya Manthramurthy
bf9b619d86 Set the policy mapping for a user or group (#8036)
Add API to set policy mapping for a user or group

Contains a breaking Admin APIs change.

- Also enforce all applicable policies
- Removes the previous /set-user-policy API

 Bump up peerRESTVersion

Add get user info API to show groups of a user
2019-08-13 13:41:06 -07:00
kannappanr
bc79b435a2 Doc: Fix PutObject API known limitations text (#7704) 2019-08-13 08:50:43 -07:00
maihde
0ed6daab59 fix: #8051 so that stale DNS entries are cleaned-up (#8053) 2019-08-13 08:49:26 -07:00
Praveen raj Mani
e96f19c867 Use default ports for the schemes when dialing (#7960)
Fixes #7946
2019-08-13 16:43:11 +05:30
Harshavardhana
aaef18b1a3 Fail for disks which overflow upon usage calculation (#8056)
Fixes #8035
2019-08-13 02:58:43 -07:00
Kaan Kabalak
a48a034e5a Make directory path in the header editable (#8018)
This change will allow users to navigate to their desired locations,
including buckets and directories that haven't been "created" yet

Fixes #7883

Add tests

Change tooltip wording

Migrate to Font Awesome 5 to use path icon

Fix sidebar not closing on mobile
2019-08-12 22:36:19 -07:00
Harshavardhana
bf8ec8ad73
Cleanup ui-errors and print proper error messages (#8068)
* Cleanup ui-errors and print proper error messages

Change HELP to HINT instead, handle more error
cases when starting up MinIO. One such is related
to #8048

* Apply suggestions from code review
2019-08-12 21:25:34 -07:00
Harshavardhana
8ce424bacd Enhance audit logging to capture responseTimes (#8067)
Audit logging requires to have

- timeToFirstByte
- timeToResponse

timing information
2019-08-12 20:32:34 -07:00
Anis Elleuch
cea3e3f7a6 browser: Add user-agent header filter to gorilla mux route (#8040)
When a peer client which higher version sends a request to a peer
server with lower version, the returned status code is 200 OK instead
of 405 code. The reason is that the peer client request reaches the
browser handler, which registers itself by '/minio' route but without
any other constraints. Adding filtering by user agent header to the
browser route so internal requests to old endpoints versions return
405 error code.
2019-08-12 17:05:30 -07:00
Harshavardhana
af36c92cab
With ListBuckets() access-list only buckets the user has access (#8037)
This is a behavior change from AWS S3, but it is done with
better judgment on our end to allow the listing of buckets only
which user has access to.

The advantage is this declutters the UI for users and only
lists bucket which they have access to.

Precursor for this feature to be applicable is a policy
must have the following actions

```
s3:ListAllMyBuckets
```
and
```
s3:ListBucket
```

enabled in the policy.
2019-08-12 10:27:38 -07:00
Jakob Ackermann
1b258da108 [web-router] update the white list for favicons (#8024) 2019-08-11 22:17:02 -07:00
Andreas Auernhammer
35427a017d fix type conversion in UpdateKey for Vault (#8058)
This commit fixes a type conversion in the `UpdateKey`
implementation of Vault.
2019-08-11 22:20:25 +05:30
Cary FitzHugh
a0715f3b59 Adding Notes on Swarm Network Aliases (#8054) 2019-08-10 21:25:01 +05:30
Harshavardhana
5a28ef0d47 Bump readiness check upto 10000 go-routines (#8057)
Most of our current workloads reach this value
regularly, it doesn't make sense to keep 1000
go-routine limit.
2019-08-10 18:13:14 +05:30
poornas
3385bf3da8 Rewrite cache implementation to cache only on GET (#7694)
Fixes #7458
Fixes #7573 
Fixes #7938 
Fixes #6934
Fixes #6265 
Fixes #6630 

This will allow the cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

- All cached content is automatically bitrot protected.

- Avoid ETag verification if a cache-control header
is set and the cached content is still valid.

- This PR changes the cache backend format, and all existing
content will be migrated to the new format. Until the data is
migrated completely, all content will be served from the backend.
2019-08-09 17:09:08 -07:00
Anis Elleuch
1ce8d2c476 Add bucket lifecycle expiry feature (#7834) 2019-08-09 10:02:41 -07:00
Harshavardhana
a8296445ad
Safely use unsafe.Pointer to avoid crashes on ARM (#8027)
Refactor the Dirent parsing code such that when we
calculate offsets are correct based on the platform
This PR fixes a silent potential crash on ARM
architecture.
2019-08-09 08:54:11 -07:00
Nitish Tiwari
43c72374d4 Fix healthcheck to look for entry of all IPv4 addresses (0.0.0.0) (#8046)
Fixes #7993
2019-08-09 01:28:37 -07:00
Harshavardhana
d759a7ce99
Fix time formatting of Expiration field in STS (#8043)
Without explicit conversion to UTC() from Unix
time the zone information is lost, this leads
to XML marshallers marshaling the time into
a wrong format.

This PR fixes the compatibility issue with AWS STS
API by keeping Expiration format close to ISO8601
or RFC3339

Fixes #8041
2019-08-08 15:44:57 -07:00
Aditya Manthramurthy
5d2b5ee6a9 Refactor IAM to use new IAMStorageAPI (#7999) 2019-08-08 15:10:04 -07:00
Harshavardhana
9ee619ea14 Remove deprecated ENVs 2019-08-07 18:57:38 -07:00
Minio Trusted
55b385beee Update yaml files to latest version RELEASE.2019-08-07T01-59-21Z 2019-08-07 02:05:41 +00:00
kannappanr
930943f058
Fix IAM users migration regression in etcd (#8029)
PR #8008 did not migrate user data stored in etcd.
This PR fixes that.
2019-08-06 17:06:31 -07:00
Harshavardhana
e6d8e272ce
Use const slashSeparator instead of "/" everywhere (#8028) 2019-08-06 12:08:58 -07:00
Harshavardhana
b52b90412b Avoid data-transfer in distributed locking (#8004) 2019-08-05 11:45:30 -07:00
Harshavardhana
843f481eb3 Allow "tmp" directory to be not available (#8021)
Also additionally add more context to the errors
generated by filesystem, to facilitate better
debugging.
2019-08-05 11:41:29 -07:00
Andreas Auernhammer
f6d0645a3c fix DoS vulnerability in the content SHA-256 processing (#8026)
This commit fixes a DoS issue that is caused by an incorrect
SHA-256 content verification during STS requests.

Before that fix clients could write arbitrary many bytes
to the server memory. This commit fixes this by limiting the
request body size.
2019-08-05 10:06:40 -07:00
Aditya Manthramurthy
414a7eca83 Add IAM groups support (#7981)
This change adds admin APIs and IAM subsystem APIs to:

- add or remove members to a group (group addition and deletion is
  implicit on add and remove)

- enable/disable a group

- list and fetch group info
2019-08-02 14:25:00 -07:00
maihde
5cd9f10a02 Support Federation on a single machine (#8009)
When checking if federation is necessary, the code compares
the SRV record stored in etcd against the list of endpoints
that the MinIO server is exposing.  If there is an intersection
in this list the request is forwarded.

The SRV record includes both the host and the port, but the
intersection check previously only looked at the IP address.  This
would prevent federation from working in situations where the endpoint
IP is the same for multiple MinIO servers.  Some examples of where this
can occur are:
 - running mulitiple copies of MinIO on the same host
 - using multiple MinIO servers behind a NAT with port-forwarding
2019-08-02 12:40:51 -07:00
Praveen raj Mani
b976521c83 Ignore faulty disks in xl-sets Storage info (#7878) 2019-08-02 12:17:26 -07:00
SCDealy
2c3b1f01d9 Update README.md (#8006)
Since MinIO by default is not fully S3 compatible, this fact should be
specified in a prominent place in the quick start guide so people 
new to MinIO don't have to spend hours figuring it out the hard way.
2019-08-02 10:31:07 +05:30
Andreas Auernhammer
a6f4cf61f2 add UpdateKey method to KMS interface (#7974)
This commit adds a new method `UpdateKey` to the KMS
interface.

The purpose of `UpdateKey` is to re-wrap an encrypted
data key (the key generated & encrypted with a master key by e.g.
Vault).
For example, consider Vault with a master key ID: `master-key-1`
and an encrypted data key `E(dk)` for a particular object. The
data key `dk` has been generated randomly when the object was created.
Now, the KMS operator may "rotate" the master key `master-key-1`.
However, the KMS cannot forget the "old" value of that master key
since there is still an object that requires `dk`, and therefore,
the `D(E(dk))`.
With the `UpdateKey` method call MinIO can ask the KMS to decrypt
`E(dk)` with the old key (internally) and re-encrypted `dk` with
the new master key value: `E'(dk)`.

However, this operation only works for the same master key ID.
When rotating the data key (replacing it with a new one) then
we perform a `UnsealKey` operation with the 1st master key ID
and then a `GenerateKey` operation with the 2nd master key ID.

This commit also updates the KMS documentation and removes
the `encrypt` policy entry (we don't use `encrypt`) and
add a policy entry for `rewarp`.
2019-08-01 15:47:47 -07:00
Minio Trusted
dfa8835720 Update yaml files to latest version RELEASE.2019-08-01T22-18-54Z 2019-08-01 22:27:41 +00:00
Anis Elleuch
c5ac901e8d xl: Fix healing empty directories (#8013)
After some extensive refactors, it turned out empty directories
are not healed and heal status is also not reported correctly.

This commit fixes it and adds the appropriate unit tests
2019-08-01 14:13:06 -07:00
Aditya Manthramurthy
4101d4917c Fix IAM users migration regression (#8008) 2019-08-01 12:31:04 -07:00
maihde
d966d29fed fix: add integer code for Windows Subsystem for Linux filesystem (#8010) 2019-08-01 06:00:57 -07:00
Minio Trusted
c301f5882d Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z 2019-07-31 19:06:20 +00:00
Harshavardhana
123cccaed1 Honor connection pooling while tracing (#7979)
This PR fixes relying on r.Context().Done()
by setting

```
Connection: "close"
```

HTTP Header, this has detrimental issues for
client side connection pooling. Since this
header explicitly tells clients to turn-off
connection pooling. This causing pro-active
connections to be closed leaving many conn's
in TIME_WAIT state. This can be observed with
`mc admin trace -a` when running distributed
setup.

This PR also fixes tracing filtering issue
when bucket names have `minio` as prefixes,
trace was erroneously ignoring them.
2019-07-31 11:08:39 -07:00
Anis Elleuch
cbd02c58be federation: Avoid printing context canceled error (#7997)
Golang proactively prints this error
        `http: proxy error: context canceled`

when a request arrived to the current deployment and
redirected to another deployment in a federated setup.

Since this error can confuse users, this commit will
just hide it.
2019-07-31 11:08:10 -07:00
Aditya Manthramurthy
c71895f225 Listen for PolicyDB events from etcd and fix etcd watch handling (#7992) 2019-07-30 18:50:49 -07:00
Harshavardhana
b83413b167 Use GOPROXY to speed up builds (#7984)
Read more here https://proxy.golang.org proposal 
for go1.13
2019-07-30 22:27:11 +05:30
Praveen raj Mani
63e0a81760 Ignore stale notification queues in notification.xml (#7673)
Allow renaming/editing a notification config. By replying with 
a successful GetBucketNotification response, without checking 
for any missing config ARN in targetList.

Fixes #7650
2019-07-30 14:19:06 +05:30
Harshavardhana
8d47ef503c Fix crash observed in OPA initialization (#7990)
Related to #7982, this PR refactors the code
such that we validate the OPA or JWKS in a
common place.

This is also a refactor which is already done
in the new config migration change. Attempt
to avoid any network I/O during Unmarshal of
JSON from disk, instead do it later when
updating the in-memory data structure.
2019-07-29 15:58:25 -07:00
Harshavardhana
54eded2e6f Do not assume all HTTP errors as Network errors (#7983)
In situations such as when client uploading data,
prematurely disconnects from server such as pressing
ctrl-c before uploading all the data. Under this
situation in distributed setup we prematurely
disconnect disks causing a reconnect loop. This has
an adverse affect we end up leaving a lot of files
in temporary location which ideally should have been
cleaned up when Put() prematurely fails.

This is also a regression which got introduced in #7610
2019-07-29 14:48:18 -07:00
Harshavardhana
94c88890b8 Add additional logging for OPA connections (#7982) 2019-07-28 08:33:25 +05:30
Harshavardhana
e871e27562 Refactor and simplify etcd helpers used in IAM subsystem (#7980) 2019-07-26 13:42:54 -07:00
Harshavardhana
007a52b546
Add common validation for compression and encryption (#7978) 2019-07-26 02:41:16 -07:00
Praveen raj Mani
efb8b00db0 Preserve tailing backslash in URL paths (#7678)
Fixes #7649
2019-07-25 20:55:09 -07:00
Harshavardhana
d744865dc6 Enable config for NAS gateway mode (#7948)
Starting with #7751 we don't store config
in etcd anymore, allow NAS to honor config
on disk.
2019-07-25 17:41:25 -07:00