self-hosted/minio
self-hosted/minio
1
0
Fork 0
mirror of https://github.com/minio/minio synced 2024-10-15 04:12:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
2667 commits 5 branches 502 tags 200 MiB
Commit graph

35 commits

Author SHA1 Message Date
Krishna Srinivas 01a439f95b refactor: add multipart code to the object layer. 2016-04-16 16:25:53 -07:00
Krishna Srinivas 3c48537f20 refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
Harshavardhana 33cd910d3a backend/fs: More cleanup and start using checkBuckeArg. (#1306) 
backend/fs: More cleanup and start using checkBucketArg.
 2016-04-08 17:13:16 -07:00
Harshavardhana 0479d4976b objectAPI: Fix object API interface, remove unnecessary structs. 
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
 2016-04-03 15:25:01 -07:00
Harshavardhana 484ba91b08 config: Migrate to the new version. Remove backend details. 
Migrate to new config format v4.
```
{
	"version": "4",
	"credential": {
		"accessKey": "WLGDGYAQYIGI833EV05A",
		"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
	},
	"region": "us-east-1",
	"logger": {
		"console": {
			"enable": true,
			"level": "fatal"
		},
		"file": {
			"enable": false,
			"fileName": "",
			"level": "error"
		},
		"syslog": {
			"enable": false,
			"address": "",
			"level": "debug"
		}
	}
}
```

This patch also updates [minio cli spec](./minio.md)
 2016-04-02 17:29:31 -07:00
Harshavardhana efc80343e3 fs: Break fs package to top-level and introduce ObjectAPI interface. 
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.

The new interface for any ObjectAPI layer is as below

```
// ObjectAPI interface.
type ObjectAPI interface {
        // Bucket resource API.
        DeleteBucket(bucket string) *probe.Error
        ListBuckets() ([]BucketInfo, *probe.Error)
        MakeBucket(bucket string) *probe.Error
        GetBucketInfo(bucket string) (BucketInfo, *probe.Error)

        // Bucket query API.
        ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
        ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)

        // Object resource API.
        GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
        GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
        PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
        DeleteObject(bucket, object string) *probe.Error

        // Object query API.
        NewMultipartUpload(bucket, object string) (string, *probe.Error)
        PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
        ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
        CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
        AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
 2016-04-01 15:58:39 -07:00
Karthic Rao 30fc970eab Changes to TestPutBucket to catch the race 2016-04-01 15:21:16 +05:30
Harshavardhana 9dca46e156 signature: Use a layered approach for signature verification. 
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.

In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
 2016-03-26 15:21:05 -07:00
Harshavardhana aaf97ea02c config/main: Re-write config files - add to new config v3 
- New config format.

```
{
	"version": "3",
	"address": ":9000",
    "backend": {
          "type": "fs",
          "disk": "/path"
    },
	"credential": {
		"accessKey": "WLGDGYAQYIGI833EV05A",
		"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
	},
	"region": "us-east-1",
	"logger": {
		"file": {
			"enable": false,
			"fileName": "",
			"level": "error"
		},
		"syslog": {
			"enable": false,
			"address": "",
			"level": "debug"
		},
		"console": {
			"enable": true,
			"level": "fatal"
		}
	}
}
```

New command lines in lieu of supporting XL.

Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~

Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~

For 'fs' backend it starts the server.
~~~
$ minio server
~~~

For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~

Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
 2016-03-23 19:16:09 -07:00
Harshavardhana 902aa05021 main: Remove all the dead/unused code. 
This patch removes some dead and unused code.
 2016-03-21 01:12:29 -07:00
Harshavardhana fc72a0362f api: ListMultipartUploads and ListParts responded more entries. 
Issue is empty entries were added since allocating an array was
followed by an append. Keep the index and copy the right entries
precisely.

Fixes an issue reported at - https://github.com/minio/mc/issues/1642
 2016-03-19 23:52:40 -07:00
Harshavardhana 88714e7c8e bucketpolicy: Improve bucket policy validation, avoid nested rules. 
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.

This patch implements stricter checks.

Example policy to be generally avoided.
```
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"s3:GetObject",
				"s3:DeleteObject"
			],
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"*"
				]
			},
			"Resource": [
				"arn:aws:s3:::jarjarbing/*"
			]
		},
		{
			"Action": [
				"s3:GetObject",
				"s3:DeleteObject"
			],
			"Effect": "Deny",
			"Principal": {
				"AWS": [
					"*"
				]
			},
			"Resource": [
				"arn:aws:s3:::jarjarbing/restic/key/*"
			]
		}
	]
}
```
 2016-03-15 17:50:23 -07:00
Harshavardhana d5057b3c51 accessPolicy: Implement Put, Get, Delete access policy. 
This patch implements Get,Put,Delete bucket policies

Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html

Currently supports following actions.

   "*":                             true,
   "s3:*":                          true,
   "s3:GetObject":                  true,
   "s3:ListBucket":                 true,
   "s3:PutObject":                  true,
   "s3:CreateBucket":               true,
   "s3:GetBucketLocation":          true,
   "s3:DeleteBucket":               true,
   "s3:DeleteObject":               true,
   "s3:AbortMultipartUpload":       true,
   "s3:ListBucketMultipartUploads": true,
   "s3:ListMultipartUploadParts":   true,

following conditions for "StringEquals" and "StringNotEquals"

   "s3:prefix", "s3:max-keys"
 2016-03-08 17:44:50 -08:00
Harshavardhana aed62788d9 api: Implement multiple objects Delete api - fixes #956 
This API takes input XML input in following form.

```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Quiet>true</Quiet>
    <Object>
         <Key>Key</Key>
    </Object>
    <Object>
         <Key>Key</Key>
    </Object>
    ...
</Delete>
```

and responds the list of successful deletes, list of errors
for all the deleted objects.

```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Deleted>
    <Key>sample1.txt</Key>
  </Deleted>
  <Error>
    <Key>sample2.txt</Key>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
  </Error>
</DeleteResult>
```
 2016-03-06 18:31:50 -08:00
Harshavardhana ee1b86e517 api: Implement support for additional request headers. 
Now GetObject and HeadObject both support

  - If-Modified-Since, If-Unmodified-Since
  - If-Match, If-None-Match

request headers.

These headers are used to further handle the responses for GetObject
and HeadObject API.

Fixes #1098
 2016-02-28 19:34:20 -08:00
Harshavardhana 3ff8a1b719 api: Implement CopyObject s3 API, doing server side copy. 
Fixes #1172
 2016-02-27 19:51:59 -08:00
Harshavardhana ae6e774377 api: Return NotImplemented for MultiDelete and CopyObject APIs 2016-02-26 15:57:30 -08:00
Harshavardhana 91a092792a presigned: Fix a bug in presigned request verification. 
Additionally add Docker proxy configuration.
 2016-02-18 02:23:12 -08:00
Harshavardhana dd9aaa855c web/rpc: Merge ports with API server. 
Fixes #1081 and #1130
 2016-02-17 20:28:15 -08:00
Harshavardhana 5a9333a67b signature: Rewrite signature handling and move it into a library. 2016-02-16 17:28:16 -08:00
Harshavardhana 8df201ef30 multipart: Multipart session map now is based on uploadID. 
- Fixes initiating parallel uploads, and configs being quickly
  re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
 2016-02-05 23:32:30 -08:00
Harshavardhana 7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err' 
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
 2016-02-04 13:43:52 -08:00
Harshavardhana df91661ec6 flags: Remove anonymous, ratelimit, json and web-address flags. 
- Web address now uses the port + 1 from the API address port directly.
- Remove ratelimiting, ratelimiting will be achieved if necessary through
  iptables.
- Remove json flag, not needed anymore.
- Remove anonymous flag, server will be no more anonymous for play.minio.io
  we will use demo credentials.
 2016-02-02 18:37:09 -08:00
Harshavardhana 5d87fdb35c Merge pull request #1051 from harshavardhana/fix-multipart 
multipart: NewMultipartUpload shouldn't return empty UploadID
 2016-01-26 15:15:44 -08:00
Harshavardhana 2e311168ee multipart: NewMultipartUpload shouldn't return empty UploadID 
Existing code
```
{
  if os.IsNotExist(e) {
       e = os.MkdirAll(objectDir, 0700)
       if e != nil {
            return "", probe.NewError(e)
       }
  }
  return "", probe.NewError(e)  ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.

Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
 2016-01-26 15:00:34 -08:00
Harshavardhana 682020ef2f listObjects: Channel based changes. 
Supports:
 - prefixes
 - marker
 2016-01-25 20:39:38 -08:00
Harshavardhana 836f5204af minio: Add config-folder option. 
Fixes #997
 2015-12-07 12:34:09 -08:00
Harshavardhana 7c91a8495f signature/region: Remove 'milkyway' and use 'us-east-1' as default. 
Fixes #980
 2015-11-23 13:40:23 -08:00
Kanai Masumi 84de2e33c4 Fix: permit trailing slash for compatible with S3. 
ex.
s3cmd requests to path:`/<bucket>/` for PutBucket.
 2015-11-07 20:22:13 +09:00
Harshavardhana ab15f56a61 fs/bucket: Move bucket metadata into buckets.json 2015-11-01 21:25:01 -08:00
Harshavardhana 56003fded7 Add logger command - also migrate from old config to newer config 2015-10-21 00:02:16 -07:00
Harshavardhana 5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana 1256ca86d0 Add fs separator 2015-10-17 12:05:12 -07:00
Harshavardhana 704fa420a3 Reply back proper statuses for DeleteBucket/DeleteObject 2015-10-16 20:03:44 -07:00
Harshavardhana 762b798767 Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
Renamed from server_signV4_donut_cache_test.go (Browse further)