Commit graph

11 commits

Author SHA1 Message Date
Klaus Post 9803f68522
snowball: Restrict zstd window size (#16987) 2023-04-06 17:47:38 -07:00
ferhat elmas 3423028713
cleanup Go linter settings (#16736) 2023-03-04 20:57:35 -08:00
Harshavardhana 98a84d88e2
fix: trim and ignore './' directories (#16642) 2023-02-17 07:15:03 -08:00
Klaus Post 027ff0f3a8
fix: set modTime to current in snowball if archive shows empty (#16482) 2023-01-26 22:20:35 +05:30
Klaus Post a713aee3d5
Run staticcheck on CI (#16170) 2022-12-05 11:18:50 -08:00
Klaus Post bd3dfad8b9
Add concurrent Snowball extraction + options (#15836) 2022-10-18 13:50:21 -07:00
Klaus Post 038fdeea83
snowball: return errors on failures (#13836)
Return errors when untar fails at once.

Current error handling was quite a mess. Errors are written 
to the stream, but processing continues.

Instead, return errors when they occur and transform 
internal errors to bad request errors, since it is likely a 
problem with the input.

Fixes #13832
2021-12-06 09:45:23 -08:00
Klaus Post c2eb60df4a
bz2: limit max concurrent CPU (#13458)
Ensure that bz2 decompression will never take more than 50% CPU.
2021-10-18 08:44:36 -07:00
Klaus Post 5e53f767c4
Use concurrent bz2 decompression (#13360)
Testing with `mc sql --compression BZIP2 --csv-input "rd=\n,fh=USE,fd=;" --query="select COUNT(*) from S3Object" local2/testbucket/nyc-taxi-data-10M.csv.bz2`

Before 96.98s, after 10.79s. Uses about 70% CPU while running.
2021-10-14 11:11:07 -07:00
Harshavardhana 069432566f update license change for MinIO
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-23 11:58:53 -07:00
Harshavardhana d8bda2dd92
[feat] Add targz transparent extract support (#11849)
This feature brings in support for auto extraction
of objects onto MinIO's namespace from an incoming
tar gzipped stream, the only expected metadata sent
by the client is to set `snowball-auto-extract`.

All the contents from the tar stream are saved as
folders and objects on the namespace.

fixes #8715
2021-03-26 17:15:09 -07:00