# s3-utils
Unified S3 utilities for the Constructive ecosystem — client factory, file operations, presigned URLs, and bucket management. Built on AWS SDK v3 with support for multiple S3-compatible providers.
## Features
- **Multi-provider support** — AWS S3, MinIO, Cloudflare R2, Google Cloud Storage, DigitalOcean Spaces
- **Presigned URLs** — generate secure PUT and GET URLs with configurable expiry
- **File operations** — streaming upload, download, existence checks, and metadata retrieval
- **Bucket management** — create buckets with provider-appropriate policies and CORS
- **AWS SDK v3** — modern, modular SDK with tree-shaking support
## Installation
```sh
npm install @constructive-io/s3-utils
```
## Quick Start
```typescript
import {
createS3Client,
presignPutUrl,
presignGetUrl,
upload,
download,
} from '@constructive-io/s3-utils';
// 1. Create a client
const client = createS3Client({
provider: 'minio',
region: 'us-east-1',
endpoint: 'http://minio:9000',
accessKeyId: 'minioadmin',
secretAccessKey: 'minioadmin',
});
// 2. Upload a file via stream
import { createReadStream } from 'fs';
const result = await upload({
client,
bucket: 'my-bucket',
key: 'docs/report.pdf',
contentType: 'application/pdf',
readStream: createReadStream('/path/to/report.pdf'),
});
// 3. Generate a presigned download URL
const url = await presignGetUrl(client, {
bucket: 'my-bucket',
key: 'docs/report.pdf',
});
```
## API
### `createS3Client(config)`
Creates an `S3Client` with provider-specific defaults.
```typescript
import { createS3Client } from '@constructive-io/s3-utils';
const client = createS3Client({
provider: 'r2',
region: 'auto',
endpoint: 'https://account.r2.cloudflarestorage.com',
accessKeyId: 'KEY',
secretAccessKey: 'SECRET',
});
```
**`StorageConnectionConfig`**
| Field | Type | Required | Description |
|---|---|---|---|
| `provider` | `'s3' \| 'minio' \| 'r2' \| 'gcs' \| 'spaces'` | yes | Storage provider |
| `region` | `string` | yes | S3 region (e.g. `"us-east-1"`) |
| `endpoint` | `string` | non-AWS | Endpoint URL (required for all providers except `s3`) |
| `accessKeyId` | `string` | yes | Access key ID |
| `secretAccessKey` | `string` | yes | Secret access key |
| `forcePathStyle` | `boolean` | no | Override path-style URL behavior |
Path-style URLs are enabled automatically for `minio`, `r2`, and `gcs`. Virtual-hosted style is used for `s3` and `spaces`.
Throws `S3ConfigError` on invalid configuration.
### `presignPutUrl(client, options)`
Generates a presigned PUT URL for uploading. The URL is locked to the specified key, content type, and content length.
```typescript
const url = await presignPutUrl(client, {
bucket: 'my-bucket',
key: 'uploads/photo.jpg',
contentType: 'image/jpeg',
contentLength: 102400,
expiresIn: 900, // default: 900 (15 min)
});
```
### `presignGetUrl(client, options)`
Generates a presigned GET URL for downloading. Supports an optional `filename` for `Content-Disposition`.
```typescript
const url = await presignGetUrl(client, {
bucket: 'my-bucket',
key: 'uploads/photo.jpg',
expiresIn: 3600, // default: 3600 (1 hour)
filename: 'photo.jpg', // optional, triggers download prompt
});
```
### `headObject(client, bucket, key)`
Checks if an object exists and returns its metadata. Returns `null` if the object is not found.
```typescript
const meta = await headObject(client, 'my-bucket', 'uploads/photo.jpg');
// { contentType: 'image/jpeg', contentLength: 102400 } or null
```
### `fileExists({ client, bucket, key })`
Returns `true` if the object exists, `false` otherwise.
```typescript
const exists = await fileExists({ client, bucket: 'my-bucket', key: 'doc.pdf' });
```
### `upload({ client, bucket, key, readStream, contentType })`
Streams a file to S3 using multipart upload.
```typescript
import { createReadStream } from 'fs';
const result = await upload({
client,
bucket: 'my-bucket',
key: 'uploads/video.mp4',
contentType: 'video/mp4',
readStream: createReadStream('/path/to/video.mp4'),
});
// { Location, ETag, Bucket, Key }
```
### `download({ client, bucket, key, writeStream })`
Downloads an object from S3 and pipes it to a writable stream.
```typescript
import { createWriteStream } from 'fs';
await download({
client,
bucket: 'my-bucket',
key: 'uploads/video.mp4',
writeStream: createWriteStream('/tmp/video.mp4'),
});
```
### `uploadThrough({ client, bucket, key, contentType })`
Returns a `PassThrough` stream that uploads to S3 as data is written. Emits an `'upload'` event with the result when complete.
```typescript
const pass = uploadThrough({
client,
bucket: 'my-bucket',
key: 'stream-data.csv',
contentType: 'text/csv',
});
pass.on('upload', (result) => {
console.log('Upload complete:', result.Location);
});
someReadable.pipe(pass);
```
### `createS3Bucket(client, bucket, options)`
Creates a bucket with provider-appropriate policies and CORS configuration.
- **MinIO**: read-only public access policy (list + get)
- **S3/GCS**: full-access policy with CORS rules for browser uploads
```typescript
const { success } = await createS3Bucket(client, 'my-bucket', {
provider: 'minio',
});
```
Returns `{ success: true }` if the bucket was created or already exists.
## Related Packages
| Package | Description |
|---|---|
| [`@constructive-io/s3-streamer`](../s3-streamer) | Streaming uploads with automatic content-type detection and metadata extraction |
| [`@constructive-io/etag-stream`](../etag-stream) | ETag generation from streams |
| [`@constructive-io/content-type-stream`](../content-type-stream) | Magic-byte content-type detection |
| [`@constructive-io/uuid-hash`](../uuid-hash) | Deterministic UUID generation from file content |
## License
MIT
---
## Education and Tutorials
1. 🚀 [Quickstart: Getting Up and Running](https://constructive.io/learn/quickstart)
Get started with modular databases in minutes. Install prerequisites and deploy your first module.
2. 📦 [Modular PostgreSQL Development with Database Packages](https://constructive.io/learn/modular-postgres)
Learn to organize PostgreSQL projects with pgpm workspaces and reusable database modules.
3. ✏️ [Authoring Database Changes](https://constructive.io/learn/authoring-database-changes)
Master the workflow for adding, organizing, and managing database changes with pgpm.
4. 🧪 [End-to-End PostgreSQL Testing with TypeScript](https://constructive.io/learn/e2e-postgres-testing)
Master end-to-end PostgreSQL testing with ephemeral databases, RLS testing, and CI/CD automation.
5. ⚡ [Supabase Testing](https://constructive.io/learn/supabase)
Use TypeScript-first tools to test Supabase projects with realistic RLS, policies, and auth contexts.
6. 💧 [Drizzle ORM Testing](https://constructive.io/learn/drizzle-testing)
Run full-stack tests with Drizzle ORM, including database setup, teardown, and RLS enforcement.
7. 🔧 [Troubleshooting](https://constructive.io/learn/troubleshooting)
Common issues and solutions for pgpm, PostgreSQL, and testing.
## Related Constructive Tooling
### 📦 Package Management
* [pgpm](https://github.com/constructive-io/constructive/tree/main/pgpm/pgpm): **🖥️ PostgreSQL Package Manager** for modular Postgres development. Works with database workspaces, scaffolding, migrations, seeding, and installing database packages.
### 🧪 Testing
* [pgsql-test](https://github.com/constructive-io/constructive/tree/main/postgres/pgsql-test): **📊 Isolated testing environments** with per-test transaction rollbacks—ideal for integration tests, complex migrations, and RLS simulation.
* [pgsql-seed](https://github.com/constructive-io/constructive/tree/main/postgres/pgsql-seed): **🌱 PostgreSQL seeding utilities** for CSV, JSON, SQL data loading, and pgpm deployment.
* [supabase-test](https://github.com/constructive-io/constructive/tree/main/postgres/supabase-test): **🧪 Supabase-native test harness** preconfigured for the local Supabase stack—per-test rollbacks, JWT/role context helpers, and CI/GitHub Actions ready.
* [graphile-test](https://github.com/constructive-io/constructive/tree/main/graphile/graphile-test): **🔐 Authentication mocking** for Graphile-focused test helpers and emulating row-level security contexts.
* [pg-query-context](https://github.com/constructive-io/constructive/tree/main/postgres/pg-query-context): **🔒 Session context injection** to add session-local context (e.g., `SET LOCAL`) into queries—ideal for setting `role`, `jwt.claims`, and other session settings.
### 🧠 Parsing & AST
* [pgsql-parser](https://www.npmjs.com/package/pgsql-parser): **🔄 SQL conversion engine** that interprets and converts PostgreSQL syntax.
* [libpg-query-node](https://www.npmjs.com/package/libpg-query): **🌉 Node.js bindings** for `libpg_query`, converting SQL into parse trees.
* [pg-proto-parser](https://www.npmjs.com/package/pg-proto-parser): **📦 Protobuf parser** for parsing PostgreSQL Protocol Buffers definitions to generate TypeScript interfaces, utility functions, and JSON mappings for enums.
* [@pgsql/enums](https://www.npmjs.com/package/@pgsql/enums): **🏷️ TypeScript enums** for PostgreSQL AST for safe and ergonomic parsing logic.
* [@pgsql/types](https://www.npmjs.com/package/@pgsql/types): **📝 Type definitions** for PostgreSQL AST nodes in TypeScript.
* [@pgsql/utils](https://www.npmjs.com/package/@pgsql/utils): **🛠️ AST utilities** for constructing and transforming PostgreSQL syntax trees.
## Credits
**🛠 Built by the [Constructive](https://constructive.io) team — creators of modular Postgres tooling for secure, composable backends. If you like our work, contribute on [GitHub](https://github.com/constructive-io).**
## Disclaimer
AS DESCRIBED IN THE LICENSES, THE SOFTWARE IS PROVIDED "AS IS", AT YOUR OWN RISK, AND WITHOUT WARRANTIES OF ANY KIND.
No developer or entity involved in creating this software will be liable for any claims or damages whatsoever associated with your use, inability to use, or your interaction with other users of the code, including any direct, indirect, incidental, special, exemplary, punitive or consequential damages, or loss of profits, cryptocurrencies, tokens, or anything else of value.