64 lines
5.1 KiB
Markdown
64 lines
5.1 KiB
Markdown
|
# Chunked & Resumable File Upload Example
|
||
|
|
|
||
|
|
This is a reference implementation for chunked, parallel, and resumable file uploads using a minimal API and vanilla JavaScript.
|
||
|
|
|
||
|
|
The implementation was developed within the constraints of using on-premise local disk storage hosted behind a reverse proxy with payload size limits that would otherwise prevent handling large file uploads.
|
||
|
|
|
||
|
|
## Architecture
|
||
|
|
|
||
|
|
The storage is designed around direct random-access offset writing. As data chunks arrive, they are written directly into a single `upload.part` file using `FileShare.ReadWrite`. The exact byte position of each chunk is calculated using `offset = index * chunkSize`. This removes double-storage overhead by avoiding temporary file duplication and reassembly on disk.
|
||
|
|
|
||
|
|
This approach makes file assembly basically instant. Finalizing a file takes less than 1 ms and operates with O(1) time complexity because the server only needs to atomically rename `upload.part` to its destination name using `File.Move`. This eliminates the need for background worker services, disk-copying loops, or complex progress-tracking channels between the server and client.
|
||
|
|
|
||
|
|
To guarantee end-to-end data integrity across parallel chunk transmissions, the program uses a two-tiered SHA-256 verification model alongside deterministic session hashing:
|
||
|
|
|
||
|
|
- **Per-Chunk Verification**: On the client, SHA-256 hashes are calculated lazily for each chunk right before transmission using `crypto.subtle.digest`. Upon receipt, the server streams the chunk payload, independently computes its SHA-256 digest, and validates it against the client's hash using constant-time byte comparison (`CryptographicOperations.FixedTimeEquals`). Once validated, the chunk is written to disk alongside a `{index}-{hash}.done` marker file.
|
||
|
|
- **Composite End-to-End File Verification**: When all chunks finish uploading, the client orders the individual chunk hashes, joins them into a comma-separated string (`hash_0,hash_1,..., hash_n-1`), and computes a final composite SHA-256 hash. When `/Files/{fileId}/complete` is invoked, the server independently recalculates the composite hash from its stored `.done` marker files. The atomic `File.Move` rename executes only if the composite hashes match, guaranteeing that no chunks were corrupted, omitted, or reassembled out of sequence.
|
||
|
|
- **Deterministic Session Identification**: The session `fileId` is created by hashing the file's primary metadata (`name-size-lastModified`) into a standard GUID string. This ensures that re-selecting an interrupted file produces the exact same session key, allowing the program to query existing `.done` markers and resume uploading without starting over.
|
||
|
|
|
||
|
|
On the client, concurrency is controlled through a worker pool to prevent browser socket exhaustion. The client performs lazy hashing by calculating SHA-256 hashes on-the-fly right before each chunk is transmitted, keeping main thread memory usage low and the UI responsive.
|
||
|
|
|
||
|
|
Once files are successfully uploaded, the program supports immediate media range streaming. The `GET /Files/{fileId}/download` endpoint enables HTTP Range processing, allowing users to stream audio/video or view files directly within browser tabs without downloading the entire file first.
|
||
|
|
|
||
|
|
> [!NOTE]
|
||
|
|
> The program is packaged as a standalone, single-file native executable using **.NET Native AOT compilation**. All static web assets (HTML, JavaScript, CSS) are embedded directly into the executable via `ManifestEmbeddedFileProvider`. To maintain compatibility with AOT compilation, the program relies on reflection-free `System.Text.Json` source generation.
|
||
|
|
|
||
|
|
## How It Works
|
||
|
|
|
||
|
|
- **Session Initialization (`POST /Files/{fileId}`)**: Client computes a GUID from the file's metadata and requests a session. Returns `200 OK` for new uploads or `409 Conflict` if a session already exists.
|
||
|
|
- **Resumability Check (`GET /Files/{fileId}/status`)**: Server scans for existing `{index}-{hash}.done` marker files and returns an array of completed chunk indices.
|
||
|
|
- **Parallel Chunk Uploads (`POST /Files/{fileId}/upload`)**: Worker pool sends chunks concurrently. Chunks are verified against their SHA-256 hash and written asynchronously into `upload.part` at `index * chunkSize`.
|
||
|
|
- **Completion (`POST /Files/{fileId}/complete`)**: Server validates the composite hash of all completed chunks and renames `upload.part` to the final filename.
|
||
|
|
- **Retrieval (`GET /Files/{fileId}/download`)**: Serves the finalized file with full HTTP Range request support for playback or viewing.
|
||
|
|
|
||
|
|
## Building and Running
|
||
|
|
|
||
|
|
### Development Mode
|
||
|
|
|
||
|
|
Run the project directly using the .NET SDK:
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
dotnet run --project .\src\
|
||
|
|
```
|
||
|
|
|
||
|
|
### Publish as a Single-File Native AOT Executable
|
||
|
|
|
||
|
|
Compile the program into a single, self-contained native binary:
|
||
|
|
|
||
|
|
```powershell
|
||
|
|
# Windows
|
||
|
|
dotnet publish .\src\ -c Release -r win-x64
|
||
|
|
dotnet publish .\src\ -c Release -r win-arm64
|
||
|
|
|
||
|
|
# Linux
|
||
|
|
dotnet publish ./src/ -c Release -r linux-x64
|
||
|
|
dotnet publish ./src/ -c Release -r linux-arm64
|
||
|
|
|
||
|
|
# macOS
|
||
|
|
dotnet publish ./src/ -c Release -r osx-arm64
|
||
|
|
dotnet publish ./src/ -c Release -r osx-x64
|
||
|
|
```
|
||
|
|
|
||
|
|
The output binary will be located in:
|
||
|
|
|
||
|
|
`.\src\bin\Release\net11.0\<runtime>\publish\`
|