chunked-upload-example/README.md

Chunked File Upload Example

This is a reference implementation for chunked, concurrent, and resumable file uploads using a minimal API and vanilla JavaScript. The implementation was developed within the constraints of using on-premise local disk storage hosted behind a reverse proxy with payload size limits that would otherwise prevent handling large file uploads.

Architecture

The storage is designed around direct random-access offset writing. As data chunks arrive, they are written directly into a single upload.part file using FileShare.ReadWrite. The exact byte position of each chunk is calculated using offset = index * chunkSize. This removes double-storage overhead by avoiding temporary file duplication and reassembly on disk.

This approach makes file assembly basically instant. Finalizing a file takes less than 1 ms and operates with O(1) time complexity because the server only needs to atomically rename upload.part to its destination name using File.Move. This eliminates the need for background worker services, disk-copying loops, or complex progress-tracking channels between the server and client.

The program uses a two-tiered SHA-256 verification model alongside deterministic session hashing to provide end-to-end data integrity across concurrent chunk transmissions:

• Per-Chunk Verification: On the client, SHA-256 hashes are calculated lazily for each chunk right before transmission using crypto.subtle.digest. Upon receipt, the server streams the chunk payload, independently computes its SHA-256 digest, and validates it against the client's hash using constant-time byte comparison (CryptographicOperations.FixedTimeEquals). Once validated, the chunk is written to disk alongside a {index}-{hash}.done marker file.
• Composite End-to-End File Verification: When all chunks finish uploading, the client orders the individual chunk hashes, joins them into a comma-separated string (hash_0,hash_1,..., hash_n-1), and computes a final composite SHA-256 hash. When /Files/{fileId}/complete is invoked, the server independently recalculates the composite hash from its stored .done marker files. The atomic File.Move rename executes only if the composite hashes match, guaranteeing that no chunks were corrupted, omitted, or reassembled out of sequence.
• Deterministic Session Identification: The session fileId is created by hashing the file's primary metadata (name-size-lastModified) into a standard GUID string. This ensures that re-selecting an interrupted file produces the exact same session key, allowing the program to query existing .done markers and resume uploading without starting over.

On the client, concurrency is controlled through a worker pool to prevent browser socket exhaustion. The client performs lazy hashing by calculating SHA-256 hashes on-the-fly right before each chunk is transmitted, keeping main thread memory usage low and the UI responsive.

Once files are successfully uploaded, the program supports immediate media range streaming. The GET /Files/{fileId}/download endpoint enables HTTP Range processing, allowing users to stream audio/video or view files directly within browser tabs without downloading the entire file first.

Note

The program is packaged as a standalone, single-file native executable using .NET Native AOT compilation. All static web assets (HTML, JavaScript, CSS) are embedded directly into the executable via ManifestEmbeddedFileProvider. To maintain compatibility with AOT compilation, the program relies on reflection-free System.Text.Json source generation.

How It Works

• Session Initialization (POST /Files/{fileId}): Client computes a GUID from the file's metadata and requests a session. Returns 200 OK for new uploads or 409 Conflict if a session already exists.
• Resumability Check (GET /Files/{fileId}/status): Server scans for existing {index}-{hash}.done marker files and returns an array of completed chunk indices.
• Concurrent Chunk Uploads (POST /Files/{fileId}/upload): Worker pool sends chunks concurrently. Chunks are verified against their SHA-256 hash and written asynchronously into upload.part at index * chunkSize.
• Completion (POST /Files/{fileId}/complete): Server validates the composite hash of all completed chunks and renames upload.part to the final filename.
• Retrieval (GET /Files/{fileId}/download): Serves the finalized file with full HTTP Range request support for playback or viewing.

Building and Running

Development Mode

Run the project directly using the .NET SDK:

dotnet run --project .\src\

Publish as a Single-File Native AOT Executable

Compile the program into a single, self-contained native binary:

# Windows
dotnet publish .\src\ -c Release -r win-x64
dotnet publish .\src\ -c Release -r win-arm64

# Linux
dotnet publish ./src/ -c Release -r linux-x64
dotnet publish ./src/ -c Release -r linux-arm64

# macOS
dotnet publish ./src/ -c Release -r osx-arm64
dotnet publish ./src/ -c Release -r osx-x64

The output binary will be located in:

.\src\bin\Release\net11.0\<runtime>\publish\