How to handle big file uploads (3GB+) in FastAPI without memory overflow?

Backend
1

I need to handle huge file uploads in my FastAPI application. The files are around 3GB or bigger, but my server only has 2GB of available RAM. I want to process these files without loading everything into memory at once.

Backend code:

@app.post("/file-upload")
async def handle_file_upload(file_data: UploadFile = File(...)):
    # processing logic here
    return {"status": "success"}

Client code:

filename = "large_archive.tar.gz"
encoder = MultipartEncoder(fields={"file_data": open(filename, 'rb')})
base_url = "http://localhost:8000"
endpoint = f"{base_url}/api/file-upload"

response = requests.post(
    endpoint,
    data=encoder,
    headers={'Content-Type': encoder.content_type}
)

I keep getting a 422 error saying the field is missing. The error message shows:

{"detail": [{"loc": ["body", "file_data"], "msg": "field required", "type": "value_error.missing"}]}

I think the issue might be with how MultipartEncoder formats the request. What am I doing wrong here?

2

Your multipart field naming’s wrong. Change the encoder to use ‘file’ instead of ‘file_data’, or update your FastAPI endpoint to match. For the memory issue - use shutil.copyfileobj() to stream straight to disk. Works great for huge uploads without eating RAM.

3

interesting issue! are you streaming the file chunks properly? i’d try using file.read(chunk_size) in a loop instead of loading everything at once. also, test with a smaller file first - that’ll help you figure out if it’s a multipart issue or memory problem.

4

The 422 error you’re encountering indicates that your multipart form data isn’t being recognized properly. Your endpoint expects a parameter named file_data, but it seems that MultipartEncoder isn’t sending it with the correct field name. Consider using the standard files parameter in your requests instead of MultipartEncoder.

To handle the file upload efficiently and avoid memory overflow, read the file in chunks with await file.read(chunk_size), where chunk_size can typically be set to 8192 bytes. Additionally, using spoolfile from the tempfile module allows you to write the chunks directly to disk as they arrive, which significantly mitigates memory issues, even when working with files over 5GB on servers with limited RAM.