I built a background service that monitors a folder every few seconds. When someone drops a file into this folder, my service picks it up, does some work on it, and saves the results to another folder.
Now I want to add a web interface so users can submit files through a browser. But I’m stuck on how to handle the waiting part. Here’s what happens:
- User uploads file via web form
- File gets saved to the input folder
- Web page shows… what exactly?
- Background service processes the file
- Results appear in output folder
- How does the web page know it’s done?
I thought of two approaches:
- Check the output folder repeatedly using JavaScript timers
- Make AJAX calls to check if the result file exists yet
Are there better ways to design this? Would something like TCP sockets or named pipes work better here? I can’t use .NET remoting because of some COM components I need to use.
Nice setup! What processing times are u seeing? If it’s just a few seconds, simple polling might work fine. But longer jobs need something better. Have u tried a message queue like Redis? Also curious about file sizes - small docs or huge uploads that could timeout?
Had the same issue last year building a document converter. Polling works but hammers your server unnecessarily. Here’s what I did instead - set up a simple database table to track jobs. When someone uploads a file, create a record with a unique job ID and set status to ‘pending’. Your background service flips it to ‘processing’, then ‘completed’ with the output file path. Now your web interface just polls that one database record instead of constantly checking if files exist. Way more efficient. You can throw in progress percentages and error messages too. On the frontend, use exponential backoff for your AJAX polling - fewer server hits as jobs run longer. Scales much better than watching the file system and you get audit trails without extra work.
What’s the processing time like? If it’s just seconds, simple polling works fine. But for longer jobs, maybe try a message queue like RabbitMQ? How heavy does your file processing get? That’d change the whole approach.
websockets are overkill but they’r perfect here. upload the file, open a websocket connection, and ur bg service pushes updates straight to the browser - no polling. did this for img processing last month and users loved watching real-time progress. if u don’t need two-way comm, server-sent events are way simpler.