Hey everyone! I’m working on a project where I have a daemon that checks a folder every 5 seconds for new files to process. It then creates output files when done. Now I want to add a web frontend but I’m not sure how to handle the waiting part.
Here’s what I’m thinking:
- User uploads a file through the web
- File goes into the input folder
- ??? (This is where I’m stuck)
- Daemon processes the file
- Output file appears
- ??? (How do I know it’s finished?)
I thought about checking the output folder every few seconds or using AJAX to ping a service. But I’m wondering if there’s a smarter way to do this? Maybe something with TCP or named pipes? (Can’t use remoting because of a DCOM thing)
Any ideas on how to make this work smoothly? Thanks!
From my experience with similar setups, I’d recommend implementing a message queue system. This approach decouples your web interface from the file processing daemon, allowing for more efficient communication.
Here’s how it could work:
When a user uploads a file, your web app can send a message to the queue with the file details. The daemon can then subscribe to this queue, picking up new file notifications as they come in. Once processing is complete, the daemon can send a message back to another queue, which your web app monitors.
This method eliminates constant polling and provides real-time updates. It’s scalable and can handle multiple file uploads and processing tasks simultaneously. Popular message queue systems like RabbitMQ or Apache Kafka would be suitable for this scenario.
Remember to implement proper error handling and logging throughout the system for robustness.
have u tried websockets? they could work great 4 ur setup. when the daemon finishes processing, it could send a message thru the websocket to ur web frontend. this way u get real-time updates without constant polling. its pretty easy to implement too!