Web Interface for File Processing Daemon: Design Advice Needed

Frontend
1

Hey everyone,

I’ve got this program that checks a folder every 5 seconds. When it spots a new file, it processes it and creates output files in another folder. Now I want to add a web interface, but I’m not sure how to handle the wait time for processing.

Here’s what I’m thinking:

  1. User uploads a file through the web page
  2. File goes into the input folder
  3. ??? (This is where I’m stuck)
  4. Processing happens
  5. Output file shows up
  6. ??? (How do I know it’s done?)

I’ve thought about checking the output folder every few seconds from the web page, or using Ajax to ask a web service if the file’s ready. But I’m wondering if there’s a smarter way to do this.

Could TCP or Named Pipes help? (Can’t use remoting because of a DCOM thing.)

Any ideas on how to make this work smoothly? Thanks!

2

hey there! have u thought about using a message queue system? it could be pretty neat for ur setup. when someone uploads a file, u pop it into the queue with a unique ID. then ur daemon can grab jobs from there instead of watching folders. for the web part, u could make a cool status page that updates automatically. it’d show if the job’s waiting, processing, or done. this way, everything’s smoother and users can see what’s happening. maybe check out stuff like RabbitMQ? just an idea to play with!

3

hey, why not try websockets? they’re pretty cool for real-time stuff. when a user uploads a file, send em a unique ID. your daemon can use that ID to notify the web interface when it’s done processing. no need for constant polling or anything. it’s like magic - the user gets instant updates without refreshing. plus, it’s way less work on your server. just a thought!

4

Have you considered implementing a job queue system? This could streamline your process nicely. When a user uploads a file, you’d create a job in the queue with a unique ID. Your daemon could then process jobs from this queue instead of watching a folder directly. For the web interface, you could provide a status page that users can refresh or that auto-updates via AJAX, showing the current state of their job (queued, processing, complete). This approach decouples the upload from the processing, allowing for better scalability and user experience. It also simplifies error handling and provides an easy way to show progress to users. You might want to look into lightweight queue systems like Beanstalkd or RabbitMQ for this purpose.