Hey folks, I’m working on a project where I’ve got this Python script that crunches a ton of data into a memory structure. It’s got these cool fetch functions to grab info from it. Now I want to slap a basic web interface on top, but here’s the catch: I can’t afford to reload the data every time someone hits the page. It’s just too much of a resource hog.
What I’m after is a way to keep the Python interpreter alive between web requests, so my data stays put in memory. I know PHP’s exec() is out because it fires up a new Python instance each time. Someone mentioned mod_python might do the trick, but I’m not sure.
Any ideas on how to pull this off? I’m open to different approaches as long as I can keep that data structure hanging around in memory. Thanks in advance for any pointers!
Have you considered using Flask with a WSGI server like Gunicorn? This setup can maintain your Python environment between requests. You could initialize your data structure when the app starts, then access it in your route functions. For persistence across server restarts, you might want to look into periodically serializing your data to disk or a database. This approach gives you the flexibility of a web framework while keeping your data readily available in memory. Just be mindful of thread safety if you’re modifying the data structure from multiple requests simultaneously.
have u considered using a persistent process manager like supervisord? it could keep ur python script running constantly, and u could set up a simple API endpoint to query the data. that way, ur data stays in memory between requests. might be worth exploring! what kinda data are u working with, btw?
have u tried using gunicorn? it can keep ur python app running as a persistent process. pair it with nginx as a reverse proxy and u got a solid setup. gunicorn can manage multiple worker processes too, so u can handle more requests without reloading data each time. might be worth a shot!