I’m working on a Python web application and need to show content from another website inside my interface. The problem is that the target site blocks iframe embedding.
My idea is to use my backend server to fetch the website content and then display it to users through my application. I know I’ll need to grab not just the HTML but also all the CSS files, JavaScript files, and other assets to make it work properly.
What’s the best approach for implementing this kind of proxy setup? What should I handle on the server side when fetching and processing the content, and how should I structure things on the client side to display everything correctly?
hmm, good point! if the css/js has relative urls, you might need to rewrite them to absolute paths. oh, and yeah, check for anti-scraping stuff too! could really complicate your setup if they limit requests or block you!
Building a proxy server for this isn’t straightforward - there’s quite a bit to think through. You’ll basically create an endpoint that grabs the target site’s content, processes it, and serves it through your app. Make sure you handle CORS headers correctly and build solid error handling for timeouts and failed requests. The tricky part is session management - lots of sites depend on cookies and auth tokens that need to stay consistent between your server and the target. Don’t forget to add caching so you’re not fetching the same static assets over and over. Also worth checking the legal side of proxying someone else’s content. Performance will take a hit since you’re adding a middleman, so use async request handling and connection pooling to keep latency down.
yeah, safety’s super important! make sure to restrict access only to trusted sites and clean any html stuff before showing it. otherwise, it’s a big risk for XSS issues, and we don’t want that!