I’m having trouble with my Django app using Celery for async tasks. I’ve got Rabbitmq as the broker and Redis for results, both on an Ubuntu 14.04 VM. Celery workers are running on Windows 10 machines.
Here are the issues:
Tasks remain in the ‘PENDING’ state even after completion or failure
Failed tasks do not retry, and I receive a connection refused error
The results backend doesn’t seem to be working
My setup details:
The Django server has the Celery configuration but no workers
Remote Windows machines execute the Celery tasks
When I intentionally change the broker settings to an incorrect password, I get connection errors. However, altering the result backend settings does not affect task execution.
Redis is configured for remote access (with bind 0.0.0.0 in redis.conf).
Could anyone help identify what might be causing these issues? Thanks!
It sounds like you’re facing a tricky situation with your distributed Celery setup. From my experience, cross-platform configurations can be particularly challenging. Have you verified that your Windows workers can actually connect to the Redis backend on your Ubuntu VM? It might be worth checking firewall settings and network connectivity between the systems.
Another potential issue could be with serialization. Ensure that you’re using a serializer that’s compatible across your different environments. I’ve found that using ‘json’ as the serializer can help avoid some cross-platform pitfalls.
Regarding the tasks remaining in ‘PENDING’ state, this often indicates that the result backend isn’t being updated properly. Double-check your CELERY_RESULT_BACKEND setting in Django and make sure it’s correctly pointing to your Redis instance.
For the retry issue, you might want to look into your task retry settings. Sometimes, network hiccups can cause connection refused errors, so implementing proper retry logic can help mitigate this.
hmm, interesting setup! have u considered using a cloud-based redis service instead? it might simplify things. also, whats ur CELERY_TASK_SERIALIZER setting? sometimes switching to ‘json’ can help with cross-platform issues. curious to hear if youve tried running everythin on the same OS? might help isolate the problem
hey ava89, sounds like a headache! have you tried checkin if the windows workers can actually reach the redis server? might be a firewall thing. also, make sure your celery settings in django are pointing to the right redis address. sometimes its the simple stuff that trips us up!