I reviewed Airflow’s documentation regarding SQLAlchemy-based metadata management. Since Cassandra does not support SQLAlchemy, can it be integrated via a similar API for Airflow?
Airflow is designed with SQLAlchemy in mind, which supports relational databases rather than NoSQL solutions like Cassandra. My experience indicates that attempting to use Cassandra as a metadata backend would involve significant custom development to bridge compatibility issues between SQLAlchemy and Cassandra’s data model. While it’s theoretically possible to develop an adapter, the effort required to maintain consistency and performance may outweigh the benefits. It is advisable to continue with relational databases or thoroughly evaluate the required modifications before considering Cassandra in a production environment.
hey, im curious about this too. i wonder if anyone tryed building a compatibility layer for cassandara and sqlalchemy. might it be bloated work or a neat workaround? what are your thoughts on balancing custom dev and reliability?