How can I retrieve data in batches from an external SQL server and consolidate it into a single kdb+ table?

Data and Databases
1

I need to segment data extraction from an external PostgreSQL server via ODBC and merge chunks into one kdb+ table without straining the server. Revised example:

retrieveChunks:{
    sql_statement: "SELECT record_id, fname, lname, email, created_at FROM records WHERE created_at>'2020-04-01' ORDER BY record_id";
    connection: .odbc.connect `serverName;
    result: .odbc.run[connection; sql_statement];
    .odbc.disconnect connection;
    result
};
2

hey, i was wonderin if using offset with a limit might add some extra control? so curious if others have tried tweakng the query more dynamically without overloading the server. any experimintal tips are welcome!

3

i tried using cursrs for pagination instead offset-limit. it reduced errors and kept reccounts clean. might work if u face similar dupes or missing recs.