Performance regression when reading scoring files during matching
Created by: nebfield
We used to parse CSV files with polars and save IPC files, it's super fast
Streaming pyarrow batches is terribly slow in comparison (when working on UK Biobank). I think i was worried about memory usage when I wrote this.
This might also drop the pyarrow dependency in pgscatalog.core