In first-week triage of messy bank exports (CSV, image PDFs, portal dumps), what tool stack has held up for you? I supervise an eight-person forensic audit team; we pushed 2.6M transactions across seven banks last quarter using IDEA + Python/pandas + Postgres for profiling and Relativity for comms linkage, but I’m weighing a shift to ACL/Diligent + Power BI for speed and audit trail — any hard lessons on accuracy, throughput, or onboarding juniors?
Quick example: we stage everything to Parquet and do ‘schema-on-read’ triage with DuckDB + Polars locally, then only push flagged populations into ACL/Power BI — https://duckdb.org has been a lifesaver when CSVs are 50–200GB, a shop-vac before the deep clean. Concrete step: on ingest, coerce types and add a source_fingerprint hash per row so dedupe and cross-bank joins are trivial, and it’s been 5–8x faster than pandas while keeping an audit trail. @OP is the current choke point image-PDF OCR or entity resolution across portals?
Pre-step: ocrmypdf+Tesseract with confidence logs; flag low-confidence pages before ACL/Power BI; ‘2.6M’ flows faster. https://ocrmypdf.readthedocs.io.
Skip the gear stipend angle; the drop-off is usually transit. We’ve kept 20–30 hour trainees by loading a $25 bus card on day one and using a simple ‘3-shift commitment’ in writing — cheap and it sticks. With your three wrapping by Mar 15, I’m open to piloting that with two starts the week of Mar 18.
What’s worked for us is a quick “fingerprint pass” on ingest: normalize date/amount/account/memo, compute an xxhash per row into a temp SQLite, and drop exact dup files/rows before any heavy lifting — cuts volume about 15% and keeps an audit trail by storing the hash ledger with the batch (GitHub - Cyan4973/xxHash: Extremely fast non-cryptographic hash algorithm). Want the tiny script?
, CSV date chaos drives me nuts; for first‑week triage on “2.6M across seven banks,” I skip pandas and load raw CSVs into DuckDB after a quick CSV→Parquet convert, then run profiling/joins in‑process before anything touches Postgres. I piggyback on @camila_j55’s hashing idea by storing the hash and source file/page in a small DuckDB staging table so exact dup files never hit your main store, and you can trace back fast. Only caveat is weird encodings — an iconv‑to‑UTF‑8 sweep first prevents brittle failures, and it should slot cleanly into your IDEA → Postgres handoff.
Quick win before tool churn: add a small “control table” in Postgres or DuckDB that logs file_id, source, in/out row counts, parse errors, checksum, and parser version per ingest, then point Power BI at it for first‑week exceptions — it’s like labeling boxes before the move. If you go ACL/Diligent, I’d still keep DuckDB for scratch joins and layer Great Expectations (https://greatexpectations.io) on the Parquet staging to auto‑flag field anomalies; @hwright76, have you tried incremental refresh on that control table to keep PBI snappy?