Using CLI tools for reproducible research workflows

The project began with a public procurement data investigation. The sources were split across a bulk publication, an API, and document metadata. The first jobs were practical: capture the source files, inspect their fields, map the available API calls, and find out what the data could support.

A small CLI made those jobs repeatable. Separate commands downloaded a source snapshot, normalized nested records into tables, profiled fields, ran retrieval rules, and wrote reports. Each command left named files behind, so a later question could start from an existing snapshot or normalization run.

The workflow used two tools for different jobs. Parquet is a columnar file format: it stores values from the same field together. That fits questions which inspect a few fields across many records. A monthly count reads dates; a document-coverage check reads document references; an identifier audit reads identifiers. The query does not need every field from every normalized record.

Each normalization run wrote its releases, tenders, awards, contracts, document references, and relationship tables as Parquet files. Those files were the preserved result of a run, alongside its source hash and manifest. A later report could start from the same files. Another tool could read them too. The normalized tables remained available as Parquet files after a DuckDB query ended.

DuckDB is an embedded analytical SQL database. It runs from the CLI and reads Parquet files directly. Each run directory remained a discrete query input. A query could inspect one table from one run, join related tables from that run, or compare tables from separate runs by reading their Parquet paths. For example, a query could inspect a table in one run directory:

SELECT status, count(*) AS records
FROM read_parquet('runs/2026-09-10/contracts.parquet')
GROUP BY status
ORDER BY records DESC;

The same pattern supported grouping activity by month, checking conflicting identifiers, counting document URLs, and writing smaller query-oriented tables. DuckDB supplied the SQL workspace for the current question; Parquet kept the normalized files available after the query ended. That split made it easy to rerun a report, compare two runs, or open the tables with another tool without moving the source data into a service.

SQLite would have been a sound choice for this small, single-user project. It keeps tables and SQL in one local database file, and it suits record updates and transactional state. JSON files and jq remain useful for looking at a raw source or extracting one field. The Parquet-and-DuckDB split suited repeated analytical queries across related records while keeping each normalization run as a portable file set. It made row-level editing inconvenient and provided no shared serving layer. The investigation had no requirement for either.

The resulting setup was small: source snapshots, Parquet tables, DuckDB queries, and reports, all operated through named CLI commands. A new question could use an existing run, produce a new report, and keep the source and query conditions alongside the result.