THE VDP FRAMEWORK: A NEW FILE FORMAT FOR PRESERVING INFORMATION IN THE DATA WRANGLING PIPELINES

Data processing is a critical yet often undocumented step in data science, frequently leading to a loss of reproducibility and provenance. This paper proposes the Verifiable Data Pipelines framework, an innovative file format designed to encapsulate datasets along with their complete transformation history. This format uses a JSON-based structure defined by a Backus-Naur Form grammar, integrating schema definitions, metadata, and an immutable record of operations. We demonstrate the framework through a command-line interface prototype that validates data integrity and logs preprocessing steps, ensuring that all modifications are verifiable. Preliminary results indicate that this framework increases transparency and facilitates the auditability of data preparation, filling a significant gap in current analytical workflows.