Ingestion Made Easy!

Priya Srinivasan

ZooDB already possessed an ingestion script, capable of efficiently processing hundreds of rows of zooarchaeological bone data. This script was traditionally executed from the command line by the developer. However, to enhance the user experience and streamline the ingestion process, we created a user-facing feature to allow the researchers to upload and process their data via the web application. In this article, I describe the crucial parts of this feature and how it was developed.

The Web Ingestion feature introduces a convenient upload functionality, enabling users to effortlessly transfer their zooarchaeological bone data. This is achieved through an intuitive modal, where users can select and upload an Excel file containing the zooarchaelogical bone data. On successful ingestion, the data gets ingested without any errors.

If there are errors, the ingestion is not successful and the user can click on the “Display Errors” button to view the various errors that occurred during the ingestion, and fix them in the spreadsheet accordingly to re-ingest the data once again.

Here is the “Ingestion Errors Modal”, that displays all the errors that occurred during the previous ingestion in a modal. The user can scroll through the modal, to read the errors one by one and fix them accordingly to re-ingest it again.

These errors recorded by the logger are stored in a Django session, and then cleared once the page is refreshed.

To conclude, the introduction of the Web Ingestion feature ensures user-friendly data upload and ingestion, while offering robust error handling, including real-time logger messages displayed on an intuitive modal, this helps to ensure a successful ingestion process and seamless integration, providing an important piece towards a reliable platform for our researcher clients.

---