Over the last year, we’ve been adapting DBSSIN (Database-Spreadsheet-Ingestion) to better fit current projects, and it’s come with its share of development challenges. The ARCsoft team has been pushing the library forward, cleaning up the mapping system, and handling some unique security and access issues.
In academia, it is quite common to collect data in spreadsheets due to its simplicity and ease of use. Managing a full-fledged database is boss-level stuff in comparison. However, a database is necessary for making that data available via the web, so we're having to load spreadsheet data into databases in multiple projects. Here's how we started to build a generic solution we can use for multiple projects.
ZooDB already possessed an ingestion script, capable of efficiently processing hundreds of rows of zooarchaeological bone data. This script was traditionally executed from the command line by the developer. However, to enhance the user experience and streamline the ingestion process, we created a user-facing feature to allow the researchers to upload and process their data via the web application. In this article, I describe the crucial parts of this feature and how it was developed.
In this blogpost, we will be discussing the basics of the python library openpyxl, and about how to ingest an excel spreadsheet data into database such as SQLite.
Hello! Stephen here. From September to October I have been working on a database of excavations and bone counts for the archaeological department here at the University of Victoria–a project dubbed “ZooDB”. In this blog post, I will be going over how I went about one of my first tasks: automatically turning a spreadsheet of data into a Postgres database.