×

Wildlife Insights Bulk Uploads

Wildlife Insights Getting Started Guide

Upload your already catalogued data

If you've already catalogued your data and identified species in images, you don't need to let all that hard work go to waste! You can bring all of your old and new data together in Wildlife Insights. Continue reading below to learn how you can bulk upload all of your catalogued data into Wildlife Insights.

Note: Bulk upload services are currently only available as a paid service. If you are interested in bulk uploading your data, contact the Wildlife Insights team at [email protected] for pricing.

1. Download the batch upload templates

To batch upload your annotated data into Wildlife Insights, you'll need to make sure your data is in the bulk upload template format. The four files needed to complete a bulk upload for one project are: Project, Camera, Deployment and Image .csv files. To get started:

2. Complete the templates with your data

Enter your information into each of the templates. Refer to the Batch Upload Data Dictionary for default values, accepted formats, and required fields. Below are a few important steps to complete and recommendations to consider as you complete the templates:

  • Map taxa or common names used in your project to the Wildlife Insights taxonomy. All taxa and common names must be mapped to and changed to align with the Wildlife Insights taxonomy before bulk uploading.  The wi_taxon_id is also required for each image in column F in the images.csv file. You can find the wi_taxon_id , taxa and common names for any species using the Wildlife Insights Taxonomy Search Tool. You can also visit the Wildlife Insights Taxonomy Github repository and download the WI_Taxonomy.R file to access the most current version of the Wildlife Insights taxonomy.
  • Please avoid spaces and any special characters (hyphens and underscores are OK) in your folder paths and file names. The images.csv includes fields for your image filenames and their location on the Google Cloud Platform or other publicly accessible cloud storage. Please follow best practices and avoid spaces and special characters for these fields.
  • Leave non-required fields blank if it is not applicable to your project. If a field is not required and is not relevant to your project, please leave the field blank as you complete the template. Do not enter "NA" or "None". For example, the deployments.csv template includes a field for importing subproject_name. If your project does not use subprojects, please leave the field blank.
  • Create separate CSV files for each project. If you are uploading multiple projects, please create separate CSV files for each project. For example, if you have two projects to be batch uploaded, you will have 8 files in total: 2 project.csv files, 2 deployment.csv files, 2 camera.csv files and 2 images.csv files.
  • Save templates as a CSV and not an Excel file. To preserve date/time formats and trailing zeros, please save your completed template as a CSV file.
  • Consider building off of existing data migration code. The Wildlife Insights team has written code to transform data exported from popular camera trap software into the Wildlife Insights format. Code exists to transform data exported from Wild.ID, the Sanderson method, Camelot, and a few custom transformations. If you're writing your own custom code, this code is a great starting point! 

3. Submit your completed templates to Wildlife Insights

Once you finalize your batch upload template, please contact the Wildlife Insights team at [email protected] with a short description of your project and the completed templates. The Wildlife Insights team will validate that your templates are correctly completed and contact you about the next steps.
 

4. Upload your images to Wildlife Insights

You can transfer images to Wildlife Insights by making them available from a public URL (e.g., a Google Drive, AWS link, etc.)  or by uploading them directly to a private storage bucket in Wildlife Insights' Google Cloud Platform.

Note: Please avoid spaces and special characters (hyphens and underscores are OK) in your folder paths and file names.

Uploading images to Google Cloud Platform

A. Please share your Google account name with the Wildlife Insights admins at [email protected]. This can either be a Gmail or an associated business Google Cloud account. The Wildlife Insights admins will associate your account with a private storage object in Wildlife Insight's Google Cloud Platform and assign you access to the storage bucket. 

B. Install gsutil

  1. Please go to this Google Cloud Platform (GCP) page, select your operating systems and install gsutil.
  2. Open a terminal in MAC or Windows DOS Prompt, depending on your operating system.
  3. Enter the following at the command prompt: curl https://sdk.cloud.google.com | bash
  4. Choose a Directory to download the file to (e.g., Desktop)
  5. You’ll be asked if you want to help improve the Google Cloud SDK. Type “Y” or “N”.
  6. Type “Y” to continue when prompted to modify profile
  7. Enter the following at the command prompt: exec -l $SHELL
  8. Enter the following at the command prompt: gcloud init
  9. If prompted to pick a configuration, enter "1" to reinitialize the default configuration
  10. Choose the account you would like to use for your configuration
  11. You’ll be asked if you would like to create a project. Type “N”.

C. Upload Your Images to GCP

Once you have gsutil installed, the next step is to upload your images. The Wildlife Insights admins will create a top-level "bucket" to house images for your organization. Underneath that directory, we will create a project "folder" for each project. In each of these project folders, you will upload all of the images in your project. In this example, the bucket is called gs://zsl_image_upload/LondonHogwatch

To copy files from your local storage location (i.e., laptop, attached drive, SAN, etc.), you will need to follow these steps:

  1. Open a terminal in MAC or Windows DOS Prompt, depending on your operating system.

  2. Run the commands below to ensure you’re logged in locally with the same username you used for the Gmail or associated business Google account shared with the Wildlife Insights admins: 

    gcloud auth login gsutil config

  3. Find the local path to the folder you want to upload all the images for your project. The images.csv batch upload file should include the local path to each image. In this example, the folder name is Desktop/LH_Project1

  4. Use the following command to upload the images:

    gsutil -m cp -r YOUR_TOP_LEVEL IMAGE_DIR gs://zsl_image_upload/LondonHogwatch

The above command means the following:

-m: starts many computer threads to upload (i.e., this means it will upload more than one image at a time);
cp: copy;
-r: recursive, or that all images and directories below your "top level image dir" will be copied.

In this example, the command would be 

gsutil -m cp -r Desktop/LH_Project1 gs://LondonHogwatch