How (and why) we built the Image Database


Ben here, Gordon's son and partner in AlpenImage.  I designed and built the AlpenImage Database (with a tremendous amount of help), and I wanted to explain what it is, what it does and how it works, as well as share why and how we built it.


What is the AlpenImage Database?

The AlpenImage database is Gordon Wiltsie's complete photography archive and all associated information.  Beyond being a great way for the public to explore Gordon's collection, the AlpenImage Database is a tool that we created ourselves to streamline or automate many of the main operations for the business of photography.   It compiles all relevant information about every image in our collection, provides a management dashboard of summary data and helps us manage all the photos (and the tremendous amount of associated data).  It is basically a fancy web spreadsheet connected to lightroom that contains all our image information and stitches together the AlpenImage website, the photography archive, the webstore and our social media accounts.  I'll dive into more details later for those who are interested...

Here is what my working view of the Database looks like:

Information organized into datasheets (tabs at the top): Photo ID, Prefix, Collections, SKUs, Web store Products, Product Specs & Shipping, Unique Print IDs, Published Photos, Model Releases, Social Media, Lightroom Metadata, Inventory, Analytics plus some others. A dashboard of tools, apps and automations on the right.


Why build a database?

Besides Gordon, nobody has dealt with these images more than I have. 

Let me tell you... THERE ARE SO MANY PHOTOS, SO MANY COPIES OF THE SAME PHOTOS OR SIMILAR PHOTOS AND SO MUCH ASSOCIATED INFORMATION!  In addition to the information about the content (subjects, activities, locations, keywords) of decades of photos, we also keep detailed records for rights-managed stock photography, limited edition artist prints and model releases. 

More than 40 years of records in many different often overlapping formats. AlpenImage has always kept meticulous records but lacked organization, accessibility and summary information. For years I have joked about needing a database just to manage the information.  About a year ago, we set out to publish the entirety of Gordon's photography archive to the web and set up an assembly line to scan, archive and publish thousands more slides from archive.  We quickly realized we would need to catalogue all existing work and set up a process to import new work. While planning how to approach this huge challenge, we decided it was finally time to get organized and started compiling information to create the initial database. 


Here are some slides I created early in the process explaining the problem and proposed solution:
Our initial goals at the time (12/09/20) for V1 image database
More details on the problem and initial version work here for anybody interested (click to enlarge)

So why did we build the image database? It started out as a tool to help us organize and publish our existing catalogue and it grew from there.


What does the AlpenImage Database do?

We now use the AlpenImage Database everyday.  It currently manages the AlpenImage Photography Archive, powers our daily operations including product sales, marketing, social media, analytics and record keeping. It also supports our efforts to stay organized as we continue to scan Gordon's entire archive. And also powers the public AlpenImage Database - a cool way to browse the photography archive).

In fact, it's a monster! I view it as a 17-legged octopus with tentacles in every part of AlpenImage. The database interacts with lightroom, our website, store, social media, and is the tool we use to compile, organize and manage photos and all the associated information for web publishing, production, social media posts, sales, licensing, and much more.

Internal functionality:

Here are some of the main functions I use frequently with images below (click to enlarge):
  • Image lookup with complete information

  • Roll-up and summary information about the entire collection (or any sub-set/group of photos)

  • Summary timeline about the collection (or any sub-set/grouping of images)

  • Summary mapping and geolocation information about the collection (or any sub-set/grouping of images)

  • Manage products in the web store and create new SKUs drawing from image titles/captions in Lightroom matched with current Product Specs

  • Social Media authoring, scheduling & posting automation.  By pulling photos, captions and information directly from our database we can quickly author social media posts, schedule them and link directly back to the correct photo/gallery in the archive, or product in the store.

  • Link in bio functionality to track latest posts and associated links, captions

  • Send images and additional information & links to view on the web


External functionality:

The database was originally (and primarily still is) an internal tool we use to manage the photos, records, information and processes required by the business of photography.  However, as we began to compile more information, we found that the database could be a great tool for the public as well.  By publishing sub-set “Views” of the database, configured with select fields to be visible to the public we were able to create a number of "widgets" that provide a window into the database.  Starting this fall (Database V3), we made parts of the database public.

AlpenImage Database (Public) -  This page was built to provide the public with a comprehensive and interactive way to search, navigate and browse through the entire photography archive.  The widgets also can display image metadata (from Lightroom), or database information (from our records).  Additionally, collections, galleries and groupings can display ‘roll-up’ information such as: total images in collection, timespan, locations, published by and much more comprehensive information.  By using a combination of widgets pulling information such as links to the images, galleries, and collections inside the Photography Archive (Photoshelter) directly from our database in Airtable, we are able to build a rich interactive browsing experience for the public to search, navigate and browse through Gordon's massive collection of photography. 

Examples of live AlpenImage Database Widgets on this page:
Additionally, information from the database in a number of places:
  • Product Catalogue - Keeps an up-to-date list of all our available products (SKUs) with links to the product page in the web store

  • Mini-Print Catalogue - Sub-set of the product catalogue, links to all Mini-Print SKUs

  • Links | AlpenImage - A custom built LinkInBio clone.  This widget draws from our social media calendar to display links for the most recent social media posts. 

  • Product Specifications - Up-to-date product specs, pricing, sizing drawn directly from our database. 


How does the image database work?

 
If one were to follow a single image through the process, it would look like this:
  1. A new photo is created (either on a camera, or by scanning a 35mm slide from the slide stacks)

  2. The Digital Master image file is uniquely numbered (photo ID), processed, titled, keyworded and captioned before being transferred onto the main database server

  3. Adobe Lightroom automatically detects new files in the database and imports new additions

  4. The image is auto-sorted into Smart Collections (Galleries) inside lightroom based on location, date, keywords, rating and other variables. 

  5. The new image is published to photography.alpenimage.com in our base prefix folder structure, and any/all Smart Galleries. 

  6. Information about the new image, including the link on the web (inside the photography archive) and metadata from Lightroom are synced to the AlpenImage Database 'base'

Once a new image is in the database, and published to the web inside the photography archive:
  • The image is also automatically added to all relevant public widgets/catalogues on alpenimage.com/database and alpenimage.com/catalogue and becomes searchable. 

  • Analytics about the image (website, store, archive and social) are captured and reported back to the database including views, sales, licensing, carts, etc

We (admins) can then do stuff like:
  • Add the same underlying digital master images to infinite different galleries in the photo archive (without creating a copy)

  • Publish photo to our website, blog or 3rd party website (with linkage back to the original)

  • Add a SKU (product) to the web store using that image by combining information about the photo with a specific product spec.  Now a link to the product page is available in the database. 

  • Create, schedule and post social media posts that draw from titles, captions and keywords in Lightroom combined with hashtags, phrases, tags and mentions in our social media calendar and link to relevant products, galleries, pages or posts. 

  • Provide clients, customers or the public with any sub-set of photo/collection information via email, web, light box etc.

  • Plus all the other common internal functions shown above


Components of the synced image database

The main components of the AlpenImage Database are Adobe Lightroom, Photoshelter, Airtable, Squarespace and Google Analytics.

Here is a diagram of how the current AlpenImage Database syncs photos and information to our website, web store and photography archive.  

Key:

Circles denote the different components, arrows show the flow of data/information.  Green arrows are an automatic sync, Yellow is a triggered process, Red is a manual process.  At the top left, new photos are added into the photography archive. At the top right, new information/data are stored as records, sales, transactions, inventory, analytics, non-photo metadata into the database) In the yellow box, data syncs directly between the Photography Archive (Lightroom) and the AlpenImage Database (Airtable).  From the Photo Archive/Database, information syncs out to the photography archive (photoshelter), alpenimage.com (squarespace), alpenimage.com/store (squarespace), alpenimage.com/database via widgets, publish plugins or spreadsheet sync.  Google Analytics tracks views, interactions, sales, etc and reports the information back into the database.  (Not shown is sync to marketing calendar/automation). 


The full components (covering both the Gordon's photography archive and the AlpenImage database) include:

  • A massive hard drive containing the "Digital Masters" (all images, organized and stacked with the best available digital file of every image we have, uniquely named by "Photo ID" and organized into a folder structure we call "Prefixes")

  • A dedicated Adobe Lightroom Catalogue with very detailed organization including titles, captions, keywords, locations, dates, camera data and much more.  This catalogue auto-sorts images into a series of Smart Collections (Nth level grouping) and publishes the entire archive or sub-sets of the archive to different end-points through 3rd party and custom lightroom publish plugins. 

  • Photoshelter, provides the sync publishing and web hosting infrastructure.  This is the primary publish endpoint of the image database - We publish a complete copy of the photography archive here, referred to as the "Web Masters."

  • Airtable (an online spreadsheet with fancy features - aka smartsheet) provides the database and automation functionality.  We developed 2 custom integrations to support the critical linkage between Lightroom and Airtable. 

      • The first component - the "AlpenImage Metadata lightroom publish plugin" - uses the EXIF-tool to export all relevant EXIF, IPTC and Photoshelter metadata from lightroom to a spreadsheet.  

      • The second component is a Terminal-based Python script to sync the spreadsheet of image metadata to Airtable (LRMD tab) using the Airtable API.  These two components collectively form the lynchpin of the AlpenImage database, as they keep data (in the airtable database), in sync with the photos and their metadata (in lightroom). 

    • Once in Airtable, information is compiled and organized into these datasheets: 

      • Photo ID - every unique image we have in the archive- 39k records.  Each image has 57 potential fields of information.

      • Lightroom MetaData (LRMD) - Synced table of IPTC and select EXIF metadata of all digital master image files in the photography archive

      • Collections - Information about groupings of images including web galleries, client submissions, regions, keywords and prefixes (our base filing schema). 

      • Product Specs - Small, Medium, Large, Mini-Print, Etc. Pricing, dimensions, shipping information, descriptions and full specifications of our physical products.

      • SKUs - Our physical products including artist prints, Mini-Prints, books, cards and more.  SKUs combine information about the image (Photo ID), with information about the specific product (Product Specs). 

      • Webstore Products - The information that is synced to our web store, and contains links to product pages

      • Unique Print ID - Records of all the limited edition artist prints, transactions and associated information

      • Inventory - Tracks our SKU inventory of signed and unsigned limited edition artist prints, books and other products

      • Published Photos - Records of all published photos including client, publication, rights usage

      • Model Releases - Records of all our model releases

      • Web Analytics - Records of web analytics across the photography archive, website, webstore and social media

      • Social Media Database- Records of all social media postings, repository of future postings.  Source of social media automation and analytics. 

      • National Geographic Image Collection Data (NGIC) - Specific sub-set photo ID data containing captions, keywords, model releases and locations specifically formatted for NGIC stock photography. 

  • Squarespace is our front end website and webstore.  We sync information from the database to the store to create SKUs.  We also draw upon the photography archive (photoshelter) to host images and galleries for the website and blog posts. 

  • Google Analytics, tracks everything and records it back into the database. 


Lightroom and Image Organization

A dedicated Lightroom Catalogue with a very detailed organization system provides the image management. Smart Collections (Nth level grouping) and parallel publishing to multiple end-points through 3rd party and custom lightroom publish plugins are key features of our system.

Here are some shots explaining of the organization and automations used in Lightroom (click to enlarge):

Airtable and Data Schema

Airtable (an online spreadsheet with fancy features - aka smartsheet) provides the database and automation functionality.  We developed 2 custom integrations to support the critical linkage between Lightroom and Airtable.  Once in Airtable, information is compiled and organized into datasheets: 

Here are several images showing the database schema and how the datasheets are connected together in Airtable:  

How we built the AlpenImage Database:

At its core, the image database is just a giant spreadsheet (90k total records) with information about photos, collections, sales, transactions etc.  The main challenge in creating the image database was compiling and (re)formatting and (re)organizing information we already had. 

Here is a slide details out our initial steps in the process

The first working version of the image database used Dropbox as an intermediary step between lightroom and photoshelter.  Also we had not yet developed a mechanism to sync data between lightroom (photos) and airtable (information). 

Here is a diagram of the first working system.  

After almost a year of hard work, tons of help from some brilliant people, I am very excited that the AlpenImage Database lives! Thanks to SoEasie for diving in to fill in the all the gaps where my technical skills fall short. A huge thank you to Gordon for putting up with all the strict rules I enacted to fit square pegs (art) into round holes (structured data). Nick, my brother the systems engineer developed the custom Lightroom and Airtable Integrations- brilliant. Check out the public side of the database for yourself at alpenimage.com/database.


What's next?

The database has met and exceeded all of our original goals.  It is the single source of truth for information about every photo in the collection and currently powers our website, webstore, social media.  We will continue to update both the internal and public features of the AlpenImage database as needed.  The database has the potential to be a valuable tool for story telling by providing maps, timelines and rich additional information for any image or group of images.  In the future, I hope to be able to develop the timeline, mapping and lightbox functionality to enhance our website and blog. 

Mainly I hope that the AlpenImage Database continues to be a valuable tool to assist us in publishing the incredible body of work that Gordon has created (and continues to create) over his career. 

To date, this is our progress in building the photography archive:
There is still a lot to go… 

Thanks for the interest! If you would like to receive updates as we develop the image database and add more work to the photography archive, Subscribe to our newsletter!

If you are interested in learning more about the AlpenImage database, how it works or how we built it, feel free to reach out to me via the Contact Us page.  I'd be happy to answer any questions. 

- - Ben

Previous
Previous

A Refugee Camp in Bangladesh, 1977

Next
Next

Petra and the Wadi Rum