Processing a Photobatch in Agisoft Metashape

Overview #

After the photographs and coordinate points for a photobatch have been taken in the field, they are uploaded to the server for processing by the 3D Spatial team. The 3D Spatial team is responsible for processing the models as efficiently as possible, and for letting the excavators know when the model is “good” so that they can continue excavating in the area of the context in question.

In addition to keeping track of the photobatches themselves, you should keep track of the computer resources available to you at any given time. 3D models can be run in the background on most project computers, which themselves will be spread throughout the compound. You can connect to these computers and build models on them using Microsoft Remote Desktop. (See here for information on how to set this up and here for a list of computers that can run Metashape.)

A photobatch will be produced to represent the bottom surface of most contexts excavated at Kaymakçı. The unique identifier used for the photobatch is the photobatch number; this will be used to name most of the files produced and will be the primary key for the 3D spatial database. In some cases, a context will require a top photobatch and a bottom photobatch to be taken; in this situation, the context should only have one photobatch number, with subfolders in the photobatch folder named “top” and “bottom”. (See here for more discussion.)

What if a photobatch is incorrect/incomplete and cannot be finished? #

In some cases, a model may never be completed. This is usually the result of a context being closed at the end of the day, without enough time for the model to be run before the team leaves the field. Targets are not left in place at the end of the day because there is no way to ensure that they do not move overnight. If there is a problem with a photobatch that requires more photos and the targets have already been picked up, the photobatch must be retaken and a new photobatch number used. If this happens, the photobatch information in the Excavation database will be replaced, but the original photobatch folder and its entry in the 3D spatial database should not be deleted. Rather, the suffix “_RESHOT” should be added to the original photobatch folder, and a note should be entered into the database to indicate why the photobatch was not completed.

Photobatch Processing Workflow #

1. You will first receive a notification from an excavator via Google Chat that a new photobatch is being uploaded. The excavator should provide you with the following information: the photobatch number, the number of photos, the number of CTs (or Ground Control Points [GCPs/Targets]), the prefix for the coordinate points, the area easting and northing, and the context number. If the excavator does not provide all of this information but you do know the context number, the remaining information should be available in the main Excavation database form.
   1. Excavators may sometimes communicate with the 3D Spatial team via other channels (e.g., WhatsApp, in person). However, as much as possible, communications about new photobatches and whether models are “good” should be kept in a dedicated Google Chat created for each season; this helps us by creating an accessible archive of photobatch progress and any issues that may arise.
2. At this stage, you should decide which computer will be used to run the model. Models over 200 photos should be run on kubaba, as it is the most powerful machine.
3. After all the photos are downloaded, copy the folder to the desktop of the computer you are using (unless you are using kubaba, in which case the folder does not need to be moved). This will speed up the modeling process, as the photos will not need to be transferred over the network from the server to the computer in use.
4. Within the photobatch folder, create a new folder called “sfm”, and move all images to this new sfm folder.
5. Open Agisoft Metashape Professional and save the empty project to the local photobatch\sfm folder, using the photobatch number as the file name.
6. At the bottom of the left-hand panel of the screen select the “Reference” tab and then “Settings” at the top-right corner of the panel.
   1. Check that the coordinate system is EPSG: 32635. If it is not, set it to EPSG: 32635.
   2. Set Camera accuracy (deg) to 2
   3. Set Marker accuracy (pix) to 0.1
   4. Click OK.
7. From this point on, there is a recommended order of operations to complete the processing of the model, including multiple “checks” to ensure that the model is of high enough quality to proceed with excavation. You do not need to follow this order exactly, but you should develop a workflow or a checklist for yourself that ensures that you do not miss any steps. The recommended workflow is the following:
   1. Adding photos
   2. Aligning photos
   3. Georeferencing
   4. Working with Ground Control Points/Coded Targets
   5. Building dense point cloud
   6. Point export
   7. Building Mesh
   8. Building Texture
   9. Building Digital Elevation Model (DEM)
   10. Building Orthomosaic
   11. Exporting

8. In Metashape under “Workspace” (left hand side) select the Add Photos button or go to Workflow> Add Photos, navigate to your local photobatch\sfm folder and select all the photos.
   1. Make sure that:
      1. you select the local copy of the photobatch folder rather than the folder on the server (if you are not working on kubaba), and 
      2. any photos that were used for marking the beginning or end of a photobatch (e.g., a photo of someone’s hand) are not included.
   2. These will be automatically brought into chunk 1 in the cameras folder. Rename the chunk with the date on which it is first processed (e.g., 20230718). [If further processing is necessary after the end of the original field season, such as to increase the resolution of an orthophoto, these changes should be made in a new chunk named with the date of the reprocessing. If one needs to make multiple chunks on the same day, add alphabetic suffixes to indicate this (e.g., 20230718a, 20230718b).]
   3. If you incorporate drone photos in the model, remove its location data before processing. Under Cameras in the Reference pane, note that the drone photos have location and accuracy data that is automatically added to the project. To remove this data (which will conflict with the later georeferencing processes), select all drone photos in the pane, right click, and select “Clear.” This will open a “Clear Reference” dialog box. Make sure all the options are checked and click OK.
9. Next, go to Workflow> Align Photos and follow the steps listed here. This step will produce a sparse cloud, which you should check for any holes or errors before moving on. This is the first “check” the model must pass; if it does not pass, communicate with the excavators about collecting new/additional data.
10. Once the photos are successfully aligned and you are happy with the sparse cloud, you can proceed to Georeferencing following the steps here. This step will ensure that the models appear in the right place in 3D space and that they have a low error. This is the second “check” the model must pass; if it does not pass, communicate with the excavators about collecting new/additional data.
11. Once you have produced a good sparse cloud with a low error, the next step is to create a dense point cloud. Go to Workflow> Dense Cloud and follow the steps here. This is the third and final “check” the model must pass; if it does not pass, communicate with the excavators about collecting new/additional data.
12. Once you have an acceptable dense point cloud, you can produce some of the necessary exports (these can also be exported at a later stage if preferred).
    1. File> Export> Export Points (XYZ Point Cloud *.txt)
       1. Save in photobatch main directory (not in sfm)
          1. Settings: default
          2. filename: <photobatch number>_xyz
    2. File> Export> Export Points (LAS File)
       1. Save in photobatch main directory (not in sfm)
          1. Filename: [photbatch number]
          2. Export Points: yes
13. Once these files are exported, you are ready to build the mesh. Go to Workflow> Build Mesh and follow the steps here.
14. Once the mesh is built and the holes are closed, you can build the texture. Go to Workflow> Build Texture; use the default settings:

15. Next, you will build and export the DEM and orthophoto. These products will be used by the excavators to draw plans in the afternoon, so these need to be made ready and saved in the correct location on the server as soon as possible.
    1. To Build the DEM, follow the instructions here. 
    2. To Build the orthomosaic, follow the instructions here.

NOTE: We sometimes get behind schedule in model production, either because of technological issues or because the number of models overwhelms the available computer resources and team. If this happens, you should prioritize building the models in a way that makes it easier for the excavators to draw their day plans at the end of every day. These plans are cumulative, and therefore the contexts must be drawn in the order that they are closed. So: 1) you should prioritize the earliest contexts for each EA on each day, 2) you should not focus only on one EA to the exclusion of others, and 3) you should make sure that all models for a given EA on a given day are completed before you move on to models from the next day.

16. Now that all the other processing is complete, you will generate and export a PDF report on the model, which will help preserve its metadata for reference in the future.
    1. Go to File> Export> Generate Report
       1. Settings: default
       2. Save as pdf
       3. File path: the photobatch’s folder (not the sfm folder)
       4. File name: <photobatch number>
17. Now that the model has been processed, you are responsible for making sure that all the files get to the appropriate locations on the server. File management is an important part of the 3D Spatial team’s job. The file structure of its folder should look like what you see below, and the orthophoto and DEM exports should be filed as indicated elsewhere in these pages:
    1. Copy the entire photobatch folder (including all files and the sfm folder inside it) from its current location to the following file path on the server: E:\gygaia\3dspatial\processed\[EA]\pointclouds. The contents should include the following:
       1. sfm folder
          1. image files
          2. .files folder
          3. Metashape project(s)
       2. *.csv GNSS coordinate file
       3. *_xyz.txt (point file)
       4. LAS file
       5. PDF report