• Exective Summary
• 1. Introduction
• 2. Data Generation
• 3. Product Descriptions
• 4. MC&G Data Base Generation
• 4.1 Case Study I
• 4.1.1 Source of Data
• 4.1.2 Resources Expended
• 4.1.3 Production Process Assessment
• 4.1.4 Lessons Learned
• 4.2 Case Study II
• 4.2.1 Source of Data
• 4.2.2 Resources Expended
• 4.2.3 Production Process Assessment
• 4.2.4 Lessons Learned
• 4.3 Case Study III
• 4.3.1 Source of Data
• 4.3.2 Resources Expended
• 4.3.3 Production Process Assessment
• 4.3.4 Lessons Learned
• 4.4 Case Study IV
• 4.4.1 Source of Data
• 4.4.2 Resources Expended
• 4.4.3 Production Process Assessment
• 4.4.4 Lessons Learned
• 5. CAD Generation
• 5.1 Source Data
• 5.2 Resources Expended
• 5.3 Production Process Assessment
• 5.4 Lessons Learned
• 6. M&S Data Base Generation
• 6.1 Source Data
• 6.2 Resources Expended
• 6.3 Production Process Assessment
• 6.4 Lessons Learned
• 7. TEC Digital Elevation Models
• 7.1 Source of Data
• 7.2 Original Production Plan
• 7.3 DMA Data Evaluation
• 7.4 Standard DEM
• 7.5 DEMs From Other Sources
• 7.5.1 DMA
• 7.5.2 IFSAR
• 7.5.3 USGS
• 7.6 Resources
• 7.7 Lessons Learned
• 8. Warfighter Operational Evaluation Report
• 8.1 Evaluation Results
• 8.2 Recommendations
• 9. DEM Evaluations
• 9.1 DEMs Evaluated
• 9.2 Procedures
• 9.3 Results Summaries
• 9.3.1 Matrix to Matrix (post to post) Vertical Difference Statistical Comparison
• 9.3.2 Matrix to GPS Survey Control Comparison
• 9.3.3 Vegetated and Micro-Relief Evaluation
• 9.3.4 Relative Vertical Accuracy Analysis
• 9.4 Analysis of Results
• 9.4.1 DEM to DEM evaluation
• 9.4.2 DEM to GPS survey
• 9.4.3 DEM to GPS Ground Profile (Tree coverage areas)
• 9.4.4 DEM to GPS Ground Profile (Open micro relief area)
• 9.4.5 DEM relative vertical accuracy
• 9.4.6 Visual display analysis
• 10. Feature Data Accuracy Evaluation
• 10.1 Evaluation Plan
• 10.2 Evaluation Procedures
• 10.3 Results Summary
• 10.3.1 Control Bias
• 10.3.2 Error Estimation
• 10.3.3 Azimuth Checks
• 10.4 Analysis of Results
• 10.4.1 Discarded Points
• 10.4.2 Case I Errors
• 10.4.3 Case II Errors
• 10.4.4 Case III Errors
• 10.4.5 Case IV Errors
• 10.4.6 S1000 Errors
• 10.5 Summary
• 10.6 Lessons Learned

Figure 1 McKenna Site and Surrounding Study Area (Ft Benning, GA)

Figure 2 McKenna MOUT Site (From DMA Geodetic Survey, May 1996)

Figure 3 Aerial View of McKennat MOUT Site from Northwest

Figure 4 Virtual View of McKenna MOUT Site from Northwest

Figure 5 Generic M&S Database Construction

Figure 6 Comparison Between The Four Data Generation Cases

Figure 7 Relative Positions of Control

Table 1 Differences Between DMA Coordinates and Measurements on the Coordinate Display Form

Table 2 Differences Between Checkpoints Used for Control Between the PSI and DMA Coordinates

Table 3 Variation Between PSI and DMA Coordinates

Table 4 Comparison Between PSI and TEC Coordinates

Table 5 Azimuth Angle Differences

1. Introduction

As a step toward this goal, the Defense Modeling and Simulation Office (DMSO) and the Defense Mapping Agency (DMA) - now part of the National Imagery and Mapping Agency (NIMA) - sponsored a project to produce and evaluate digital terrain data for the McKenna MOUT (military operations in urban terrain) training site at Fort Benning, Georgia. NIMA's Terrain Modeling Project Office (TMPO) is the Department of Defense modeling and simulation executive agent for the authoritative representation of terrain. During May 1995 through August 1996, TMPO, the DMA St. Louis production facilities, the Topographic Engineering Center (TEC), the Institute for Defense Analyses (IDA), TASC, Inc., GDE, Inc., LNK, Inc., LADS-Belleview, the TRADOC Dismounted Battlespace Battle Lab (DBBL), the Marine Corps Modeling and Simulation Management Office (MCMSMO), and several other organizations planned, constructed, and evaluated both digital and paper high-resolution terrain products of the MOUT site. These products included: stereo overhead imagery, detailed geodetic and Global Positioning System (GPS) site surveys; mapping, charting, and geodesy (MC&G) elevation and feature files; Geographic Information System (GIS) databases; micro-terrain profiles; orthophotos with feature overlays; 3-D anaglyphs; maps; radar, video, and other site imagery; 3-D computer-aided design (CAD) site models of McKenna building exteriors and interiors; texture libraries; and modeling and simulation (M&S) run-time databases for virtual reality (VR) simulators. Data were collected for both McKenna and the Griswold live fire range. High-resolution data were inserted in a larger runtime database employing USGS elevation data and feature information for a 24 Km x 24 Km region surrounding Griswold and McKenna sites. The three regions are depicted in Figure 1.

The McKenna training facility was chosen as the focus for this project due to the strong interest of the Army Training and Doctrine Command (TRADOC) and the Ft Benning Battle Lab. McKenna is used for the training of US and allied infantry, marines, and law enforcement personnel. It consists of a mock European village containing 15 primary buildings, three support buildings, and an underground sewer system with five manhole cover (MHC) pop-up points. Figure 2 provides the village layout. Letters inside building outlines match the DBBL reference system (used in evaluation questionnaires), while outside letters and MHC codes correspond to the DMA site survey codes. The village is approximately 90 x 150 meters in area, and can be approached through woods. The terrain immediately surrounding the village is composed of grassy and brushy open areas, thick forests, swamps, streams, ponds, some sharp erosional features, and many light/loose surfaced roads and trails. The 5 Km x 5 Km region is approximately 80 percent wooded and exhibits variations in relief of about 70 meters. It contains a 1,125-meter dirt airstrip capable of landing C-130 aircraft, and a 300 x 300 meter helipad. Figures 3 and 4 provide live and virtual aerial comparisons of the village.

2. Data Generation

3. Product Descriptions

DMA provided the extraction rules used for generating all the high resolution feature databases. The extraction rules were based on the existing Interim Terrain Data (ITD) specification, enhanced with features and attributes agreed between DMA and the DBBL (Annex B). The enhanced ITD added to the feature description by supplying further information about buildings, point drains, roads, transportation, soils, lakes, rivers, vegetation, sewers, linear features, wrinkles, gullies, and obstacles.

Three dimensional models were created to provide the exterior and interior building details.

These three file types (DEM, ITD, and 3D Models), when combined, form the complete McKenna MOBA terrain database (MOBA TDB). One method to combine them is in the S1000 format database. This provided the user with the tools to manipulate and use the MOBA TDB.

4. MC&G Data Base Generation

4.1 Case Study I

4.1.1 Source of Data

4.1.2 Resources Expended

4.1.3 Production Process Assessment

4.1.4 Lessons Learned

4.2 Case Study II

4.2.1 Source of Data

4.2.2 Resources Expended

4.2.3 Production Process Assessment

4.2.4 Lessons Learned

4.3 Case Study III

4.3.1 Source of Data

4.3.2 Resources Expended

4.3.3 Production Process Assessment

The most serious concerns were due to the presence of extraction artifacts known as cornrows, which are a product of manually produced DTED. Historically, cornrows have not presented significant data quality problems, since the elevation post spacing on standard DTED products are quite large. With close grid spacing, however, cornrows are quite numerous, especially in heavily forested areas, where the "ground" location is uncertain. Although the effect of cornrows can be greatly reduced through processing techniques, there is a concern that such processing may adversely affect data quality.

4.3.4 Lessons Learned

The lessons to be learned from this are twofold: First, there must be a clear understanding at the outset of a project of all hardware and software inventories and capabilities for both data generators and users. Secondly, there should be a standardized tape and file naming system. Furthermore there should be a standardized means of listing and describing all data files with this information appearing, not only on the exterior label of the tape, but, internally, in soft copy. With this method, even if the tape label was lost, not copied, or incorrectly copied, a complete list of all necessary items could still be obtained by opening and reading the tape.

There were two image-related issues that need adjustment for future work. Although high resolution MC&G imagery was a requirement for this project, it took six months to get it. Missions requiring such imagery must plan sufficient lead time for imagery acquisition or arrange for higher priority for imagery procurement. The other image-related issue concerns the season when the imagery was taken. Use of winter imagery would have resulted in much more accurate elevation data sets.

4.4 Case Study IV

4.4.1 Source of Data

4.4.2 Resources Expended

4.4.3 Production Process Assessment

Image related difficulties originated from the necessity of creating small, numerous models. The small size of each aerial photograph, along with the overlap and sidelap requirements for the stereo use of such imagery dictated the creation of small stereo models. Since data extraction models can be no larger than the stereo models in which they are contained, many small data extraction models were also necessary. Model creation and closing, as well as the required internal quality assurance checks, are time consuming processes, and the greater the number of models to process, the greater the associated processing time. The MC&G imagery (used in Case Studies II and III) required the construction of only 1 stereo model each, and from 1 to 10 associated data extraction models (depending on the case study), while the aerial photography used in this case study required 28 stereo models and 31 data extraction models.

Time consuming difficulties related to the software had to do with processing of individual photo strips, conducting a software fix allowing image exploitation, and time saving automated processes that did not work well. Software restrictions allowed only one photo strip to be processed at a time. The result was that the photo analyst could not quickly move from one end of the project to the other, since it involved the cumbersome and time intensive processes of activating the proper photo strip, orienting the correct stereo model, and, finally, opening the appropriate extraction model, each being a 20 to 30 minute process.

The FE/S was designed to exploit hard copy imagery sources. Since DMA uses hard copy MC&G imagery on the FE/S almost exclusively, the orientation processes and procedures for conventional photography were not completely developed, somewhat cumbersome, and more time consuming than the orientation process for MC&G imagery. Before extraction could take place, Fortran code had to be written, allowing the ellipsoid heights in the control data to be converted to MSL values as required by the FE/S.

Several processes and procedures typically used as shortcuts to save time in the data extraction process (the use of the auto node and auto tag processes, and the use of maintenance data) could not be utilized on this project. These processes save much time and effort, since manual placement of nodes and tags is quite time consuming. On this project, however, both processes generated numerous errors, requiring much repair time. Use of auto node and auto tag was discontinued, necessitating the labor and time intensive manual approach.

There was also a hardware related problem. The FE/S hardware has a built-in restriction in it's magnification (zoom) capabilities. While this does not adversely affect extraction from MC&G imagery, it proved to be very restrictive with conventional aerial photography. The photo analyst could not "zoom up" to get an overall view of an area, in order to get a better idea of how features should be split apart or included together. Instead, mapping had to take place at a high magnification level, which does not allow the analyst a view of the "big picture", and as a result, interpretations of feature outlines required much editing.

The combination of small model sizes, restricted "zoom" capability, lengthy orientation and photo strip processing times, and inability to utilize such time saving processes as auto node, auto tag, and maintenance data, made it difficult to map, to get an overall view of the mapping area, and to easily change previously compiled linework from another photo model. More processing time directly correlates to less data extraction time, especially when a limited time frame for project completion is considered.

4.4.4 Lessons Learned

Geodetic crews should be involved in coordinating the preflight plan with the aerial photography contractor. This would allow the geodetic team to locate additional identifiable, suitable GPS control points on the ground.

When the geodetic crews were in the field, the aerial photography, photocopies of the imagery, charts and point sketches were used as guides to identify and locate the preselected control points. Since there was only one set of paper prints of the imagery available for use, individual crew members had to use photocopies of the exposures to locate the control points. In most cases, these photocopies did not show enough detail to be of much use. If duplicate sets of paper prints had been available, multiple field crews could have accessed the imagery at the same time, thus ensuring confidence in point site recognition, as well as speeding up the process.

The lessons to be learned from this are twofold: First, there must be a clear understanding at the outset of a project of all hardware and software inventories and capabilities for both data generators and users. This can easily be accomplished by exchanging lists of all hardware items, including their capacities, and all software items, including the versions used. Secondly, there should be a standardized tape and file naming system. All output tapes should clearly identify the data producer, the case study number(s), the data set(s) contained on the tape, all data and file formats, and a list and short description of the content of all data files. Furthermore there should be a standardized means of listing and describing all data files with this information appearing, both on the exterior label of the tape and in soft copy. With this method, even if the tape label was lost, not copied, or incorrectly copied, a complete list of all necessary items could still be obtained by opening and reading the tape.

There was also an image-related issue concerning the season when the imagery was taken. Both the MC&G imagery and the conventional aerial photography were "summer scenes" with full foliage on the deciduous trees. Imagery taken in the middle of winter, when all of the deciduous trees have lost their leaves, would have allowed a better view of the ground. Use of winter imagery would have resulted in much more accurate extraction over forested areas, and possibly would have made the extraction of an elevation data set by DMA for this case study feasible .

Finally, the type of imagery used to generate the best data set possible was originally believed to be the conventional aerial photography. While image interpretations made in open areas were not a problem, the overall dark tones and general lack of variability in tonal patterns, made differentiation of vegetation types in heavily forested areas quite difficult at best, and impossible at worst. The MC&G imagery was a better image source, at least in heavily vegetated areas, then the conventional aerial imagery.

5. CAD Generation

5.1 Source Data

5.2 Resources Expended

5.3 Production Process Assessment

Another observation is the shadowing introduced by the photographs. Since the photos were not taken at the same instant in time, shadowing on the buildings and ground provide "conflicting" information that the observers' brain will try to differentiate.

Still photos of building faces were shot in black and white. Texture map file format (RGB, JPEG, TIFF) can handle color. We were able to "synthetically" introduce color into the MOUT by assigning color attributes to the buildings, walls, and ground.

The process of digitization of the drawing coordinates is highly labor intensive and requires high productivity tools. Mixing detailed engineering drawings with photographs of the actual buildings was hampered by problems of misalignment due to both camera positional distortions and builder modifications not captured by the original drawings.

5.4 Lessons Learned

6. M&S Data Base Generation

Case IV represented the most challenging and interesting case for M&S database production. Use of unconstrained sources accentuated the importance of selecting the "best available" source data from alternative sources. However, this, together with the requirement to also process a 24 Km x 24 Km background maneuver area at SIMNET density, led to additional time and labor resources needed to accomplish front-end GIS processing of the available feature data. Additional data had to be acquired and processed from available CONUS sources (i.e. US Army Waterways Experimentation Station, US Geological Survey, US Census Bureau) to achieve comprehensive source coverage for the entire database footprint.

6.1 Source Data

6.2 Resources Expended

6.3 Production Process Assessment

TIN Processing also exceeded initial expectations. This was the result of a combination of factors, including the high density of the elevation data being processed, the interaction of high density road data with the elevation data in integrated TIN simplification, and the inexperience of personnel who were using the TIN generation tool for the first time.

Manual editing and texture application proved to be time consuming, but the end result was well worth the effort in the visual effect achieved by high resolution, photospecific texturing.

Production of the 24 Km x 24 Km database, into which the McKenna MOUT site was inset, proved to be slightly more time consuming than the population of the Case IV inset. GIS processing, S1000 population, and associated tasks (less the editing and texturing of the McKenna MOUT site models) took a total of 718 man-hours for the 24 Km x 24 Km maneuver box, versus 636 man-hours for the 4 x 4 high density McKenna MOUT site area. These levels of effort generally correspond to labor resource levels required for previous SIMNET database projects.

M&S database design was complicated by uncertainty as to the specific requirements and performance specifications of target simulators, including real time graphics, simulation host, and computer generated forces applications that will use this database now and in the future. We anticipate that additional database enhancements and modifications will be necessary to support currently fielded, and potential future simulation platforms. Experience gathered in testing and evaluating this database to date indicates that significant increases in the quality and efficiency of simulation platform performance will be needed to enable this database to be run at full efficiency in a virtual environment. Heightened interaction between simulation systems developers and M&S TDB producers will assist in identifying specific shortfalls and required database enhancements.

More detailed revisions to the 3D CAD models of the McKenna MOUT site models were found to be necessary, in order to remove extraneous and inefficient vertices and segments from the 3D models, and to create two levels of detail for each McKenna building model. These measures were taken in order to optimize real time updates in the 3D visual systems, which would have been degraded by retaining the model geometry as digitized from engineering blueprints. Photospecific texture application was done concurrently with modifications to 3D model geometry

TIN processing methods also diverged from the initial project plan/process model. In the initial plan, only one or two TIN iterations were planned with no subsequent modifications to 2D features as input to a CMU iTIN tool. A significant change to the initial process was repeated iterative processing of the terrain skin and associated input feature data in a CMU iTIN tool and ARC/INFO. The reprocessing of the TIN terrain surface was necessitated by excessive smoothing of the terrain surface polygons in the CMU iTIN tool output, which required multiple passes to create adequately detailed terrain surfaces. Evaluation of TIN processing output was conducted both before and after the TIN data was exported into S1000. Rather than attempting to modify the TIN surface and associated 2D transportation features in S1000, necessary 2D feature edits were done in ARC/INFO, followed by subsequent reprocessing of the TIN terrain surface in the CMU iTIN tool.

6.4 Lessons Learned

Explicit tailoring of source data products to fit modeling and simulation database design specifications is a necessary and unavoidable evil. Experience during this project was that the database producer must be able to "pick and choose" what data can and cannot be directly imported for use in the terrain database, what data will be retained for analytical purposes only, and what data will not be used at all.

Generation of multiple levels of detail building models was essential to make this database usable on even the highest performance real time graphics platforms. Real time performance would also be improved if S1000 were capable of processing hierarchical TINs at multiple levels of detail, and morphing forest canopies into individual tree models and stamps.

A very large number of site-specific photographs were made available to support the M&S database production team. These sources proved essential to communicate visual database content, and were even more vital links in the process, since none of the M&S database production team participated in the site surveys at Ft Benning. Had S1000 modelers been available to provide guidance to the original on-site photographers, and the initial photographs been taken in color, redundant photography of the McKenna MOUT site could have been alleviated.

More software tools are needed to optimize TINs, develop textures, build adjustable phototexture libraries, accurately position building models, provide quality control, and permit value adding,

7. TEC Digital Elevation Models

7.1 Source of Data

7.2 Original Production Plan

7.3 DMA Data Evaluation

Table 1 Differences Between DMA Coordinates and Measurements on the Coordinate Display Form
 

Table 2 shows the differences between the 6 checkpoints used for control between the PSI and DMA as measured in this dataset. These were measured the same way as the above data. The data shows that there is very little difference between the PSI and DMA control comparisons.

Table 2 Differences Between Checkpoints Used for Control Between the PSI and DMA Coordinates
 

7.4 Standard DEM

7.5 DEMs From Other Sources

7.5.1 DMA

7.5.2 IFSAR

7.5.3 USGS

7.6 Resources

7.7 Lessons Learned

Photo Scanning: Select a pixel size between 20 and 30 microns for the resolution of the digital data. Make sure that 8-bit data is captured as close as possible. Scan the full image including the photo identification data. This will make it easier to insure that the intended photo is viewed on the photogrammetric workstation. If the camera position needs to be entered in the header file, the above-mentioned floppy of input data should be loaded on this computer and the data pasted in.

Block Triangulation: The selection of weights for the camera and ground control points is important. If differential GPS has been used, set the weight of the ground control points to the expected accuracy of that point. If it is a panel point, a weight of 0.05 meters is recommended for xyz. For the camera control, which should be adjusted as discussed in paragraphs above, a weight of 0.05 is recommended for X and Y, but a higher weight should be used for the Z, such as 0.5 meters. This will take care of the weakness in determining the z offset between antenna and camera and the general weakness in Z determination of the camera. Another method is to fly the camera first with the differential GPS over a test area with many control points for a self calibration of the Z offset.

DEM Creation: Create DEMs with a small overlap between adjacent models. Run some sample tests first (if not familiar with the strategy files) to determine the best strategy file to use with the imagery. Use the contour display and primarily the geomorphic editor to edit the data. If there are large bodies of water, use the area editor (area fill) to edit these polygons before correlation to speed up the correlation process.

8. Warfighter Operational Evaluation Report

8.1 Evaluation Results

1. Useful high-resolution digital MC&G elevation and feature data were provided to describe the McKenna MOUT site at Fort Benning, GA.

 

2. High-resolution requirements exist; however one-meter data are not needed uniformly throughout a region, or for every circumstance.

 

3. Data capture under forest canopy is a continuing problem requiring further research and the aid of advanced sensor technologies.

 

4. Less than one percent of the MC&G high-resolution data can now be directly accommodated in SIMNET runtime load modules.

 

5. The majority of warfighters queried stated that database features should be positioned within five meters of their true location; A near majority wanted positional accuracy within one meter for dismounted operations.

 

6. More capable computer image generators are needed to support high-resolution, urban infantry operations.

 

8.2 Recommendations

1. Cognizant organizations should produce technical additions and improvements to MOBA environmental databases and dismounted warrior simulation capabilities. These would include:

 

• Exploring autocorrelation and radar technologies for deriving improved high-resolution DEMs.

 

• Building upon the existing Fort Benning MC&G data repository to develop improved Phase 2 runtime databases.
• Finding means of better representing vegetation, micro terrain, and other key features.
• Working the Multiple-Z (i.e., multiple feature elevations at an X-Y location) and data topology issues in both the McKenna and future high-resolution databases.

 

2. The DMSO should furnish an improved capability to simulate individual combatant behaviors. Actions would include:
• Continued effort to improve representations for dismounted operations. This includes improvement to hardware and environmental reasoning software germane to simulation interfaces (precision control of dismounted forces and stealth views).

 

• Improved toolkits to assist in the generation of M&S products.

 

• Improved ModSAF, ModStealth, and other Distributed Interactive Simulation (DIS) software.
3. The Army Deputy Chief of Staff for Intelligence should endorse requirements for better data collection and production, as well as improved mission simulation systems. Based on the results of this evaluation, we recommend the earmarking of resources for improvements in databases and tools critical to the responsive production of topographic products for the warfighter.

 

9. DEM Evaluations

These specific evaluations were performed on each test DEM: 1) Absolute Vertical Accuracy was established by comparing each DEM to the GPS survey points and to the base (control ) DEM. Range, mean and standard deviation of differences in Z (vertical plane only) were calculated; 2) "Cornrow"/Artifacts: The DTED level 4 data set extracted on the BC1-s stereoplotter was evaluated in a "raw" state and after post processing to minimize the manual profile (cornrow) artifact; 3) Vegetated Areas: Each test data set was compared to an accurate (sub-meter relative accuracy) field terrain profile generated by acquiring a dense profile of GPS survey points collected in heavily vegetated areas. The vegetation in this area was primarily composed of a large pine tree canopy; 4) Micro Relief: Each data sets' ability to depict micro relief was evaluated by comparison to accurate (sub-meter accuracy) field data (GPS survey points) collected over a terrain profile; 5) Relative vertical accuracy was evaluated by comparing pairs of points from the control micro relief profile data to each test data set; and 6) Various visual displays were also generated to evaluate the characteristics of the data sets. These included shaded relief, wireframe and contour portrayals.

In addition to the evaluations listed above there were additional evaluations planned for this project. Two evaluations that were not completed, as software development requirements could not be completed in time, were 1) A DX, DY, DZ shift for each DEM compared to the control DEM was to be calculated and applied to remove any horizontal or vertical shifts detected in the test DEMs relative to the control DEM; and 2) A more rigorous relative vertical accuracy was to be computed by comparing pairs of points in the control DEM with each test DEM over varying distances

9.1 DEMs Evaluated

1. meter post spacing DEM (equivalent to DTED lv 5) produced by TEC/GDE (control DEM).

 

2. DTED level 2 ( 30 meter post spacing)produced by NIMA

 

3. DTED level 3 ( 10 meter post spacing) produced by NIMA

 

4. DTED level 4 ( 3 meter post spacing) produced by NIMA (without ground control)

 

5. IFSARE DEM (2.5 meter resolution) produced by ERIM.

 

9.2 Procedures

DEMs were evaluated on the Interactive Quality Review System (SUN/UNIX environment) utilizing ARC/INFO GRID functionality with NIMA enhancements (DTED TOOLS). The TEC produced LV5 DEM was used as the control DEM data set. GPS survey data produced by NIMA was used as survey control. The data was loaded to the IQRS and DEM to DEM statistics were generated via ARC/INFO STAT functionality. DEM to survey statistics were generated via ARC INFO SEARCH functionality.

9.3 Results Summaries

9.3.1 Matrix to Matrix (post to post) Vertical Difference Statistical Comparison

9.3.2 Matrix to GPS Survey Control Comparison

9.3.3 Vegetated and Micro-Relief Evaluation

Profile 1:
 

9.3.4 Relative Vertical Accuracy Analysis

9.4 Analysis of Results

9.4.1 DEM to DEM evaluation

1. The NIMA data sets appear to be lower (mean) than the GDE/TEC data sets (level 3 and 4 about 2 or 3 meters, level 2 about 5 meters). The NIMA data sets were generated from standard MC&G triangulation solutions without incorporating the GPS survey results, thus this difference is not surprising. The GDE (and TEC) data sets were generated from imagery which utilized the GPS survey as triangulation control.

 

2. The GDE level 5 data set to TEC control data set comparison yielded larger than expected 90% (8.5 meters) and min/max range values, this may indicate that some "intermediate" data set generated by GDE was shipped to NIMA. All of the level 5 data sets were supplied to NIMA via TEC.

 

3. The large negative difference (min) reported for the IFSARE comparison occurred around the lake area, and appeared to possibly be caused by layover (trees?) close to the boundary of the lake.

 

9.4.2 DEM to GPS survey

1. NIMA data is confirmed to be low to the GPS survey (level 3 and 4 about 2 meters, level 2 about 4 meters).

 

2. The GDE/TEC data sets yield extremely low 90% values from this comparison. These results can be attributed to two factors; the increased resolution of the data sets (less interpolation error in the SEARCH directive) and the fact that the imagery was triangulated using this very data set as control.

 

9.4.3 DEM to GPS Ground Profile (Tree coverage areas)

1. None of the data sets did a particularly good job of portraying the ground surface under the tree canopy. This conclusion is based on analysis of the graphical profile data generated via SPYGLASS Transform software.

 

2. The IFSARE and GPS generated data sets were portraying elevations more closely representative of the top (or near to the top) of the tree canopy (reflective surface).

 

9.4.4 DEM to GPS Ground Profile (Open micro relief area)

9.4.5 DEM relative vertical accuracy

9.4.6 Visual display analysis

1. The TEC control level 5 data set and the GDE level 5 data set exhibit extremely high resolution detail on shaded relief displays in the open (dirt) areas, and are for the most part devoid of extraction/edit artifacts. It is obvious that the tree covered areas were interpolated to this resolution from a less dense post spacing.

 

2. The NIMA data exhibits the manual profile artifacts a.k.a. "corn rows". NIMA cartographers attempted to profile the ground surface through the tree cover. Generally the severity (magnitude) of the artifact varies with the obscuration factor (in this case tree cover). Open (dirt) areas are less affected than tree covered areas. Attempts to edit out this artifact using production software are generally only semi-successful as evidenced by the "SPRINT" data displays.

 

3. The DPS produced level 3 data exhibits very good resolution in the open (dirt) areas, with little extraction or edit artifacts evident. The tree areas were intentionally left as extracted by the correlator, only large positive/negative spikes were removed.

 

4. The ISARE data also does a good job of portraying the ground surface in the open (dirt) areas. This data set also portrays tree canopy (reflective surface) heights over dense tree coverage areas, similar to the DPS correlator solution.

 

10. Feature Data Accuracy Evaluation

An absolute and relative horizontal positional accuracy evaluation of the feature data sets prepared by DMA and GDE for the MOUT Project was conducted by TEC. This evaluation, which was performed in the Topographic Technology Laboratory (TTL) of TEC, is closely related to the evaluation of other quality aspects of the feature data sets performed by the Military Operations in Built-up Areas Terrain Database Technical Evaluation Working Group. Some of the insights that resulted from the metric accuracy assessment are included in this report and were provided to TEC's Digital Concepts and Analysis Center (DCAC) for use in the working group's assessment. The source data (photogrammetric data base) used in the TTL evaluations was also used to provide products for use in the Operational Warfighter Evaluations.

10.1 Evaluation Plan

10.2 Evaluation Procedures

During the early part of the evaluation, a variation between the PSI and DMA GPS Control Surveys was discovered. Table 3 shows the differences at the six paneled ground points established by PSI. The DMA Triangulation Report shows the same approximate differences. DMA applied these offsets to its photo stations before doing triangulation. There was considerable discussion among DMA, TASC, NOAA, and TEC about these differences, but the consensus seemed to be that the results seen are at about the limits of GPS positioning accuracy. TEC received informal information that a DMA resurvey of the area did not resolve the differences.

The TEC Photogrammetric Block was initially triangulated using only the PSI supplied GPS Control for the camera stations. Later two of the paneled ground points established by PSI were incorporated into the solution in an attempt to improve the vertical accuracy of the block. In addition to the internal checks and evaluations made by TEC in triangulating the photogrammetric block, some additional accuracy assurances were obtained by comparing coordinates of ground points established by GPS with those read from the block.

Table 4 compares coordinates measured photogrammetrically at TEC with the PSI GPS coordinates for the three paneled points which fall within the block. Except for one of the elevations, the two sets of coordinates agree to within less than 0.1 meter at these three points. This indicates a high degree of consistency between the block and points used to control it.

The horizontal bias (most probable error) detected closely approximates the differences between the DMA and PSI Surveys. Consequently, the TEC Coordinates were shifted by the amount of the difference between the DMA and PSI Coordinates. The comparison of these shifted coordinates to the DMA Coordinates shows only a slight bias (a circular error of .08 meter) and standard deviations of .36 meter in easting and .23 meter in northing. From these values a CPE of .35 meter was calculated. Thus, 50 percent of the well defined points within the block will be within .35 meter of their true locations relative to other points in the block and virtually all points (99.78%) will be within 1.04 meters. With respect to the DMA Control, 50 percent will be within .43 meter and virtually all within 1.12 meter. Given the nature of the test points this is a very good match. At the three paneled ground points established by PSI, the match is .16 meter (circular) or less. Two of the points with larger differences, 69017 and 69022, were omitted in the DMA Triangulation Report indicating that DMA also had trouble with them. For three others, DMA also showed large differences, though not necessarily in the same coordinate.

The vertical bias has not been explained. TTL members used equations provided by TEC's Geodetic Applications Division to check some of the camera station elevations used in the triangulation. No significant differences were found. The source of error may be in converting the elevation from aircraft antenna to camera lens, although the amount of error seems rather large for this explanation. Note that the vertical bias decreased to less than one-third meter after the ground control was incorporated into the triangulation. The DMA Triangulation Report indicated vertical biases of approximately the same magnitude.

10.3 Results Summary

10.3.1 Control Bias

10.3.2 Error Estimation

10.3.3 Azimuth Checks

The largest errors are in Case II which is not surprising, given that the imagery was smaller in scale and the buildings were much smaller. The most probable errors indicate a counterclockwise rotation of from .23 to .44 degrees between Cases I, III, and IV and the TEC Photogrammetric Block. The Case II data seems to be rotated by nearly 2 degrees in the other direction. The SIMNET S1000 azimuths show little overall bias from TEC. Azimuths over a longer baseline, from Point 10 to Point 42, were compared to check the validity of these conclusions. For this baseline, TEC, Case I, and Case IV Azimuths were essentially identical. The Case III Azimuth still showed a counterclockwise rotation, but by about half as much. The Case II Azimuth now showed a counterclockwise rotation of about one degree.

The S1000 Azimuths, individually, do not track their source (Case IV) too well, but have an internal consistency that is not much worse.

10.4 Analysis of Results

10.4.1 Discarded Points

10.4.2 Case I Errors

1. The maximum easting error in this data set occurred on Point 37, a lone tree. TEC found this to be one of the more well-defined trees and the Case IV compilers positioned it much more accurately. The maximum northing error was on Point 9, a road fork. From the diagram of the point, it appears that the roads had to be intersected by TEC. The roads meet at a shallow angle making it difficult to position accurately. Nearby, there is a fenced installation with a small building and a tower of some sort which would have provided excellent test points, but none of the compilers included it.
2. Some of the other largest errors in this case were in compiling hard stand corners (Points 13, 14, and 15) which were obscured by trees and shadows. Enough sections of the edges were visible to have allowed determining their alignments well enough to position the corners much better.
3. Point 30 might have proved better if GDE had connected the trails at the intersection. TEC had to connect the end nodes of the two cross trails with a straight line. The large error on Point 26 could have been avoided by taking more care in delineating the timber boundaries. Several other of the larger errors were on lone trees which were definitely difficult to see in the stereo models.

 

10.4.3 Case II Errors

1. The largest northing error in this set is on Point 5, a trail fork or trail end. On the large scale imagery, there does not seem to be any indication of a trail extending to the point included in Case II. The largest easting error was on Point 14, a hard stand corner. Some other of the larger errors were on other similar corners. While TEC has not seen the imagery used, the comments concerning these points in the GDE analysis are believed to be applicable here, too.

 

2. Large errors were also made in delineating Point 26, a vegetation/lake or wetlands boundary. This, as in the previous case, seems to be a lack of care in delineating the boundary.

 

3. Other significant errors were on buildings, which were probably quite small on the imagery, and road intersections. (Points 7, 8, 10, and 34.) The latter may have been caused by trees obscuring the side roads, although the road alignments appear different from the other cases.

 

10.4.4 Case III Errors

1. The maximum easting error in this set was on a point (16, a clearing corner) on one of the added features. This particular clearing was determined to be wetlands based on ground truth verification. The boundaries shown do not seem close to the edge of the clearing except in the northernmost section of the plate. The delineation was distorted by being connected to an edge from Case II. The largest northing error was also on an added feature (26, a vegetation corner). Though not readily discernible on the ortho, there is a pretty clear right angle in the tree boundary at this point which none of the compilers has shown properly.

 

2. Except for Points 9 and 10, road intersections from Case II, which were not very accurate, were retained. The intersection at Point 10 was redone much more accurately and the intersection at Point 9 was added fairly well. All the buildings; Points 1-4, 36, 39-44, and 46-48; were redone without in general improving the accuracy of their delineation. Also retained were some of the least accurately positioned hard stand corners in Case II (11 and 13-15).

 

3. Of the other features added, the one that created the road fork designated Point 28 was the least accurate. This was because the trail added was joined to another trail from Case II which was not correctly aligned. A fairly large error at the drain fork (Point 24) is due to poor alignment of the drains added. The trail added to form Point 28, a T intersection, is poorly aligned resulting in a significant error at that point. The same is true at Point 31. Some of the other larger errors were caused by tying to Case II features which were misaligned. This compiler, like the others, had understandable difficulty with the lone trees.

 

10.4.5 Case IV Errors

1. The largest error is a northing error for Point 5, a trail end. This trail was dropped about 16 meters short of the trail junction delineated in Case I. The largest easting error was on a hard stand corner, Point 14. Like the others, this compiler did not use the edges of the hard stand to position the corner correctly.

 

2. The next most significant error was on Point 9, a road fork. The same comments as in paragraph 15.6.2 1. above apply. This is a point that perhaps should have been omitted. Another large error is seen at Point 34, a T intersection. The side road in this data set is drawn as if it curved into the other road.

 

3. Other relatively large errors were experienced in delineating hard stand corners. The size of the error at Point 6, the road and railroad crossing, is surprising. A similar error at Point 26, a vegetation corner, is the lowest for this point in any of the cases. However, the delineation of the timber edge is not any better. Most of the other large errors were in delineating lone trees, which was a problem for all compilers.

 

10.4.6 S1000 Errors

10.5 Summary

Compilers using care and perhaps improved techniques could lower the range of error considerably. Evidence that this is possible was provided by recalculating the statistics based on the most accurate 30 points in each case, except Case II which only included 28 of the test points. The accuracies estimated from these limited points are shown below. The control biases also changed in both magnitude and direction, decreasing to .17 meter for Case I and increasing to 1.29 meters and .84 meters for Cases III and IV, respectively. This provides further indication that the plotting accuracies are the major source of error.
 

10.6 Lessons Learned

Imagery to be used for generation of data to be evaluated should be flown at the ideal time. The Fort Benning Area, with its dark coniferous tree cover and light sandy soil, is definitely a difficult area to image. The situation was made worse when the imagery was flown early in the morning adding many large shadows to the already high contrast imagery. Larger scale imagery flown later in the year and more nearly in the middle of the day is much clearer and allows a much more accurate positioning, even of lone trees. Incidentally, use of a digital photogrammetric system for data compilation in difficult areas such as this offers the advantage of being able to adjust the image photometry in any particular situation to improve positioning accuracy.

Standard coverage naming and feature classification conventions should be established in advance and given to all compilers. Also, the projection and datum should be standardized in advance to avoid errors that might occur in converting from one to another.

Some changes in techniques for and more care in compiling the features should provide improved accuracies. Compilers could be given some idea in advance about how the comparisons will be done.

The techniques for overlaying feature data on orthophotos used for this project was very effective in comparing the data sets. This technique may have potential in quality control.