Search All

Our Website

1. Microsoft Word - NCGDAP_interim_rept_Draft60108b.doc

   https://www.lib.ncsu.edu/ncgdap/documents/NCGDAP_InterimReport_June2008.pdf

   ...ata is generally of much higher resolution than the state/federal data, with data resolution ranging from six inches ground surface per pixel in urban areas to two feet per pixel in some rural a... Read more

   ...ata is generally of much higher resolution than the state/federal data, with data resolution ranging from six inches ground surface per pixel in urban areas to two feet per pixel in some rural areas. Statewide orthophotos at a lower resolution (one meter) were created through a combined state and federal effort for the years 1993 (black and white) and 1998 (color infrared). More recently statewide efforts have been supplanted by a combination of individual county flights and regional collaborative flights. Orthophoto flights are increasingly carried out under federal cost sharing programs. In North Carolina 99 counties currently have digital orthophotos and counties typically conduct orthophoto flights roughly every two to five years. File sizes for NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 11 an individual county flight can total in the hundreds of gigabytes, and 79 counties have had multiple flights.8 Overall frequency of orthophoto flights is increasing as is the quantity of data generated in each subsequent flight. Digital Maps A wide variety of digital maps are also being acquired. These maps represent intellectual content and meaning beyond that found in the underlying datasets, as the maps result from the combination of a number of components including: data layer selection and ordering, symboliz ation, classification, output of data models, and annotation. Some of these digital maps are georeferenced or produced in a multi-page atlas format. The rapid increase in production of PDF maps at the county level since project start was a major surprise and introduced unexpected project challenges and opportunities in the area of PDF selection and curation. Key Finding: PDF has emerged as a significant geospatial format. The ability of PDF to capture and preserve elements of cartographic representation makes it a powerful tool for capturing finished output in a way that the underlying datasets cannot. At the same time, complex PDF documents, including those in the proprietary GeoPDF format, present new preservation challenges of their own. There has been a surge of interest on the part of local agencies in the issue of making their older, analog content available in digital form either for their own project development or for public consumption.9 Local agencies are increasingly building time into their applications, and the general public is keenly interested in seeing this historic information. To the extent that this content already exists in digital form the content will be targeted for acquisition by NCGDAP. More to the point, the local interest in historic content provides an opportunity to connect with those agencies on the issue of preserving current digitally-born content. Key Finding: There is significant local agency interest in resurrecting old analog maps for use in the digital environment. This interest creates a point of engagement and dialog around the issue of preserving current geospatial data for use in future historical analyses. Historical map imagery also helps to create an emotional connection to the issue of data preservation. Tabular Data Tabular data are numeric or textual data stored in database, spreadsheet, comma separated value, or other like formats. A given geographic feature may be assigned many different attribute values based on tabular data that might be associated with that feature. Some tabular data may be associated with geographic features such as land parcels or census tracts. This project focuses solely on non-federal tabular data occurring at a granularity greater than county level. This largely concerns tax assessment data, which associates land parcel records with such attributes as property value, purchase price, purchase date, building type, construction date, square footage, zoning, land use, and 8 Inventory of County Orthophoto Flights in NC: http://www.nconemap.com/Portals/7/documents/ortho_dates.pdf 9 Local agency interest in digitizing older analog content was documented in the NCGDAP Frequency of Capture Survey: http://www.nconemap.com/Portals/7/documents/NCOneMap_NDIIPLocalGovSurvey_1106.pdf NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 12 owner name. These data are produced by county tax assessment agencies and are made publicly available according to public records law, often bundled together with land parcel vector data. Risks to Geospatial Data While key feature data layers such as land records, street centerlines, jurisdictional boundaries, and zoning are constantly changing, current data management practice commonly involves overwriting of older versions of data which are then no longer available for historical or trends analysis. Emerging web services- or API-based technologies pose further challenges to the archive development process as it becomes easier to get and use data without creating a local copy—secondary archives often being in part a by-product of providing data access. Even if the data has been saved, there is a chain of possible failure events that can impede permanent access to data: • To the extent that such data is saved, it may be stored in such a way that it is not discoverable. • If the data is discoverable, polic ies may not have addressed the issue of what sort of access should be provided to older versions of data. • If the data is accessible, there is a possibility that the storage media will no longer be readable. • If the media is readable, the data files themselves may be corrupt. • If the files are not corrupt, it is possible that the files will be in a format that is no longer supported by current software. • If the format is useable, it is possible that the documentation needed to use and understand the contents of the data will not exist. Unlike vector data, digital orthophotography is not typically at risk of overwrite, yet data from older flights are known to have become less discoverable and less accessible. While digital geospatial data inherits preservation challenges that apply to digital resources in general, this content area also presents a number of domain-specific challenges to the preservation process. Unique Data Formats Geospatial vector data file formats are highly complex and extremely sensitive to both format migration and software display environment, and backwards compatibility between versions of the same software can be problematic. The absence of widely adopted, open vector formats ensures that a preponderance of vector data exists in proprietary formats. Due to the complexity of the content, migration between formats can lead to unacceptable data distortion and data loss. Data resources are often available from the data producers in multiple formats. Content may be available in “early stage” (e.g., pre-rectification orthoimagery), “middle stage” (e.g., rectified and quality-controlled orthoimagery), and “late stage” (e.g., compressed orthoimagery for delivery) formats. Data formats at one or more of the stages may be more NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 13 suitable for long-term preservation. Information may be both added and lost during the early-to-late progression. For example, sometimes only the delivery version of a parcel data set includes attached assessment data. Conversely, deliv ery versions of images may be in lossy compressed formats in which information is lost. Data in late stages is usually easier to acquire than data in early stages . Spatial Database Complexity The emergence of spatial databases has further complicated the preservation of digital geos patial data. Spatial databases may consist of multiple individual datasets or “data layers,” while also storing components such as behaviors, relationships, classification schemes, data models, or annotations that are external to or in addition to the datasets themselves. The whole of the spatial database is greater than the sum of the parts, as database components that build on the individual data layers add value. These complex databases can be difficult to manage over time due to the complexity of data models, uncertainty over long-term support of proprietary database models, and reliance on specific database back ends for data storage. Local agencies are increasingly turning to spatial databases to manage geospatial data. Fragility of Cartographic Representation The true counterpart to the old, preserved map is not the current GIS dataset but rather the cartographic representation that builds on that data. The representation is the result of a collection of intellectual choices and application of current methods with regard to symboliz ation, classification, data modeling, and annotation. Unfortunately this representation is typically stored either: a) in proprietary project file for which there is no preservation-safe alternative; b) in a complex PDF document in which the underlying data linkages have been severed; or c) as part of a web services-driven interface for which the displayed results are ephemeral. Exporting or converting these complex documents to preservable image formats captures the data view but loses the underlying data intelligence. There are semantic concerns as well, as the symbologies employed have particular meanings within particular contexts at particular points in time. While the original proposal focused more exclusively on the data itself, growing awareness of the long-term value of these representations led to a focus on this issue that was not envisioned in the original work plan. Clearly, however, any preservation of cartographic representation should occur in addition to—not instead of--preserving the underlying data. Key Finding: The true counterpart to the old, preserved map is not the current GIS dataset but rather the cartographic representation that builds on that data. The representation is the result of a collection of intellectual choices and application of current methods with regard to symbolization, classification, data modeling, and annotation. These representations typically occur in a complex proprietary project file format (difficult to preserve) or in an ephemeral web services interaction. Increasingly PDF is providing an option for static representations. Semantic Issues Heterogeneous approaches to dataset naming, attribute naming, and attribute classification schemes create both short- and long-term barriers to understanding and use of content. NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 14 Data producers are discovering that naming and coding inconsistencies complicate the process of data sharing even in the context of present day use. While good metadata can make it possible to interpret these components, such metadata is unfortunately often absent or may not include the data dictionaries associated with names and codes found in the data. “Framework data” content standards provide some hope for improved consistency in the content and structure of geospatial data.10 Time-Versioned Content At the local level many vector data resources are continuously or at least periodically updated. County cadastral (land parcel) datasets, for example, are typically updated on a daily or weekly basis. Such time-versioned content, if preserved, can form the basis of time series analyses such as land use change analysis. Time-versioned content presents three distinct challenges to the archiving process: • The updated data in many cases is simply over-written or otherwise modified with no digital knowledge of the historic version maintained. • Even if a data provider captures historic information, the absence of a standard identifier scheme, such as associated with serial public ations, makes it difficult to relate data versions outside of a local data collection context. • An optimal capture frequency is difficult to determine for any particular type of data given the significant variation in update frequencies among data producers.11 Metadata Unavailability or Inconsistency In the Unites Stated, the geospatial metadata standard since 1994 has been the Federal Geographic Data Committee (FGDC) Content Standard for Digital Geospatial Metadata, commonly referred to as FGDC metadata.12 Geos patial metadata presents several distinct challenges: • In terms of government data, while FGDC metadata is mandated at the federal level, it is less common at the state level, and only rarely available at the local level.13 • In cases where metadata is absent, an archive may able to populate some FGDC record sections, yet only the data producers have the information needed to populate sections such as data quality and lineage. • Even if metadata exists, the metadata information is often asynchronous with the data (e.g., the metadata may not have been updated to reflect format or datum change) or the metadata may simply be incorrect. 10 Framework data comprises seven themes of geospatial data (geodetic control, orthoimagery, elevation, transportation, hydrography, governmental units, and cadastral information) used by most GIS applications. These data include an encoding of the geographic extent of the features and a minimal number of attributes needed to identify and describe the features. From: FGDC Framework: http://www.fgdc.gov/framework. Framework data standards and practices are also addresses at the state level, such as in the case of the NC GICC Standards: http://www.ncgicc.com/Default.aspx?tabid=141 11 NCGDAP Frequency of Capture Survey Results: http://www.nconemap.com/Portals/7/documents/NCOneMap_NDIIPLocalGovSurvey_1106.pdf (See Appendix A) 12 FGDC Content Standard for Geospatial Metadata: http://www.fgdc.gov/metadata/geospatial-metadata-standards 13 According to the 2003 NC Local Government GIS Data Inventory only 26% of local agencies were creating FGDC-compliant metadata: http://www.nconemap.com/Portals/7/documents/local-inventory-pack.zip. In the NCGDAP Frequency of Capture Survey 25% of responding local agencies indicate that they were archiving FGDC-compliant metadata with their data. NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 15 • Since the original FGDC standard was a content standard for which no standard encoding was defined; existing metadata commonly requires some degree of structural normalization in order for the metadata to be interoperable with a repository. • The FGDC standard, while extensive, does not provide container spaces for the added technical and administrative metadata elements needed for archival processes. Examples of metadata not supported by the standard include: data of data acquisition by the archive, rights of the archive vis-à-vis the data, plans for future transfers, technical information about method of transfer, and method of assuring integrity of the data. Content Packaging Geospatial data is characterized by complex, multi-file formats. In addit ion, datasets are often accompanied by metadata and ancillary documentation or data files which need to be bundled with the core dataset files. Furthermore, archival technical or administrative metadata elements not accommodated by FGDC records, such as non-producer rights information, must be bundled up with the data in some other way. Unfortunately, the geos patial industry has not adopted a standard content packaging scheme. Ancillary files include metadata records, data dictionaries, additional data documentation, legend files, data licenses, disclaimers, and associated images. In many cases an individual ancillary file will be shared by many or all datasets in a given collection, creating the requirement that the ancillary files either be replicated for bundling with individual datasets or referenced separately through a persistent access mechanism. Other Archiving Challenges Other preservation challenges include: • Securing and adequately defining archival and use rights for content • Providing long-term support of coordinate systems and datums • Maintaining the independence of the preserved content from any particular repository software environment Source Organizations The scope of NCGDAP includes state and local geospatial data in North Carolina. Data produced at the state level is typically more detailed and current than data produced at the federal level but not so detailed and current as data produced at the county and municipal level. Data producing roles at the state versus the local level do vary somewhat state to state within the U.S., with some states following a decentralized data production pattern, as found in North Carolina, and other states having data production more strongly centralized at the state level. The scope of data production at the different levels of government in North Carolina are outlined here. NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 16 State Agencies In addition to NCCGIA, over 20 state agenc ies in North Carolina have active GIS programs that are involved with geospatial data production and use.14 Key agencies include the Department of Transportation (NCDOT), the Department of Environment and Natural Resources (NCDENR), the North Carolina Flood Mapping Program, the Department of Agric ulture, and the Department of Health and Human Services. Both NCDOT and NCDENR include several sub-departments with their own GIS programs. Many of the geos patial data resources created by these state agencies are included in the NC OneMap viewer and data download systems managed by NCCGIA15, yet many more data resources are not available in a central state repository. Data from these agencies are produced direc tly by state agencies, or are produced by private contractors, with the data coming into the ownership of the state agency under public records law. Some resources are developed in collaboration with or under a cost sharing agreement with federal agencies (e.g., orthophotography and surface hydrography).16 County Agencies Geospatial data development has typically been initiated by the county tax assessment (or land records) office and focuses on development of resources such as high-resolution digital orthophotos, street centerlines with addresses, and municipal boundaries, all of which are needed for tax assessment operations. Digital orthophotos provide the base material for creation of vector layers such as land parcels and so are typically the first data resource created. This data is usually much more detailed and more current than data available from the state or federal level. Much of the data is initially produced by private contractors for the county, which then takes ownership of the data under public records law and continues data maintenance. Base data, including orthophotography, is often taken by other county departments in order to create additional data layers for the county. As county GIS programs mature, the data comes to be employed by a wide range of agencies outside of tax operations, and more data layers representing features such as school locations, utility lines, elevation, and land use zones are created. To meet broader demand for GIS services, operations are often later moved under the IT department or into a separate GIS department in order to provide services to the entire county government. Municipal Agencies Many municipalities in North Carolina have developed GIS systems, which are used as part of ongoing work in areas such as land use planning, zoning, utilities management, park and open space planning, and emergency response. City governments typically acquire the relev ant county data as a base resource and then develop additional, needed data layers as a result of ongoing operations.17 14 State agency data resources are documented in the NC GIS Inventory: http://www.nconemap.com/GISInventory/tabid/288/Default.aspx 15 NC OneMap Map Services Catalog: http://www.nconemap.com/Default.aspx?tabid=298 16 Cost Share Opportunities and NC OneMap: http://www.nconemap.com/Home/CostShare/tabid/279/Default.aspx 17 NCSU Libraries NC City GIS agency list: http://www.lib.ncsu.edu/gis/cities.html; municipal data resources are listed in the NC GIS Inventory: http://www.nconemap.com/GISInventory/tabid/288/Default.aspx NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 17 Lead Regional Organizations Lead Regional Organizations include Council of Governments (COG) and Municipal Planning Organizations (MPO). There are 17 COGs in North Carolina, with each county in the state having membership in one of these regional groups, which are voluntary associations of county and municipal governments. These regional associations address issues which are better addressed at the regional rather than local level, such as economic development and environmental protection. Many of the COGs provide GIS services to their constituent counties and municipalities.18 The 17 MPOs in North Carolina19 have responsibility for planning, programming and coordination of federal highway and transit investments. Regional agencies sometimes acquire data from constituent counties and municipalities in order to assemble seamless regional datasets or create new datasets that use the local data as a base. Other Data Producing Organizations Data is also available from university research groups. This data is often derived from other state, federal, or local data resources. Facilities data from university campuses is available in some cases. Non-governmental organization data is available from groups such as Triangle Land Conservancy and private land-holding organizations also maintain extensive GIS datasets pertaining to assets. Data Inventories in Content Identification and Selection One of the biggest challenges in archive development is determining what data is available in the 100 counties and many municipalities in North Carolina. Earlier experience had shown that a content identification approach based on individual phone calls to agencies does not scale and also threatens to damage the overall network by contributing to data producer contact fatigue. The information acquired in this manner is spotty, subjec t to transcription errors, and quickly becomes outdated. Formalized, comprehensive, routinized inventory processes which have been vetted by the data producers and stakeholders more efficiently serve general industry data discovery and access needs while also supporting archival efforts. An added benefit is that inventory systems can produce basic metadata for documented data resources. Key Finding: A major challenge of the archive development process lies in minimizing the intrusion on the time of local agency data producers which, especially in rural counties, operate with small staffs which are often as small as one person. “Contact fatigue” arising from redundant requests for data or for information about data holdings can stretch local staffing resources and blunt enthusiasm for local participation in infrastructure-based approaches, such as formal inventories. Historical Inventories Since the mid-1990s, a number of efforts have been undertaken to track the spread of GIS activity among local government agencies. These inventories typically have been 18 GIS Capabilities of North Carolina Councils of Government: http://www.tjcog.dst.nc.us/gis/giscaps.htm 19 Image Map of Councils of Government and Lead Regional Organizations: http://www.tjcog.dst.nc.us/nccogs.shtml NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 18 incomplete given the complexity of the task of surveying 100 counties and 140 cities. All of these inventory results became obsolete quickly in the absence of an ongoing process for update. In the early stages of the project NCGDAP undertook an analysis of these older surveys in order to find out what information is available. The project also sought to learn from past inventory experiences in order to provide input into future data survey efforts. Following is a list of key state and federal data inventories or surveys: Survey Year Scope Response Rate FGDC National Geospatial Data Framework Survey20 1997 US Over 200 state, local, and federal agencies responded to the NC component of the survey. NC County GIS Survey21 1997 NC 94 out of 100 NC counties responded to this survey. NCSU Libraries Local Government Data Acquisition Project22 2000 NC Information was informally gathered for roughly half of NC counties. NC Floodplain Mapping Program23 2000-2001 NC Information was gathered for all NC counties in order to support the flood mapping effort. NC Dept of Transportation Surveys 2000?-present NC Selected information is maintained for all counties to support NCDOT needs. NC Dept of Agriculture Surveys 2001?-present NC Selected information is maintained for all counties to support Dept. of Agriculture needs. NC Local Government GIS Data Survey24 2003-2004 NC The initial survey gathered data for 91 counties and 92 (out of 141) municipalities. NC Flood Mapping Project survey information was used to fill gaps. NC GIS Inventory25 2006-present US The NC GIS Inventory (ongoing) comprises the NC portion of RAMONA. 64 counties and 36 municipalities had responded as of Dec. 2007. 2003 NC Local Government GIS Data Survey Initially, the 2003 NC Local Government GIS Data Survey served as the primary content identification resource for the project, and results from the survey were incorporated into various project reports characterizing content within the project domain. The survey was extremely detailed, with 218 questions covering such topics as contact information, rights and sharing statements, and data layer availability. Specific survey information of use in the project content identification and selection process included: • Contact information 20 1997 FGDC Framework Data Survey results available from Internet Archive: http://web.archive.org/web/20050415063150/http://www.fgdc.gov/framework/survey_results/download.html 21 1997 NC County survey results available from Internet Archive: http://web.archive.org/web/20061102190756/http://www.cgia.state.nc.us/cosurvey97/index.html 22 Archived data and inventory information available from the NCSU Libraries County and City GIS Directories: http://www.lib.ncsu.edu/gis/counties.html and http://www.lib.ncsu.edu/gis/cities.html 23 NC Floodplain Mapping Program: http://www.ncfloodmaps.com/ 24 2003 NC Local Government GIS Data Survey http://www.nconemap.net/Portals/7/documents/local-inventory-pack.zip 25 NC GIS Inventory: http://www.nconemap.net/GISInventory/tabid/288/Default.aspx NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 19 • Data holdings information • Information about rights issues • Format, coordinate system, datum, etc. • Distribution method In order to inform future inventory efforts the project also considered possible survey improvements that would support preservation efforts. One key suggestion included providing more granular information about data formats for different data layers in order to generate data for use in measuring format utilization as a risk assessment metric. At the project outset it was anticipated that the various earlier surveys would point out resources that did not surface in the survey. For example, county agencies will sometimes omit older orthophotos that were originally produced as hard copies (“mylar”) and later scanned and georeferenced since these resources are older and of lower quality than more current products. Orthophoto flight years recorded in the NC OneMap inventory information was compared with NCSU, NCDOT, and NCDA records as well as the '97 NC County survey. Discrepancies, particularly in the way of omission of older flights, were reported for inclusion in the NC OneMap orthophoto flight history. Possible Future Work with Inventories Historical surveys and inventories might be used for future research in terms of time series analysis focused on: • Shifts in use of formats over time • Changes in agency responsibility for geospatial data management and services • Data availability • Trends in access and distribution policies • Shifts in use of commercial software packages Key Finding: Key data community documents such as inventories, standards, and policy or best practices documents must themselves need to be archived. In more than one case inventory information that had been retired from agency websites was retrieved from the Internet Archive. Present-Day NC GIS Inventory The 2003 NC Local Government GIS Data Survey was completed making use of the Surv eyMonkey commercial Internet service. The original NCGDAP work plan proposed development of a next-generation inventory instrument that would support an ongoing inventory process for NC One Map. However, in the period between the original proposal and the completion of the final project work plan in December 2004 the National States Geographic Information Council (NSGIC) decided to take an “all states” approach to the development of a survey instrument so that each state need not develop it’s own process and tools. The NSGIC-led effort made a North Carolina-specific solution unnecessary. NSGIC, of which NCGDAP co-PI Zsolt Nagy was outgoing President at the time, developed the RAMONA (Random Access Metadata tool for Online National Assessment) inventory tool with funding from the National Oceanic and Atmospheric Administration, the Department of Homeland Security, and the Federal Emergency Management Agency NC GEOSPATIAL DATA ARCHIVING PROJECT JUNE 2008 20 (FEMA).26 RAMONA’s primary purpose is to track the status of GIS in US state and local government, aiding the planning and building of Spatial Data Infrastructures. RAMONA provides one consistent platform for the nation that is designed to work in concert with the federal Geospatial One-Stop (GOS) portal, a metadata clearinghouse operated by the USGS National Geospatial Programs Office. FEMA has direct access to the national database as a resource supporting emergency management operations. In 2006 RAMONA was implemented within the state as the inventory component of the NC OneMap Program. In support of NCGDAP data inventory requirements, NCCGIA led implementation and will be continuing with administration, reporting, and analysis operations related to the RAMONA tool’s implementation within North Carolina as the NC OneMap GIS Inventory.27 Individual data producing agencies within the state are respons ible for inputting and updating information about data holdings and data projects. Availability of the inventory information supports acquisition efforts by lowering contact and selection costs and minimizing impact on data custodians. The federal GOS portal automatically harvests the metadata building blocks generated in the NC OneMap GIS Inventory. As of November 14, 2007 a total of 206 NC agencies had registered, including 164 data producing agencies. An interim report summarizing data availability and related results was released in February 2008.28 Frequency of Capture Many vector data layers are subject to ongoing update, the frequency of which may be a reflection of the frequenc y with which the described features themselves change or a function of the operational processes of a particular agency. Cadastral or property data, for example, will tend to change on a fairly continuous basis in some agencies, while other agencies may handle updates in batch processes. On the other hand, road and municipal boundary data also change but at a lower rate. One challenge faced by NCGDAP was to determine, with stakeholders, the frequency with which specific vector data layers should be acquired for archival purposes. Such a plan would have to be both cost effective and minimize the amount of data loss between captures. Motivation for a Survey Early in the project, as outreach to local agencies was underway, it became clear that there was much to learn from individual agencies which were already creating data snapshots for their own business needs. While some anecdotal information about current practice had been acquired in the course of site vis... Read less

See 1 website result