Search

GIS Data Processing for Big Data

The client is a startup that collects, stores, and provides access to large-scale public and private geospatial datasets. Their users and partners use these datasets for various business purposes. Currently, they are focusing on transportation.
GIS Data Processing for Big Data

Table of Content

Subscribe to latest Insights

By clicking "Subscribe", you are agreeing to the our Terms of Use and Privacy Policy.

Project Overview

The objective was to design and implement a cloud-based system which is capable of capturing, processing, and providing access to diverse geospatial datasets from multiple sources. These datasets, while focused around transportation, were significantly different in structure and content, such as traffic density, traffic lights, car telemetry, etc.  

Scope: 

  • The solution required to handle varying schemas and formats in an efficient manner while providing real-time querying capabilities. 
  • Solution should be able to integrate with business intelligence (BI) tools for better analysis. 
  • The solutions should also be able to keep the infrastructure costs low using AWS services. 

Key Challenges

  • Varied Data Schemas Each dataset came from different sources, with no consistent schema. Therefore, it was a challenge to create a unified processing system that had the capability to handle everything from traffic density to car telemetry, while still preserving the geospatial component.
  • Scalability and Cost Constraints The client, being a startup, needed a solution that was scalable as well as cost-effective. This was because the client had limited initial resources.
  • Geospatial Complexity The data included geospatial components that required efficient modeling and querying. This made it necessary to implement specialized algorithms and tools that had the capability to handle the complexity of geospatial data.

Our Solution

Serverless Containers

Data processing was handled by a serverless AWS ECS Fargate container. The process involved the identification of data schema and metadata extraction for further analysis.

Data Lake

Given the unstructured nature of the datasets, AWS S3 was used as a data lake to store the raw data.

Big Data Analytics

The project used Amazon EMR to handle large-scale data processing. Apache Hive was used as the metastore to organize and catalog data, while PrestoDB was used for low-latency querying of the geospatial data stored in S3.

Geospatial Schema Standardization

To model real-world geospatial data across various datasets, the Well-Known Text (WKT) format was used. This common format allowed the system to correlate, join, and analyze datasets in an efficient and effective manner. 

Geospatial Data Processing Tools Python Libraries

We used Python packages, such as GDAL, PySAL, and GeoPandas along with custom algorithms in order to convert and process geospatial data into different formats such as ESRI, GeoJSON, GML, and KML.  

Key Results

fi 9727410

Efficient Data Processing

The AWS ECS Fargate containers provided a scalable solution for processing vast amounts of geospatial data with minimal cost. This led to the seamless handling of diverse datasets.

fi 6582140

Unified Data Storage

The AWS S3 data lake helped the client to efficiently store unstructured data. This led to the data lake offering high availability and durability at low cost.

CrossML

Real-Time Querying

Using PrestoDB on EMR, the client was able to perform complex queries on geospatial data with low latency. This allowed the users to analyze large datasets in real-time.

CrossML

Cost-Effective Cloud Solution

The architecture was designed in order to keep the initial costs low by using AWS services, such as ECS, S3, and EMR. These services helped to deliver a scalable and resilient solution within the client’s budget constraints.

Latest Insights

Explore In-Depth Insights
and Industry Trends

How AI Drives Digital Transformation in Retail?

Digital transformation in retail uses several AI technologies such as predictive analytics, computer vision, cloud computing, AR, VR, IoT, robotics, and headless e-commerce.

11 Ways of Using Generative AI to Improve Transportation Safety And Compliance

Generative AI improves transportation safety and compliance by predicting and preventing risks and streamlining compliance for the transportation industry.

Data Analytics in Transportation: Use Cases and Benefits

Data analytics in transportation makes organizations more efficient, safe, and cost-effective.

What Role Does Real-Time Inventory Tracking Play In Reducing Stockouts And Overstock Situations?

Real-time inventory tracking helps organizations avoid stockouts and overstocking by keeping track of future demand through predictive analytics.

Embrace AI Technology For Better Future

Integrate Your Business With the Latest Technologies

Stay updated with latest AI Insights