IDP Insight is designed to automate the collection, processing, and visualization of data on Internally Displaced Persons (IDPs). It serves as a tool for the Early Warning Preparedness Unit (EWPU) team from the World Food Program (WFP), providing a user-friendly dashboard with reliable, up-to-date, and detailed information about IDPs.
There's a need for an automated and reliable system to collect, process, and visualize accurate data on IDPs. Traditional methods involve manual work of accessing and aggregating data from various sources, which is time-consuming and prone to errors.
IDP Insight automates the entire process, from data collection, to data processing, to visualization. It gathers data from different sources, cleans and processes it, and presents it in charts. This eliminates the need for manual data aggregation and ensures the data is current and reliable.
- Automated Data Collection: Fetches data from various reliable sources, such as International Organization for Migration (IOM), UNHCR and IDMC and stores it for processing.
- Data Processing: Transforms raw data into clean, organized formats ready for the analysis.
- Visualization: Displays processed data and provides the possibility to create charts.
- Continuous Updates: Regularly updates data to ensure the latest information is always available.
- fetch-data: Python script deployed to AWS Lambda (
devgurus-fetch-data). It serves to collect raw data from different sources hosted on HDX and store it indevgurus-raw-data. Please refer to serverless.yml for the deployment of this lambda function. - idp-frontend: The front-end application, built with React, deployed to
devgurus-frontend-deployment-bucket. - process-data: Python script in AWS Lambda (
devgurus-process-data) triggered by uploads todevgurus-raw-data. Processes raw data into clean CSV files and stores them indevgurus-processed-data. Please refer to serverless.yml for the deployment of this lambda function. - serverless-api: Python-based serverless API deployed to
devgurus-serverless-api. Acts as an interface between data buckets and the frontend, providing access to datasets, raw, and processed files. Please refer to serverless.yml for the deployment of this lambda function. - .github/workflows: Contains GitHub Actions workflows for deploying to AWS.
Each folder in this repository contains its own README for more specific details. Please refer to these for in-depth information about each component of the application.
Please refer to artifacts for the artifacts for this project.