Merge branch 'master' into 393-foster-import

Author: c-simpson

README.md

@@ -12,61 +12,33 @@ animal care programs), administration and development efforts are coordinated by
 
 ## [The Data Pipeline](https://codeforphilly.org/projects/paws_data_pipeline)
 
-This project seeks to provide PAWS with an easy-to-use and easy-to-support tool to extract 
-data from multiple source systems, confirm accuracy and appropriateness, 
-clean/validate data where necessary (a data hygiene and wrangling step), 
-and then load relevant data into one or more repositories to facilitate 
-(1) a highly-accurate and rich 360-degree view of PAWS constituents 
-(Salesforce is a likely candidate target system; already in use at PAWS) and 
-(2) flexible ongoing data analysis and insights discovery (e.g. a data lake / data warehouse). 
-
 Through all of its operational and service activities, PAWS accumulates data regarding donations, 
 adoptions, fosters, volunteers, merchandise sales, event attendees (to name a few), 
-each in their own system and/or manual (Google Sheet) tally. This vital data that can 
+each in their own system and/or manual tally. This vital data that can 
 drive insights remains siloed and is usually difficult to extract, manipulate, and analyze. 
-Taking all of this data, making it readily available, and drawing inferences through analysis 
-can drive many benefits:   
-
-- PAWS operations can be better informed and use data-driven decisions to guide programs 
-and maximize effectiveness;  
-- Supporters can be further engaged by suggesting additional opportunities for involvement 
-based upon pattern analysis;  
-- Multi-dimensional supporters can be consistently (and accurately) acknowledged for all 
-the ways they support PAWS (i.e. a volunteer who donates and also fosters kittens), 
-not to mention opportunities to further tap the potential of these enthusiastic supporters.
-
-## [Code of Conduct](https://codeforphilly.org/pages/code_of_conduct)
-
-This is a Code for Philly project operating under their code of conduct. 
-
-## Getting started
-see [Getting Started](GettingStarted.md) to run the app locally
 
-## Project Plan
+This project provides PAWS with an easy-to-use and easy-to-support tool to extract 
+constituent data from multiple source systems, standardize extracted data, match constituents across data sources,  
+load relevant data into Salesforce, and run an automation in Salesforce to produce an RFM score. 
+Through these processes, the PAWS data pipeline has laid the groundwork for facilitating an up-to-date 360-degree view of PAWS constituents, and 
+flexible ongoing data analysis and insights discovery.
 
-### Phase 1 (now - Jan 15 2020) 
+## Uses 
 
-**Goal**: Create a central storage of data where 
+- The pipeline can inform the PAWS development team of new constiuents through volunteer or foster engagegement
+- Instead of manually matching constituents from volunteering, donations and foster/adoptions, PAWS staff only need to upload the volunteer dataset into the pipeline, and the pipeline handles the matching
+- Volunteer and Foster data are automatically loaded into the constituent's SalesForce profile
+- An RFM score is calculated for each constituent using the most recent data 
+- Data analyses can use the output of the PDP matching logic to join datasets from different sources; PAWS can benefit from such analyses in the following ways: 
+    - PAWS operations can be better informed and use data-driven decisions to guide programs and maximize effectiveness;  
+    - Supporters can be further engaged by suggesting additional opportunities for involvement based upon pattern analysis;  
+    - Multi-dimensional supporters can be consistently (and accurately) acknowledged for all the ways they support PAWS (i.e. a volunteer who donates and also fosters kittens), not to mention opportunities to further tap the potential of these enthusiastic supporters.
 
-1. Datasets from top 3 relevant sources can be uploaded as csvs to a central system: a) Donors, b) Volunteers, 
-c) Adopters
-2. All datasets in the central system can be linked to each other on an ongoing basis
-3. Notifications can be sent out to relevant parties when inconsistencies need to be handled by a human
-4. Comprehensive report on a person’s interactions with PAWS can be pulled via a simple UI (must include full known history)
-
-### Phase 2 (Jan 15 - May 15 2020)
-
-**Goal**: Expand above features to include all relevant datasets and further automate data uploads
-Datasets from all other relevant sources can be uploaded as csvs to a central system ( a) Adoption and Foster applicants, 
-b) Foster Parents, c) Attendees, d) Clinic Clients e) Champions, f) Friends)
-Where APIs exist, create automated calls to those APIs to pull data
-
-### Phase 3 (May 15 - Sept 15 2020)
+## [Code of Conduct](https://codeforphilly.org/pages/code_of_conduct)
 
-**Goal**: Create more customizable analytics reports and features (eg noshow rates in clinicHQ)
+This is a Code for Philly project operating under their code of conduct. 
 
 ## Links
 
-[Slack Channel](https://codeforphilly.org/chat?channel=paws_data_pipeline)
-
-[Google Drive](https://drive.google.com/open?id=1O8oPWLT5oDL8q_Tm4a0Gt8XCYYxEIcjiPJYHm33lXII) 
+[Slack Channel](https://codeforphilly.org/chat?channel=paws_data_pipeline)  
+[Wiki](https://github.com/CodeForPhilly/paws-data-pipeline/wiki)

src/client/src/pages/Admin.js

@@ -243,7 +243,7 @@ export default function Admin(props) {
                                         <Typography variant="h5" styles={{paddingBottom: 5}}>Run New Analysis</Typography>
                                         <form onSubmit={handleExecute}>
                                             <Button type="submit" variant="contained" color="primary"
-                                                    disabled={statistics === 'Running' || isNewFileExist === false}>
+                                                    disabled={statistics === 'Running'}>
                                                 Run Data Analysis
                                             </Button>
                                         </form>

src/helm-chart/templates/ingress.yaml

@@ -1,11 +1,7 @@
 {{- if .Values.ingress.enabled -}}
 {{- $fullName := include "helm-chart.fullname" . -}}
 {{- $svcPort := .Values.service.port -}}
-{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
-apiVersion: networking.k8s.io/v1beta1
-{{- else -}}
-apiVersion: extensions/v1beta1
-{{- end }}
+apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: {{ $fullName }}
@@ -33,9 +29,12 @@ spec:
         paths:
           {{- range .paths }}
           - path: {{ . }}
+            pathType: Prefix
             backend:
-              serviceName: {{ $fullName }}
-              servicePort: {{ $svcPort }}
+              service:
+                name: {{ $fullName }}
+                port:
+                  number: {{ $svcPort }}
           {{- end }}
     {{- end }}
   {{- end }}

src/server/alembic/versions/45a668fa6325_postgres_matching.py

@@ -0,0 +1,106 @@
+"""postgres matching
+
+Revision ID: 45a668fa6325
+Revises: fc7325372396
+Create Date: 2022-02-10 16:19:13.283250
+
+"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision = '45a668fa6325'
+down_revision = 'fc7325372396'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('manual_matches',
+    sa.Column('source_type_1', sa.String(), nullable=False),
+    sa.Column('source_id_1', sa.String(), nullable=False),
+    sa.Column('source_type_2', sa.String(), nullable=False),
+    sa.Column('source_id_2', sa.String(), nullable=False),
+    sa.PrimaryKeyConstraint('source_type_1', 'source_id_1', 'source_type_2', 'source_id_2')
+    )
+    op.create_table('salesforcecontacts',
+    sa.Column('_id', sa.Integer(), nullable=False),
+    sa.Column('contact_id', sa.String(), nullable=True),
+    sa.Column('first_name', sa.String(), nullable=True),
+    sa.Column('last_name', sa.String(), nullable=True),
+    sa.Column('account_name', sa.String(), nullable=True),
+    sa.Column('mailing_country', sa.String(), nullable=True),
+    sa.Column('mailing_street', sa.String(), nullable=True),
+    sa.Column('mailing_city', sa.String(), nullable=True),
+    sa.Column('mailing_state_province', sa.String(), nullable=True),
+    sa.Column('mailing_zip_postal_code', sa.String(), nullable=True),
+    sa.Column('phone', sa.String(), nullable=True),
+    sa.Column('mobile', sa.String(), nullable=True),
+    sa.Column('email', sa.String(), nullable=True),
+    sa.Column('json', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
+    sa.Column('created_date', sa.DateTime(), nullable=True),
+    sa.PrimaryKeyConstraint('_id')
+    )
+    op.create_table('shelterluvpeople',
+    sa.Column('_id', sa.Integer(), nullable=False),
+    sa.Column('firstname', sa.String(), nullable=True),
+    sa.Column('lastname', sa.String(), nullable=True),
+    sa.Column('id', sa.String(), nullable=True),
+    sa.Column('internal_id', sa.String(), nullable=True),
+    sa.Column('associated', sa.String(), nullable=True),
+    sa.Column('street', sa.String(), nullable=True),
+    sa.Column('apartment', sa.String(), nullable=True),
+    sa.Column('city', sa.String(), nullable=True),
+    sa.Column('state', sa.String(), nullable=True),
+    sa.Column('zip', sa.String(), nullable=True),
+    sa.Column('email', sa.String(), nullable=True),
+    sa.Column('phone', sa.String(), nullable=True),
+    sa.Column('animal_ids', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
+    sa.Column('json', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
+    sa.Column('created_date', sa.DateTime(), nullable=True),
+    sa.PrimaryKeyConstraint('_id')
+    )
+    op.create_table('volgistics',
+    sa.Column('_id', sa.Integer(), nullable=False),
+    sa.Column('number', sa.String(), nullable=True),
+    sa.Column('last_name', sa.String(), nullable=True),
+    sa.Column('first_name', sa.String(), nullable=True),
+    sa.Column('middle_name', sa.String(), nullable=True),
+    sa.Column('complete_address', sa.String(), nullable=True),
+    sa.Column('street_1', sa.String(), nullable=True),
+    sa.Column('street_2', sa.String(), nullable=True),
+    sa.Column('street_3', sa.String(), nullable=True),
+    sa.Column('city', sa.String(), nullable=True),
+    sa.Column('state', sa.String(), nullable=True),
+    sa.Column('zip', sa.String(), nullable=True),
+    sa.Column('all_phone_numbers', sa.String(), nullable=True),
+    sa.Column('home', sa.String(), nullable=True),
+    sa.Column('work', sa.String(), nullable=True),
+    sa.Column('cell', sa.String(), nullable=True),
+    sa.Column('email', sa.String(), nullable=True),
+    sa.Column('json', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
+    sa.Column('created_date', sa.DateTime(), nullable=True),
+    sa.PrimaryKeyConstraint('_id')
+    )
+    op.create_index('idx_pdp_contacts_source_type_and_id', 'pdp_contacts', ['source_type', 'source_id'], unique=False)
+    op.create_index(op.f('ix_pdp_contacts_mobile'), 'pdp_contacts', ['mobile'], unique=False)
+    op.create_index(op.f('idx_pdp_contacts_lower_first_name'), 'pdp_contacts', [sa.text('lower(first_name)')], unique=False)
+    op.create_index(op.f('idx_pdp_contacts_lower_last_name'), 'pdp_contacts', [sa.text('lower(last_name)')], unique=False)
+    op.create_index(op.f('idx_pdp_contacts_lower_email'), 'pdp_contacts', [sa.text('lower(email)')], unique=False)
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_index(op.f('ix_pdp_contacts_lower_email'), table_name='pdp_contacts')
+    op.drop_index(op.f('ix_pdp_contacts_lower_last_name'), table_name='pdp_contacts')
+    op.drop_index(op.f('ix_pdp_contacts_lower_first_name'), table_name='pdp_contacts')
+    op.drop_index(op.f('ix_pdp_contacts_mobile'), table_name='pdp_contacts')
+    op.drop_index('idx_pdp_contacts_source_type_and_id', table_name='pdp_contacts')
+    op.drop_table('volgistics')
+    op.drop_table('shelterluvpeople')
+    op.drop_table('salesforcecontacts')
+    op.drop_table('manual_matches')
+    # ### end Alembic commands ###

src/server/api/API_ingest/ingest_sources_from_api.py

@@ -1,7 +1,7 @@
 from api.API_ingest import shelterluv_api_handler
 
-def start():
+def start(conn):
     print("Start Fetching raw data from different API sources")
     #Run each source to store the output in dropbox and in the container as a CSV
-    shelterluv_api_handler.store_shelterluv_people_all()
+    shelterluv_api_handler.store_shelterluv_people_all(conn)
     print("Finish Fetching raw data from different API sources")

src/server/api/API_ingest/shelterluv_api_handler.py

@@ -1,10 +1,12 @@
-import os
-import requests
 import csv
+import os
 import time
 
-from constants import RAW_DATA_PATH
+import requests
+import pandas as pd
 from api.API_ingest.dropbox_handler import upload_file_to_dropbox
+from constants import RAW_DATA_PATH
+from models import ShelterluvPeople
 
 try:
     from secrets_dict import SHELTERLUV_SECRET_TOKEN
@@ -60,7 +62,7 @@ def write_csv(json_data):
 
 ''' Iterate over all shelterlove people and store in json file in the raw data folder
 We fetch 100 items in each request, since that is the limit based on our research '''
-def store_shelterluv_people_all():
+def store_shelterluv_people_all(conn):
     offset = 0
     LIMIT = 100
     has_more = True
@@ -90,8 +92,9 @@ def store_shelterluv_people_all():
     file_path = write_csv(shelterluv_people)
     print("Finish storing latest shelterluvpeople results to container")
 
-
     print("Start storing " + '/shelterluv/' + "results to dropbox")
     upload_file_to_dropbox(file_path, '/shelterluv/' + file_path.split('/')[-1])
     print("Finish storing " + '/shelterluv/' + "results to dropbox")
 
+    print("Uploading shelterluvpeople csv to database")
+    ShelterluvPeople.insert_from_df(pd.read_csv(file_path, dtype="string"), conn)