CoP: Data Science: SEIE Survey Analysis

[ExperimentsInHonesty]

Overview

We need to report on our progress for our partner at the Department of Neighborhood Empowerment

Action Items

Phase 1--Completed by Sathwik:

• NLP engineering/analysis on a large survey featuring free text columns.
• Run TF-IDF analysis
• create word clouds
• Create a presentation to show data to DONE and neighborhood council members. Tooling could include scikit-learn, spacy, pandas. First version here

Phase 2--Picked up by Henry (8.2.2021):

• Create filtering system on google sheets (labels)
• Revisit preprocess script and apply to dataset (ex. remove stopwords, puncuation, duplicates, etc)
• Frequency counts on most common unigrams, bigrams and trigrams for One Question
  - [x] On the entire dataset
  - [x] By Region
• Create clear view of responses containing key phrases
• Number of council members who mention the "top" topics
• Present to Julien

Phase 3

• Expand to the rest of the questionnaire
• Create dashboard with results of quantitative analysis, showing comparisons of themes topic by region, and comparisons of regions grouped by topic. (Dashboard link)
• Develop presentation/write-up for final delivery to Julien

Resources/Instructions

• HFLA Final Writeup
• Supporting Datapack for Final Writeup
• Final Slide Deck
• Spreadsheet provide by DONE
• SEIE Survey Summary (So. LA) provided by DONE

Currently Underway:
@henrykaplan

[ExperimentsInHonesty]

@RinkalAgarwal can self assign when she has accepted the github invite.

[ExperimentsInHonesty]

@sathwikkes Please provide process update

1. Progress
2. Blockers
3. Availability
4. ETA for total completion (if possible), otherwise indicate when you will be delivering the next progress report .

[sathwikkes]

1. Progress: loaded survey data and performed basic exploratory data analysis such as bar plots and word clouds, identified word pairings using CountVectorizer
2. Blockers: need to preprocess and clean text to generate better results with less noise, lowercase all text to stay consistent
3. Future: standardizing text data, removing irrelevant information, produce keyword/name recognition

screenshots of sample work are provided below

[ExperimentsInHonesty]

@RinkalAgarwal @sathwikkes
Please provide update

1. Progress
2. Blockers
3. Availability
4. ETA

[sathwikkes]

1. Progress: standardized/normalized text, performed tf-idf vectorizer on a sample column, removed irrelevant content, touched on entity recognition, added more visualization
2. Blockers/Future: updating word clouds with acquired weights, performing sentiment analysis, documenting results
3. ETA/Availability: next data science meeting

[sathwikkes]

1. Progress: created, finished, and workshopped the presentation
2. Blockers/Future: waiting on client availability
3. ETA/Availability: next data science meeting

[snooravi]

Presented updated to Julien here based on progress made in Collab notebook

[snooravi]

Affinity Map here