Skip to content

VER-304: Optimize get_snippets filters and searching to fix timeout issues #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
quancao-ea merged 10 commits into from
Mar 2, 2026
Merged

VER-304: Optimize get_snippets filters and searching to fix timeout issues #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c74cc4c
Optimize get_snippets filters to fix timeout issues
rajivsinclair Jan 29, 2026
34a19c3
Optimize get_filtering_options and get_trending_topics functions
rajivsinclair Jan 29, 2026
8aa07ac
Optimize get_recording_filter_options and add missing FK indexes
rajivsinclair Jan 29, 2026
05ecffa
Remove 17MB of unused indexes and optimize embedding lookup
rajivsinclair Jan 29, 2026
44cb67a
Restore sorting indexes that are used by get_snippets ORDER BY
rajivsinclair Jan 29, 2026
c875082
Optimize state and source filters in get_snippets
rajivsinclair Jan 30, 2026
b402c2a
Add dislike_count tracking to snippet like trigger
quancao-ea Feb 27, 2026
51463ce
Optimize database by removing unused function and adding FK indexes
quancao-ea Feb 27, 2026
08a45e7
Merge get_snippets into single CTE chain and fix dislike_count
quancao-ea Feb 28, 2026
66dfd6c
Update migration files
quancao-ea Feb 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 22 additions & 35 deletions supabase/database/sql/get_filtering_options_function.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
CREATE
OR REPLACE FUNCTION get_filtering_options (
-- Optimized get_filtering_options function
-- Uses materialized view (filter_options_cache) instead of scanning 1M+ row audio_files table
-- Performance improvement: 3.7s avg -> <10ms
--
-- IMPORTANT: Call refresh_filter_options_cache() after adding new radio stations or states

CREATE OR REPLACE FUNCTION get_filtering_options (
p_language TEXT DEFAULT 'english',
p_label_page INT DEFAULT 0,
p_label_page_size INT DEFAULT 5
Expand Down Expand Up @@ -52,53 +57,35 @@ BEGIN
'items', labels
);

-- Fetch unique states from the audio_files table
WITH unique_states AS (
SELECT DISTINCT location_state
FROM public.audio_files
WHERE location_state IS NOT NULL
)
-- Fetch states from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', location_state,
'value', location_state
'label', label,
'value', value
)
) INTO states
FROM unique_states;
FROM filter_options_cache
WHERE option_type = 'states';

-- Fetch unique radio station codes from the audio_files table
WITH unique_sources AS (
SELECT DISTINCT
radio_station_code,
radio_station_name
FROM public.audio_files
WHERE radio_station_code IS NOT NULL
)
-- Fetch sources from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', CASE
WHEN radio_station_name IS NOT NULL
THEN radio_station_name || ' - ' || radio_station_code
ELSE radio_station_code
END,
'value', radio_station_code
'label', label,
'value', value
)
) INTO sources
FROM unique_sources;
FROM filter_options_cache
WHERE option_type = 'sources';

-- Fetch unique primary languages from the snippets table
WITH unique_languages AS (
SELECT DISTINCT language->>'primary_language' AS primary_language
FROM public.snippets
WHERE language IS NOT NULL
)
-- Fetch languages from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', primary_language,
'value', primary_language
'label', label,
'value', value
)
) INTO languages
FROM unique_languages;
FROM filter_options_cache
WHERE option_type = 'languages';
Comment on lines +60 to +88
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's find and read the SQL file
find . -name "get_filtering_options_function.sql" -type f | head -20

Repository: PublicDataWorks/verdad

Length of output: 124

🏁 Script executed:

# Let's also look for the file using git
git ls-files | grep -i "get_filtering_options"

Repository: PublicDataWorks/verdad

Length of output: 122

🏁 Script executed:

cat -n supabase/database/sql/get_filtering_options_function.sql

Repository: PublicDataWorks/verdad

Length of output: 3844

🏁 Script executed:

# Find where filter_options_cache is defined
git ls-files | xargs grep -l "filter_options_cache" | head -20

Repository: PublicDataWorks/verdad

Length of output: 181

🏁 Script executed:

# Search for the CREATE statement of filter_options_cache
rg "CREATE.*filter_options_cache" -A 5

Repository: PublicDataWorks/verdad

Length of output: 1477

🏁 Script executed:

cat -n supabase/migrations/20260129_optimize_filter_functions.sql | head -50

Repository: PublicDataWorks/verdad

Length of output: 1933

🏁 Script executed:

# Check if filter_options_cache is explicitly created with a schema prefix
rg "CREATE.*VIEW.*filter_options_cache" -B 2 -A 2

Repository: PublicDataWorks/verdad

Length of output: 550

🏁 Script executed:

# Check for any explicit schema prefixes in the migration file
cat supabase/migrations/20260129_optimize_filter_functions.sql | grep -E "(filter_options_cache|SET search_path)"

Repository: PublicDataWorks/verdad

Length of output: 423

🏁 Script executed:

# Check if there's a SET search_path clause in the function
cat -n supabase/database/sql/get_filtering_options_function.sql | grep -i "search_path"

Repository: PublicDataWorks/verdad

Length of output: 48

🏁 Script executed:

# Check the full migration file to see if there's a schema-qualified creation
cat supabase/migrations/20260129_optimize_filter_functions.sql | head -100

Repository: PublicDataWorks/verdad

Length of output: 3050

Schema‑qualify filter_options_cache to public.filter_options_cache in this SECURITY DEFINER function.

The function inconsistently qualifies schema names: it explicitly uses public.labels and public.audio_files elsewhere but leaves filter_options_cache unqualified on lines 67, 77, and 87. In SECURITY DEFINER functions, unqualified object names can be hijacked through search_path manipulation (including pg_temp). Qualify the view consistently to prevent that risk.

Proposed fix 
-    FROM filter_options_cache
+    FROM public.filter_options_cache
     WHERE option_type = 'states';
 
-    FROM filter_options_cache
+    FROM public.filter_options_cache
     WHERE option_type = 'sources';
 
-    FROM filter_options_cache
+    FROM public.filter_options_cache
     WHERE option_type = 'languages';
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
-- Fetch states from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', location_state,
'value', location_state
'label', label,
'value', value
)
) INTO states
FROM unique_states;
FROM filter_options_cache
WHERE option_type = 'states';
-- Fetch unique radio station codes from the audio_files table
WITH unique_sources AS (
SELECT DISTINCT
radio_station_code,
radio_station_name
FROM public.audio_files
WHERE radio_station_code IS NOT NULL
)
-- Fetch sources from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', CASE
WHEN radio_station_name IS NOT NULL
THEN radio_station_name || ' - ' || radio_station_code
ELSE radio_station_code
END,
'value', radio_station_code
'label', label,
'value', value
)
) INTO sources
FROM unique_sources;
FROM filter_options_cache
WHERE option_type = 'sources';
-- Fetch unique primary languages from the snippets table
WITH unique_languages AS (
SELECT DISTINCT language->>'primary_language' AS primary_language
FROM public.snippets
WHERE language IS NOT NULL
)
-- Fetch languages from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', primary_language,
'value', primary_language
'label', label,
'value', value
)
) INTO languages
FROM unique_languages;
FROM filter_options_cache
WHERE option_type = 'languages';
-- Fetch states from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', label,
'value', value
)
) INTO states
FROM public.filter_options_cache
WHERE option_type = 'states';
-- Fetch sources from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', label,
'value', value
)
) INTO sources
FROM public.filter_options_cache
WHERE option_type = 'sources';
-- Fetch languages from cached view (fast!)
SELECT jsonb_agg(
jsonb_build_object(
'label', label,
'value', value
)
) INTO languages
FROM public.filter_options_cache
WHERE option_type = 'languages';
🤖 Prompt for AI Agents 
In `@supabase/database/sql/get_filtering_options_function.sql` around lines 60 -
88, The three SELECTs that read from filter_options_cache (assigning INTO
states, sources, languages) must reference the view with an explicit schema to
avoid search_path hijacking in this SECURITY DEFINER function; update the three
FROM clauses that currently use filter_options_cache to use
public.filter_options_cache so these queries (the ones building jsonb_agg into
the variables states, sources, and languages) consistently match other
schema-qualified references like public.labels and public.audio_files.


RETURN jsonb_build_object(
'languages', languages,
Expand Down
Loading