Skip to content

[f] VER-261 - Optimize get snippets with labels #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
quancao-ea merged 3 commits into from
Sep 23, 2025
Merged

[f] VER-261 - Optimize get snippets with labels #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9a304d9
Fetch necessary labels data in snippets
quancao-ea Sep 23, 2025
e9a4122
Create indexes for efficient snippet fetching
quancao-ea Sep 23, 2025
27b403a
Remove unused file
quancao-ea Sep 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 8 additions & 17 deletions supabase/database/sql/get_snippet_labels_function.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,28 +18,19 @@ BEGIN
WHEN p_language = 'spanish' THEN l.text_spanish
ELSE l.text
END,
'created_by', l.created_by,
'is_ai_suggested', l.is_ai_suggested,
'applied_by', sl.applied_by,
'applied_at', sl.created_at,
'upvoted_by', COALESCE(upvote_users, '[]'::jsonb)
'upvote_count', COALESCE(upvote_counts.count, 0),
'upvoted_by_me', COALESCE(upvote_counts.upvoted_by_current_user, false)
)), '[]'::jsonb)
) INTO result
FROM public.snippet_labels sl
JOIN public.labels l ON sl.label = l.id
LEFT JOIN (
SELECT lu.snippet_label, jsonb_agg(jsonb_build_object(
'id', u.id,
'email', u.email,
'upvoted_at', lu.created_at
)) AS upvote_users
LEFT JOIN LATERAL (
SELECT
COUNT(*) AS count,
BOOL_OR(lu.upvoted_by = current_user_id) AS upvoted_by_current_user
FROM public.label_upvotes lu
JOIN auth.users u ON lu.upvoted_by = u.id -- Join to get user email
WHERE lu.snippet_label IN (
SELECT id FROM public.snippet_labels WHERE snippet = snippet_id
) -- Filter to only include upvotes for the specific snippet
GROUP BY lu.snippet_label
) lu ON sl.id = lu.snippet_label
WHERE lu.snippet_label = sl.id
) upvote_counts ON TRUE
WHERE sl.snippet = snippet_id;

RETURN COALESCE(result, jsonb_build_object('snippet_id', snippet_id, 'labels', '[]'::jsonb));
Expand Down
53 changes: 36 additions & 17 deletions supabase/database/sql/get_snippets_function.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,33 @@ BEGIN

user_is_admin := COALESCE('admin' = ANY(user_roles), FALSE);

CREATE TEMP TABLE filtered_snippets AS
CREATE TEMP TABLE filtered_snippets AS (
-- Pre-compute all label data with upvote counts
WITH label_data AS (
SELECT
sl.snippet,
COALESCE(jsonb_agg(
jsonb_build_object(
'id', l.id,
'text', CASE
WHEN p_language = 'spanish' THEN l.text_spanish
ELSE l.text
END,
'upvote_count', COALESCE(upvote_counts.count, 0),
'upvoted_by_me', COALESCE(upvote_counts.upvoted_by_current_user, false)
)
), '[]'::jsonb) AS labels
FROM public.snippet_labels sl
JOIN public.labels l ON sl.label = l.id
LEFT JOIN LATERAL (
SELECT
COUNT(*) AS count,
BOOL_OR(lu.upvoted_by = current_user_id) AS upvoted_by_current_user
FROM public.label_upvotes lu
WHERE lu.snippet_label = sl.id
) upvote_counts ON TRUE
GROUP BY sl.snippet
)
Comment on lines +34 to +58
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The label_data CTE calculates label information for all snippets by scanning the entire public.snippet_labels table. This could be a performance regression compared to the previous implementation, which invoked the get_snippet_labels function only for the snippets that passed the filters. If the snippet_labels table is large and the filters are selective, this change could slow down the query.

Consider restructuring the query to filter snippets before aggregating label data. One approach is to use a preliminary CTE to select the IDs of filtered snippets and then use those IDs to constrain the data processed in the label_data CTE.

SELECT
s.id,
s.recorded_at,
Expand All @@ -55,24 +81,22 @@ BEGIN
s.confidence_scores,
s.language,
s.context,
(get_snippet_labels(s.id, p_language) -> 'labels') AS labels,
COALESCE(ld.labels, '[]'::jsonb) AS labels,
jsonb_build_object(
'id', a.id,
'radio_station_name', a.radio_station_name,
'radio_station_code', a.radio_station_code,
'location_state', a.location_state,
'location_city', a.location_city
) AS audio_file,
CASE
WHEN us.id IS NOT NULL THEN true
ELSE false
END AS starred_by_user,
us.id IS NOT NULL AS starred_by_user,
ul.value AS user_like_status,
uhs.snippet IS NOT NULL AS hidden,
like_counts.likes AS like_count,
like_counts.dislikes AS dislike_count,
uhs.snippet IS NOT NULL AS hidden
like_counts.dislikes AS dislike_count
FROM snippets s
LEFT JOIN audio_files a ON s.audio_file = a.id
LEFT JOIN label_data ld ON ld.snippet = s.id
LEFT JOIN user_star_snippets us ON us.snippet = s.id AND us."user" = current_user_id
LEFT JOIN user_like_snippets ul ON ul.snippet = s.id AND ul."user" = current_user_id
LEFT JOIN user_hide_snippets uhs ON uhs.snippet = s.id
Expand All @@ -88,11 +112,7 @@ BEGIN
-- If user is admin, show all snippets (including hidden ones)
-- If user is not admin, only show non-hidden snippets
user_is_admin OR
NOT EXISTS (
SELECT 1
FROM user_hide_snippets uhs
WHERE uhs.snippet = s.id
)
uhs.snippet IS NULL
)
AND (
p_filter IS NULL OR
Expand Down Expand Up @@ -273,14 +293,13 @@ BEGIN
WHEN p_order_by = 'upvotes' THEN s.upvote_count + s.like_count
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This ORDER BY clause will fail because s.upvote_count and s.like_count are not available as columns in this context.

  • like_count is available as like_counts.likes from the like_counts lateral join.
  • upvote_count is now calculated per-label inside the labels JSON array and is not available as a top-level column on the snippet for sorting.

You will need to either calculate the total upvotes for a snippet to sort by it (e.g., by processing the labels JSON) or remove this sorting option if it's no longer supported.

WHEN p_order_by = 'comments' THEN s.comment_count
WHEN p_order_by = 'activities' THEN
CASE
CASE
WHEN s.user_last_activity IS NULL THEN 0
ELSE EXTRACT(EPOCH FROM s.user_last_activity)
END
WHEN p_order_by IS NULL OR p_order_by = 'latest' OR p_order_by = '' THEN EXTRACT(EPOCH FROM s.recorded_at)
ELSE EXTRACT(EPOCH FROM s.recorded_at)
END DESC,
s.recorded_at DESC;
s.recorded_at DESC -- Default for all other cases, including p_order_by = 'latest'
);

SELECT COUNT(*) INTO total_count
FROM filtered_snippets;
Expand Down
10 changes: 10 additions & 0 deletions supabase/database/sql/optimize_label_indexes.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
-- Indexes to optimize the CTE-based label aggregation query

-- 1. Additional index for label_upvotes to optimize the LATERAL join
CREATE INDEX IF NOT EXISTS idx_label_upvotes_snippet_label_upvoted_by
ON public.label_upvotes USING btree (snippet_label, upvoted_by);

-- 2. Index for main snippets filtering
CREATE INDEX IF NOT EXISTS idx_snippets_status_confidence
ON public.snippets USING btree (status, (((confidence_scores->>'overall'))::INTEGER))
WHERE (status = 'Processed'::processing_status);
84 changes: 0 additions & 84 deletions supabase/database/sql/search_related_public_snippets.sql
View file
Open in desktop

This file was deleted.