Investigate why so many rows have no entries in the change time calculation

[JEElsner]

This is a continuation of Issue #88 addressed by PR #90.

Counting the number of row with no start times, about half of the data have no start times:

def count_start_times(row):
    table = stimulus_presentations
    change_frame = row['change_frame']
    
    if np.isnan(change_frame):
        return np.nan
    
    start_times_len = len(table[table.start_frame == change_frame]['start_time'].values)
    return start_times_len

n_start_times = trials.apply(count_start_times, axis=1)
print(f'number of zero entries: {np.count_nonzero(n_start_times == 0)}')
print(f'total entries: {len(n_start_times)}')

number of zero entries: 254
total entries: 567

trials['start_time_count'] = n_start_times
trials[trials.start_time_count == 0].head()

	start_time	stop_time	initial_image_name	change_image_name	is_change	change_time_no_display_delay	go	catch	lick_times	response_time	...	hit	false_alarm	miss	correct_reject	aborted	auto_rewarded	change_frame	trial_length	change_time_with_display_delay	start_time_count
trials_id
0	28.08763	29.05453	im036_r	im036_r	False	NaN	False	False	[28.55387, 28.73684, 29.30404]	NaN	...	False	False	False	False	True	False	-99	0.96690	NaN	0
2	37.09446	40.78107	im078_r	im078_r	False	NaN	False	False	[40.48052]	NaN	...	False	False	False	False	True	False	-99	3.68661	NaN	0
4	50.60569	51.75679	im111_r	im111_r	False	NaN	False	False	[51.43985]	NaN	...	False	False	False	False	True	False	-99	1.15110	NaN	0
5	52.10729	53.62505	im111_r	im111_r	False	NaN	False	False	[53.19128, 53.3242]	NaN	...	False	False	False	False	True	False	-99	1.51776	NaN	0
7	62.61569	67.41981	im047_r	im047_r	False	NaN	False	False	[67.12002]	NaN	...	False	False	False	False	True	False	-99	4.80412	NaN	0

5 rows × 22 columns

I also have a Jupyter notebook and pkl file of the problematic rows that I can share.

Edit: change start_time to start_frame

[corbennett]

In the above function, you still have a bug. Where you have:
table[table.start_time == change_frame]
you should have:
table[table.start_frame == change_frame]

I'd be interested to see the notebook you have working through this. Thanks!

[JEElsner]

Aha good catch! It looks like I updated it in my notebook, but not the GitHub post, so the data beneath should be correct.

I'll get that notebook over to you.

[corbennett]

Ok. I see the issue. The latest SDK release treats the 'change_frame' column a bit differently than past releases. Now, for aborted trials that don't have a change frame, the value is -99 (it used to be nan). So the new function (which behaves the same as the one you wrote, but is a little cleaner) should be:

def get_change_time_from_stim_table(row):
    '''
    Given a particular row in the trials table,
    find the corresponding change time in the
    stimulus presentations table
    '''
    table = stimulus_presentations
    change_frame = row['change_frame']
    if change_frame==-99: #indicates aborted trial
        return np.nan

    change_times = table[table.start_frame==change_frame]['start_time'].values

    return change_times[0]