Conversation
Added StimParam file and builder, as well as Log Processor and my version of buildAnalysisParam for MonkeyLogic specific tasks that I ran during August 2018.
Added Ephy Parameter to "buildAnalysisParam" checking for Offline sorted. Added code to read .mat export from Plexon offline sorted (Done a certain way) to preprocessSpikes. Path handling was somewhat sloppy but it seems to work for now.
aboharbf
changed the title
Changed structure being recieved (from .mat to .xls) to remove possible overwriting of MATLAB files from MonkeyLogic, as well as simplifying the extracting.
- preprocessStrobe has been updated with appropriate figure titles - averageEvoked function now exists - averages LFP per trial. Needs Error bars integratted. - Draft of processRunBatch by Serene included. - previous line in runAnalyses forced colors to be letters - appropriate check implemented.
Changed colors and chColor variables in runAnalyses to cell arrays, and changed relevant code to properly read.
processRunBatch script which allows for batch running of data, as well as a buildRunList function which builds the required cell array based on a directory being selected. - Also made changes to processRun which are agnostic to variable number.
- Changed processRun Argument managing. - fixed bug in preprocessStrobe which skipped level parameter retrieval from input params in "hardcodeAnd*" inputs.
- Added compatibility for monkeylogic's native output format (.bhv2) to logprocessor - created behavioralsummary function without as many dependencies on ML code, capable of working with few additions -created directory for monkeylogic dependences in dependency folder. - integrated AnalysisParam switch for Photodiode use.
- Updated MonkeyLogic Summary page which is created when log file is processed to include many variables relating to the recording day, either extracted from a directory above the data (.xls) or from the MonkeyLogic file itself if present (all of my experiments after Sept '18. - Copied Fix Serene did for preprocessSpikes which will now not forcibly assign a unit to "unsorted" in the event of no unsorted units. - Fixed processRunBatch to work with a recent change in processRun argument handling. Tested and works.
- Added a Try Catch loop to processRun incase it crashes deep into an analysis, it doesn't abort the entire run (Had issues with code near very end of runAnalysis). - Changed a few fileseps to '/' to be congruent with paths as represented in phyzzy hardcoded strings.
- Added new StimParam file builder corresponding to changes in the way I label videos now. - Added Defense against Offline Spike sorting files not being available, began work on using Wave_clus in preprocessSpikes (commented out) - Process Run now has an Assert checking that SOME event is observed. - Added switch for Analysis Rig PC w/ appropriate paths, and changed my file extension of choice is bhv2 for taskFilenames.
- Merged Changes from Master - Did not merge processRunBatch.
- moved my buildAnalysisParamSocialVids to the right folder. - Finished a buildStimParamFileSocialVids which automatically creates the right structure for social videos and their decompositions, as currently labeled with the appropriate code. - Added a new 'monkeyLogic' feature to eyeCalParam.methods list, accounting for monkeyLogic's Log file output. Changed the checkAnalysisParamFile to not change this new method, as well as the appropriate additional code in preprocessEyeSignals. - Added "monkeyLogicShift" feature in preprocessEyeSignals to fix overwrite eye signals with those recorded in log file in circumstances where blackrock file is incorrect. - Changed some of the preprocesseyeSignal strcmps into a switch cases - preprocessLogFileMonkeyLogic: changed the structure to segregate processing of the log file compared to the blackrock eventmarkers. Also found the correct data structure in the log file for juice times, and assigned them correctly, as well as adding the linear regression to convert timestamps. per trial eye data is now passed out in the taskData struct, which can then be referenced by preprocesseyeSignals in the event of issues w/ blackrock's eye data.
This reverts commit 1647449.
- Line endings seem to have changed, hopefully this won't mess anything up.
- Deleted files in the main directory which were producing problems. - commented out a figure from the linear model alignment. - swapped in an elseif loop where appropriate.
- removed monkeyLogic shift feature in preprocessEyeSignal due to different clock speeds between machines making lining them up not straight forward. May try again in the future.
- Added alternative processor lines to alignAnalogIn - created "alignAnalogInMonkeyLogic (Doesn't work yet) - Added processor field to buildAnalysisParamFileSocialVids, pointing to this new function.
- Removed alignInMonkeyLogic - preprocessLogFileMonkeyLogic now passes out NEV Start times - Restored monkeyLogic Shift code, using NEV start times to construct artificial eye array, replacing both X and Y to avoid problems of alignment. - Still working on imageEyeMap in runAnalyses.
- EyeMap produces many traces and the images at the same time, unfortunately still haven't figured out the proper way to get the animated lines to remain on top of a playing video (new frames are added which cover the animated lines, still figuring out how to manipulate figure children in a way to produce correct display). - Integrated Waveclus into Preprocess spikes - added saveFig to Preprocess spikes figures - Changed preprocessSpikes to use cell arrays for colors, added color option in buildAnalysisParamFile.
- changed filename parsing in parse_data_NSX to be compatible with phyzzy style, OS incongruent slash pattern. - changed "Do_clustering" to not close images, to allow them to stay open for saving. - Added latest version of Blackrock openNSx, as required by wave_clus, in its folder, which should restrict its use to these cases (appropriate directory managment done within wave_clus block).
- buildAnalysisParamFileSocialVids updated to accept arguments - processRun input parsing now using switch case and throwing error instead of warning if argName is unrecognized. - processRun now saves all the necessary outputs to use "preprocessed" arguments. in runAnalyses - Added '-append' to all save calls after the first. - fixed bug around line 110. - put short trialDatabase script into its own function, defined at the end of the script.
Put calculations performed with spikeTimes and PSTH calculations in function form at the end of the script.
- batchAnalysis meanPSTH rewritten a bit at the end, added a few plots. Will be further extending this to generating N * 3 matrix of PSTHes for all stimuli, instead of first clustering into 'broad catagories'. This will be done at the end. - meanPSTH also now generates tmp structure after clustering to make plot code quicker to troubleshoot.
…on, color function addition - runBatchAnalysis functions, specifically meanPSTH and frameFiringRates, were heavily modified. Specifically by changing intermediate data structures from heavily nested cell arrays to 3D Arrays, which allows for better retrieving of data when you know each dimension ahead of time. - frameFiringRates previously somehow generated figures indicating labels existed for animation when they did not (it is all 'bkg'). This has mostly been resolved, though objects have a frame or two labeled w/ 'body' which don't exist. - frameFiringRates also had 3 sequential loops through the data, once for retrieval, a second time for restructing, a third for plotting. the 1st and 2nd have been combined and data no longer needs to be restructued, it is now saved correctly the first time. - Bug I've not ID'ed the cause of: sometimes saveSpikeDataBank saves a very small file without the proper data. Included assert to help prevent but still working on. - new function plotIndex takes in stimuli, 'broad labels', and stimParamFile and generates a stim*label matrix which can be used to index into data structures which stack things by stimuli. - distinguishable_colors function from MATLAB File exchange added into dependencies, mostly used in meanPSTH to generate PSTHes as traces instead of heatmap images.
…t function - Updated many of the structures in slidingWindowANOVA, introduced parfor loop in ANOVA across bins segment, and introduced temporary intermediate structures stored in analyses folders. Basic figures generated. Accidentally overwrote intermediate structure, will finish this function once structure is regenerated overnight. - next step is making trueCellCount outputs correctly structured (moving from many independently labeled structures to a single, indexed structure.) Adding recordingDepth and grid hole fields which will match run index, allowing for comparison of activity across these fields.
- trueCellCount now returns more useful structures, cell counts, depths, and grid holes for each recording. This is used later in runBatchAnalysis. - runBatchAnalysis: meanPSTH function takes advantage of a animation label to exclude or include animation data. - slidingWindowANOVA now uses parfor (Though it isn't clearly faster for it). Baseline normalization with Z score also returned. - all PSTHes now actually use the proper function (instead of some using imagesc as before) for consistency of color map. - stimParamFile now generates labels for head turning, allowing plotIndex to generate head turning index without need for hard coded indicies. - saveFigure now uses fullfile instead of strcat, as this is more robust to variations in strings being fed in. - plotPSTH now returns colorBar handle as an optional output argument, makes relabeling more straight forward.
- replaced in loop determination of sortOrder based on firing rates with single function 'genStats' which runs before plot generating loops. Also pulled 'sigStruct' generating code from the stimPSTHOverlay function and put into this function. Will allow for different statistical tests in a single place - changing sigStruct structure from having various named fields for Unsorted, Unit, and MUA activity to having a single 3D Cell array, with the indicies matching the fields in the sigStruct.IndInfo field. - processRunBatch currently non-functional (the excel part) in light of above changes and needs to be updated, but doesn't quite seem like the best use of time atm so I will let that sit.
…, genStats performs ANOVA. - PSTH directories are now generated with more descriptive names (fixAligned, normalized, onlyAnims, etc). - Z-scoring is now params.normalize = 1, instead of 2. Likely what I'll be using. - saved .fig files have numbers in their file names for easier scanning. numbers aren't in the title of the figure though. - exportFig is now a parameter attached to params, which has been set to 0 (will be useful only once you know you've generated the figures you want, otherwise a waste of processing time). - stimCatTwoWayANOVA and stimCatIndepANOVA have been deleted, with the code for the former placed in genStats. Will be adding a mixed effects model into this function shortly.
- runAnalyses - in the genStats function, comparisons between 2 groups now use the ttest2 function instead of anovan. This reduces overhead, since the result is the same. the structures it outputs have likewise been changed, organized to be more efficient. stimCatTwoWayANOVA function removed, inputs to stimPSTHoverlay cleaned up, due to removing certain features. - colorCoded raster code cleaned up a bit, using linkProp for the axes instead of a callback modifying position. - added 'struct' option to initNestedCellArray. - output of trueCellCount modified, cleaned up, made more robust with respect to indexing. Lots of it commented out due to not being used/data being stored differently. - runBatchAnalysis has lots of switches added, in particular to change plots to not have counts attached to labels, and means removed from some plots where it is somewhat distracting.
buildRunList - added a time out to uiwait, to prevent inappropriate closing from causing MATLAB to hang. also added a catch for when nothing is selected. plotIndex - added a brief check for additional fields, to also resort. trueCellCount - ANOVA stuff is handled a few lines earlier (removing repeated recording sites with trueCellInd). Removed a great deal of old code. waveClus - added a brief section to catch an annoying storage issue with the threshold. preprocessMonkeyLogic - brief defense against NaNs added into photodiode calculations, and at the end for the generation of a fake fixationTime vector. Also added a few incorrectly named animations to the renaming function at the end. preprocessSpikes - added (end) to threshold, incase thr is stored off. processBatchAnalysis - spikeDataBank now has stimStatsTable. processRunBatch - now organized better, clear when the actually large parallel processing loop will run vs simply collecting info for excel sheet. Also no longer skips rows which have errors. runAnalyses - fixed issue with multi-channel recording code overwritting a variable (imageSortedRates) which would be used (and caused errors) on Ch2. genStats now calculates and appends 'cohensD'. probably not going to use that. runBatchAnalysis - a great deal of commenting and labeling, as well as a filter for mean firing rate across a PSTH to be included. stimParamFile - Added animSocialInteraction, animControl labels for plotting purposes.
Changes to support addition of subEvents - preprocessLogFileMonkeyLogic now loads the eventData table (from a .mat in the stimulus directory) and checks stimuli for events. If it finds subEvents, it generates new "taskEventStartTimes" + others to introduce these into normal phyzzy pipeline as events. - alignAnalogIn and alignLFP have try and catch loops integrated to allow for near end time subEvents to not run over te available data (empty spaces remain 0s). - Added 'headTurn' and 'bodyTurn' events to paramFile. - Added a few fields to and regenerated the frameMotionData files (related to frame rate, count). made changes to preprocessLogFileMonkeyFile. - Added subEvents to buildStimParamFileSocialVids_Auto, since these are currently treated as normal stimuli, and if these aren't represented in the stimParamFile, an assert fails. - processRunBatch now looks for the 2nd page of a 'dataLog' file, and can swap parameters for individual runs of a batchRun. developed with the end of preventing reliable errors due to photodiode calibration, and waveClus. Also rearranged processRunBatch first segment for readability.
…alysis to runAnalyses. - Replaced a lot of inefficient code, removed generateHeadTurnList since it is being replaced with eventDetectionApp. - added spacing and commenting for clarity.
- calcStimPSTH was taking in many variables simply to use their lengths. Those lines have been replaced and now the input structure, which has this information, is referenced. - calcSpikeTimes likewise took many variables needlessly, ontop of many variables which came from single param structs. Now instead of unpacking these variables and accepting them, the struct of interest in directly taken. - plotPSTH has had code added under 'line' plotType, which it previously didn't. Requires mseb, which is in the dependecies folder, and allows for lineprops to be defined in psthParams. default input arguments expanded at 2nd slot to allow for addition of psthErrors (which mseb can use). All calls to plotPSTH have had an additional '[ ]' added. Line option can use many additions to make lines discernable more clearly.
- made some fixes in plotPSTH's line version. - processRunBatch now checks for paramTable row before attempting to assign new values for an individual run. - errorMsg in processRunBatch now reliably recorded in xlsx output. Using 'replaceAnalysisOut' doesn't grab errored trials still. This requires some slight changes in the list compilation in the "else" block starting at 155. - subEventAnalysis now outputs a few structures with the data, which are saved to the analyzedData mat for later extraction in processBatchAnalysis. - processBatchAnalysis plotPSTH functions updated with correct argument count. - fixed bug in SocialVids_Auto which assigned scrambles inappropriate labels.
- psthBySubEvent analyses added to runBatchAnalysis. Still working on them. - rasterColorCoded bug fixed which caused labels on raster to not be generated correctly (trialLabels was in the trial loop, needed to just be in item loop). - runAnalyses changes to processing of eye data (removing lfpPaddedBy from start index for figuring out gaze target in a frame). - commented out 2nd and 3rd Epoch for now, and fixed code which I'd previously written for which this caused a bug.
…anPSTH using eventData rasterColorCoded - handle management improved, offset removed for images, not sure if it will need to come back (Image didn't quite line up with raster behind). Added imae option for presenting spikes, though it seems like it isn't working well since even w/ only blacks and whites in the image, they can be seen as grey depending on zoom. maybe a simple fix somewhere. Also option to do Saccade image instead of non-saccade. - eyeStimOverlay allows for dots to be color coded based on target of gaze. this revealed discrepenacy between what is being highlighted and what the target of gaze is. params at the top of the function. - eyeStats introduced which uses ClusterFix to add in saccade times. - fixed problem in buildStimParamFileSocialVids_Auto which led to some videos being incorrectly labeled 'goalDirected'.
- ClusterFix's introduction brought about a problem - it smooths the signal, and this smooth signal was being used to determine saccades but not showing up on eye signal visualization, meaning odd differences might emerge. I'd wanted to smooth eye signal before, so this seemed like the chance. - After trying to use the digital filter objects in the analysisParamFile, I discovered clusterFix requires unsmoothed data (even when coding a switch to show it data is smoothed and skipping the initial smoothing). - I modified clusterFix to pass out smoothed traces, modified detectSaccade to pass out these traces in the right format, and pass them to 3 other functions which use eye signal - eyeStimOverlay, attendedObj, and eyeCorrelogram. These functions now take in just eye data instead of the analogInByEvent. - There are odd discrepencies between attendedObj labels and what is visualized on videos through eyeStimOverlay. Next task is to find out why these don't match.
…o colorCodedRaster - colorCodedRaster now processes colorCodedSwitch to determine what it will do. - colorCodedRaster can now visualize saccades as an image behind spikes. - Fixed issues for visualizing attendedObj data in stimEyeOverlay.
- saccadeDetect has been updated to produce an image which has fixations at 1, Saccades at 2, and Blinks at 3. This image is laid behind rasters when prefImRasterColorCoded is set to 3. - In order to fix an issue I had with ClusterFix, I've turned the preprocessing of the signal from a 30 Hz Low pass to a 25 Hz low pass, but the signal which is passed out is still 30 Hz Low pass. - preprocessEyeSignals now outputs taskData, with the max values for each trace appended, to aid in blink removal later on.
ClusterFix now internally processing eye signal twice - Once with a 30 Hz low pass filter for export to other functions, and then with a 2nd filter (set in analysisParam) for internal processing. I may need to tune this function to avoid detecting microsaccades if attempts to do so outside the function fail. - made changes to how saccade down sampling happens (converted to interp1 like the rest). - added option to add frame number to the lower left of the video for eyeStimOverlay. - added simple saccade filtration based on mean distance (may need to revise).
- clusterFix now outputs a 7th value for every saccade and fixation (max distance in the event). - saccadeDetect function in runAnalyses now read out this new value and removes saccades based on this. - the visualization feature of saccadeDetect for every trace now color codes saccades, and puts in the legend the max and mean distances of each event. This is for testing if output is appropriate and how one might tune filters/thresholds to get the desired result. Also now uses uitab instead of subplot for more room. - clusterFix no longer has the fltIn vs fltOut, after comparing and implementing a 2nd solution for not having saccades (jumps to the end, leaves all fields blank), 30 Hz should be the only frequency used. - processRunBatch now accepts a 2nd argument which is a arg * 2 cell array, with column 1 defining a argument to replace, and arg 2 being the replacement. This was implemented for testing different filters without needing to initiate every batchRun.
- added switch for 'plotPaths' to turn off plotting of each stimuli, trial by trial, unless checking for something. - removed hardcoding of 3 epochs from processRunBatch. - removed parameter dependent analysisLabel, made earlier for testing filtering.
- processRunBatch has a feature to generate excel files, and something about the preceeding loop led to crashes where specific field wasn't being added. Now the code cycles through and identifies such missing fields instead of crashing. Output of this script needs fixing (output is now far too few entries for some reason). - saccadeDetect renamed to eyeStatsAnalysis, and eyeBehStatsByStim, which is (stim)(trial) output of clusterFix now has a 'blinktimes' field added. all 'times' fields are shifted back by psthPre to align 0 to stimulus onset. - removed part in runAnalysis that hardcoded 3 epochs. - Next steps are to modify subEventAnalysis to accept not only the subEvent table, but times for any subEvents, like blinks and saccades.
…duction - runAnalyses now has a 'pupilDilation' function, which generates a trial by trial vector of pupil dilation in a similar way to the saccade function. - individual variables with eye data have been removed and replaced with a 'eyeDataStruct' which is accepted and returned by arguments which produce such data. - modified colorCodedRaster code to accept pupilDilation, and read the eyeDataStruct. - subEventAnalysis now accepts blink and saccade data to generate psthes aligned by these events. - added modified version of ClusterFix function (BehavioralIndexPlus) which aids in removal of blinks from pupilDilation traces.
- pupilDilation now puts own Image w/o buffers, as intended. - processRunBatch excel generating code now more resistant to errors, many variables initialized and checks used to prevent table from crashing when being populated.
- spikePupilCorr does a trial by trial correlation of pupil diameter and single trial PSTH. It currently produces 2 matricies but needs to be updated to produce a better summary of the data, waiting on inputs from others. - will start changing subEventAnalysis function to plot individual traces for stim/event combinations and to output more data about stim/event combinations. current setup has most of runAnalyses beginning in a if 0 statement, and a load after to speed up testing.
- spikePupilCorr now produces plots which generate a histogram of pupil values during spikes vs duringNon-Spikes. - Next steps to fix subEventAnalysis - recent addition of individual traces seems to be adding something wrong.
- subEventAnalysis outputs restructured, expanded to include individual 'specific subEvents', along with their own PSTHes. - figure at end of subEventAnalysis shifted to tabs instead of subplots to allow for proper viewing. - changed naming convention in subEventAnalysis for readability in legend. - changed indexing to properly pull specific subEvent PSTHes for plotting correctly. - changed plotPSTH to return vertical line handles, to allow for resizing when needed. - changed draw_vert_line to return handle.
- fixed bug in spikePupilCorr. - next step is to shift figures from this function from per stimulus with plots for each unit, to per unit with some representation of each stimulus. This may mess up the code, hence this commit.
… blinks. figStruct - a struct generated in analysisParam which combines fig relevant parameters (many of the switch near the top of the default analysisParam + some new ones) and packages 'channelUnitNames' for the sake of appending to titles. runAnalyses - improved argument handlings where parameter structs don't undergo revision outside of the switches which use them (allows for a bit more transparency and less dependency issues). spikePupilCorr - updated to produce figures with the right titles, as well as a "Per Stim" style image, with all the units plotted, and a "Per Unit" style image. One form of the "Per Unit" style image is in need of revision, as it is deeply redundant in some ways. 3 switches sit inside this function - 1 to exclude pre and post stim period, a 'plotType' to decide between the two mentioned plots, a 2nd 'figureType' which decides between the 2 'plotType = 2" plots. (Bit messy...). pupilDilation now places NaNs in the place of blinks to give a more accurate picture of values known.
- runAnalyses now has a 'catIndStruct', which contains a event * catagory index of which events belong to which catagories. - pupilDilation fxn takes in catIndStruct, along with figStruct, to generate new figures. - "Pupil Catagory vs Non-Catagory" - plots histogram of stimuli associated w/ or not associated w/ a particular catagory. - "Pupil Adaptation" - plots the mean and SD of the pupil over multiple repetitions of the stimulus within a run (each trial). - To-Do: Add appropriate outputs to fxn, then save to 'analyzedData'. Add line to save figures generated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Few files to allow for MonkeyLogic Processing, as well as changes which allow for Offline spike sorting.