BML_SYNC_ANALOG time-aligns files based on a common analog sync channel
Use as
sync_roi = bml_sync_digital(cfg)
cfg - configuration structure (reuired fields)
cfg.roi - roi table with vars 'id','starts','ends','folder','name',
'nSamples','filetype'. Contains a coarse alignment of the
files, normally inferred from the OS 'Date-Modified' metadata.
'starts' and 'ends' should be given in seconds from midnight.
cfg.sync_channels - table with vars 'filetype', 'channel', 'chantype'
This table defines how channels of different filetypes will be
mapped with each other.
cfg.chunks - annot table: defines starts and ends of chunks of time to sync
in master time. Usually corresponds to sessions but can be
shorter periods.
cfg.master_filetype - string: filetype that defines filetype used as master time,
to which to align other filetypes
cfg.sync_roi - roi table with previous results of current run. If
provided, the algorithm will skip doing the same
synchronization chunks. Useful for iterative chunking.
cfg - configuration structure (optional fields)
cfg.timewarp - logical: Should slave time be warped? defaults to true.
cfg.lpf - logical: low-pass-filter alignment if true (default)
cfg.praat - logical: should synchronized files be opened in praat for
manual quality check. Defaults to false.
cfg.chunk_extend - double or double array of length 2: amount of seconds by
which to extend each sync chunk in slave files to avoid cropping out
relevant part of because of incorrect initial alignemnt
files. Defaults to 0 seconds.
cfg.resample_freq - double: frequency in Hz at which to resample master
and slave raws. Defaults to 10000.
cfg.dryrun - logical: if true no alignment is performed (defaults to
false)
cfg.env_freq - double: frequency of the envelope used for coarse
alignement (Hz). Defaults to 100. Note that
resample_freq/env_freq should be an integer.
cfg.env_scan - double: number of seconds in which to scan for initial
coarse grain alignement between master and slave's envelopes.
Defaults to 300 seconds (5 minutes).
cfg.env_penalty_wt0_min - double: penalty parameter for midpoint shift in
coarse time-warp. Defaults to 1e-3. (see BML_TIMEWARP)
cfg.env_penalty_ws1 - double: penalty parameter for time stretching in
coarse time-warp. Defaults to 1e-3. (see BML_TIMEWARP)
cfg.lpf_max_freq - double: maximum low-pass-filter cutoff frequency (Hz).
The value used is the minimum between this argument and the
master's and slave's sampling frequency. Defaults to 4000 Hz.
cfg.lpf_scan - double: number of seconds in which to scan for fine-grain
alignement between master and slave's low-pass-filter signal.
Defaults to 1 seconds.
cfg.lpf_penalty_wt0_min - double: penalty parameter for midpoint shift in
fine-grain time-warp. Defaults to 1e-6. (see BML_TIMEWARP)
cfg.lpf_penalty_ws1 - double: penalty parameter for time stretching in
fine-grain time-warp. Defaults to 1e-4. (see BML_TIMEWARP)
cfg.ft_feedback - string: default to 'no'. Defines verbosity of fieldtrip
functions
cfg.discontinuous - string or logical:
* true or 'allow' to allow discontinous files to be loaded filling
the gap with zero-padding, if possible within timetol.
* false or 'no' to issue an error if discontinous files are found
* 'warn' to allow with a warning (default)
cfg.high_pass - logical: should high pass filter be applied before
alignment. Defaults to false.
cfg.high_pass_freq - float: high pass frequency in Hz. Defaults to 5 Hz
returns roi table with vars
id: integer identification number of the synchronized file chunk
starts: start time in seconds from midnight of the represented signal
ends: end time in seconds from midnight of the represented signal
duration: duration in seconds as calculated by ends - starts
s1: first sample number of synchronization coordinate
t1: midpoint time of sample s1. Note that if s1==1 => t1=starts+0.5/Fs
s2: last sample number of synchronization coordinate
t2: midpoint time of sample s2. Note that if s2==end => t2=ends-0.5/Fs
folder:
name: file name. Note that several each file can have several file
chunks, i.e. several rows in this table
nSamples: integer total number of samples of the file
filetype:
--------------------------------------------------------------------------
The algorithm first opens time chucks (defined in cfg.chunks) of the files
defined in cfg.roi, loading the channels defined in cfg.sync_channels.
The coarse alignment of cfg.roi should be within a 60 second tolerance.
It then calculates the envelope of the channels with BML_ENVELOPE_BINABS,
and aligns these envelopes with BML_TIMEALIGN. If cfg.timewarp is true,
it applies a time-warping algorithm as defined in BML_TIMEWARP to
maximize the correlation between slave and master channels. If cfg.lpf is
true, it then repeats these processes for a low-pass filter version of
the channels. If cfg.praat is true, the resulting synchronized chunks are
loaded in praat. The function returns a synchronization roi table.0001 function sync_roi = bml_sync_analog(cfg) 0002 0003 % BML_SYNC_ANALOG time-aligns files based on a common analog sync channel 0004 % 0005 % Use as 0006 % sync_roi = bml_sync_digital(cfg) 0007 % 0008 % cfg - configuration structure (reuired fields) 0009 % cfg.roi - roi table with vars 'id','starts','ends','folder','name', 0010 % 'nSamples','filetype'. Contains a coarse alignment of the 0011 % files, normally inferred from the OS 'Date-Modified' metadata. 0012 % 'starts' and 'ends' should be given in seconds from midnight. 0013 % cfg.sync_channels - table with vars 'filetype', 'channel', 'chantype' 0014 % This table defines how channels of different filetypes will be 0015 % mapped with each other. 0016 % cfg.chunks - annot table: defines starts and ends of chunks of time to sync 0017 % in master time. Usually corresponds to sessions but can be 0018 % shorter periods. 0019 % cfg.master_filetype - string: filetype that defines filetype used as master time, 0020 % to which to align other filetypes 0021 % cfg.sync_roi - roi table with previous results of current run. If 0022 % provided, the algorithm will skip doing the same 0023 % synchronization chunks. Useful for iterative chunking. 0024 % 0025 % cfg - configuration structure (optional fields) 0026 % cfg.timewarp - logical: Should slave time be warped? defaults to true. 0027 % cfg.lpf - logical: low-pass-filter alignment if true (default) 0028 % cfg.praat - logical: should synchronized files be opened in praat for 0029 % manual quality check. Defaults to false. 0030 % cfg.chunk_extend - double or double array of length 2: amount of seconds by 0031 % which to extend each sync chunk in slave files to avoid cropping out 0032 % relevant part of because of incorrect initial alignemnt 0033 % files. Defaults to 0 seconds. 0034 % cfg.resample_freq - double: frequency in Hz at which to resample master 0035 % and slave raws. Defaults to 10000. 0036 % cfg.dryrun - logical: if true no alignment is performed (defaults to 0037 % false) 0038 % cfg.env_freq - double: frequency of the envelope used for coarse 0039 % alignement (Hz). Defaults to 100. Note that 0040 % resample_freq/env_freq should be an integer. 0041 % cfg.env_scan - double: number of seconds in which to scan for initial 0042 % coarse grain alignement between master and slave's envelopes. 0043 % Defaults to 300 seconds (5 minutes). 0044 % cfg.env_penalty_wt0_min - double: penalty parameter for midpoint shift in 0045 % coarse time-warp. Defaults to 1e-3. (see BML_TIMEWARP) 0046 % cfg.env_penalty_ws1 - double: penalty parameter for time stretching in 0047 % coarse time-warp. Defaults to 1e-3. (see BML_TIMEWARP) 0048 % cfg.lpf_max_freq - double: maximum low-pass-filter cutoff frequency (Hz). 0049 % The value used is the minimum between this argument and the 0050 % master's and slave's sampling frequency. Defaults to 4000 Hz. 0051 % cfg.lpf_scan - double: number of seconds in which to scan for fine-grain 0052 % alignement between master and slave's low-pass-filter signal. 0053 % Defaults to 1 seconds. 0054 % cfg.lpf_penalty_wt0_min - double: penalty parameter for midpoint shift in 0055 % fine-grain time-warp. Defaults to 1e-6. (see BML_TIMEWARP) 0056 % cfg.lpf_penalty_ws1 - double: penalty parameter for time stretching in 0057 % fine-grain time-warp. Defaults to 1e-4. (see BML_TIMEWARP) 0058 % cfg.ft_feedback - string: default to 'no'. Defines verbosity of fieldtrip 0059 % functions 0060 % cfg.discontinuous - string or logical: 0061 % * true or 'allow' to allow discontinous files to be loaded filling 0062 % the gap with zero-padding, if possible within timetol. 0063 % * false or 'no' to issue an error if discontinous files are found 0064 % * 'warn' to allow with a warning (default) 0065 % cfg.high_pass - logical: should high pass filter be applied before 0066 % alignment. Defaults to false. 0067 % cfg.high_pass_freq - float: high pass frequency in Hz. Defaults to 5 Hz 0068 % 0069 % returns roi table with vars 0070 % id: integer identification number of the synchronized file chunk 0071 % starts: start time in seconds from midnight of the represented signal 0072 % ends: end time in seconds from midnight of the represented signal 0073 % duration: duration in seconds as calculated by ends - starts 0074 % s1: first sample number of synchronization coordinate 0075 % t1: midpoint time of sample s1. Note that if s1==1 => t1=starts+0.5/Fs 0076 % s2: last sample number of synchronization coordinate 0077 % t2: midpoint time of sample s2. Note that if s2==end => t2=ends-0.5/Fs 0078 % folder: 0079 % name: file name. Note that several each file can have several file 0080 % chunks, i.e. several rows in this table 0081 % nSamples: integer total number of samples of the file 0082 % filetype: 0083 % 0084 %-------------------------------------------------------------------------- 0085 % 0086 % The algorithm first opens time chucks (defined in cfg.chunks) of the files 0087 % defined in cfg.roi, loading the channels defined in cfg.sync_channels. 0088 % The coarse alignment of cfg.roi should be within a 60 second tolerance. 0089 % It then calculates the envelope of the channels with BML_ENVELOPE_BINABS, 0090 % and aligns these envelopes with BML_TIMEALIGN. If cfg.timewarp is true, 0091 % it applies a time-warping algorithm as defined in BML_TIMEWARP to 0092 % maximize the correlation between slave and master channels. If cfg.lpf is 0093 % true, it then repeats these processes for a low-pass filter version of 0094 % the channels. If cfg.praat is true, the resulting synchronized chunks are 0095 % loaded in praat. The function returns a synchronization roi table. 0096 % 0097 0098 %ToDo: check that filetype of roi is consitent with cfg.sync_channels 0099 %add examples to documentation 0100 0101 sync_channels = bml_getopt(cfg,'sync_channels'); 0102 master_filetype = bml_getopt_single(cfg,'master_filetype'); 0103 chunks = bml_getopt(cfg,'chunks'); 0104 chunk_extend = bml_getopt(cfg,'chunk_extend',0); 0105 roi_os = bml_roi_table(bml_getopt(cfg,'roi'),'roi_os'); 0106 prev_sync_roi = bml_getopt(cfg,'sync_roi'); 0107 praat = bml_getopt(cfg,'praat',false); 0108 resample_freq = bml_getopt(cfg,'resample_freq',10000); 0109 dryrun = bml_getopt(cfg,'dryrun',false); 0110 env_freq = bml_getopt(cfg,'env_freq',100); 0111 env_scan = bml_getopt(cfg,'env_scan',300); 0112 env_penalty_wt0_min = bml_getopt(cfg,'env_penalty_wt0_min',1e-3); 0113 env_penalty_ws1 = bml_getopt(cfg,'env_penalty_ws1',1e-3); 0114 lpf_max_freq = bml_getopt(cfg,'lpf_max_freq',4000); 0115 lpf = bml_getopt(cfg,'lpf',true); 0116 lpf_scan = bml_getopt(cfg,'lpf_scan',1); 0117 lpf_penalty_wt0_min = bml_getopt(cfg,'lpf_penalty_wt0_min',1e-6); 0118 lpf_penalty_ws1 = bml_getopt(cfg,'lpf_penalty_ws1',1e-4); 0119 timewarp = bml_getopt(cfg,'timewarp',true); 0120 ft_feedback = bml_getopt_single(cfg,'ft_feedback','no'); 0121 discontinuous = bml_getopt(cfg,'discontinuous','warn'); 0122 high_pass = bml_getopt(cfg,'high_pass',false); 0123 high_pass_freq = bml_getopt(cfg,'high_pass_freq',5); 0124 timetol = bml_getopt(cfg,'timetol',1e-6); 0125 0126 assert(~ismember('filetype',chunks.Properties.VariableNames),... 0127 'cfg.chunks should not containt ''filetype'' variable'); 0128 assert(~isempty(chunks),'empty chunks table'); 0129 0130 0131 0132 chunks = bml_annot_table(bml_chunk_sessions(chunks),'chunks'); 0133 0134 sync_roi = table(); 0135 sync_roi_vars = {'starts','ends','s1','t1','s2','t2','folder','name','nSamples','Fs','session_id','session_part','filetype'}; 0136 sync_roi_vars_out = [sync_roi_vars,{'chantype','chunk_id','warpfactor','sync_channel','sync_type'}]; 0137 0138 filetypes=unique(sync_channels.filetype); 0139 slave_filetypes = setdiff(filetypes,master_filetype); 0140 master_channel = sync_channels.channel{strcmp(sync_channels.filetype,master_filetype)}; 0141 master_chantype = sync_channels.chantype{strcmp(sync_channels.filetype,master_filetype)}; 0142 0143 extended_chunks = bml_annot_extend(chunks,chunk_extend); 0144 0145 if ~isempty(prev_sync_roi) %previous attempts to sync 0146 prev_sync_roi = bml_roi_table(prev_sync_roi,'prev'); 0147 assert(all(ismember({'sync_type','sync_channel','chantype'},prev_sync_roi.Properties.VariableNames)),... 0148 "variables sync_type, sync_channel and chantype required for cfg.sync_roi"); 0149 else %checking files before commiting to first round of synchronization 0150 for chunk_i=1:height(chunks) 0151 chunk_roi_os = bml_annot_intersect(roi_os, chunks(chunk_i,:)); 0152 for filetype_i=1:length(filetypes) 0153 cfg=[]; cfg.ft_feedback=ft_feedback; 0154 cfg.channel = sync_channels.channel{strcmp(sync_channels.filetype,filetypes(filetype_i))}; 0155 cfg.chantype = sync_channels.chantype{strcmp(sync_channels.filetype,filetypes(filetype_i))}; 0156 cfg.roi=chunk_roi_os(string(chunk_roi_os.filetype)==filetypes(filetype_i),:); 0157 cfg.dryrun=true; 0158 cfg.discontinuous=discontinuous; 0159 assert(height(cfg.roi)>0,'No files for filetype %s and chunk_id %i',... 0160 filetypes{filetype_i},chunks.id(chunk_i)); 0161 bml_load_continuous(cfg); %raises error if continuity is violated 0162 end 0163 end 0164 end 0165 0166 %synchronizing 0167 for chunk_i=1:height(chunks) 0168 chunk_id = chunks.id(chunk_i); 0169 0170 %interseting with chunks for master 0171 chunk_roi_os = bml_annot_intersect(roi_os, chunks(chunk_i,:)); 0172 master_chunk_roi_os=chunk_roi_os(strcmp(chunk_roi_os.filetype,master_filetype),:); 0173 0174 %intersecting with extended chunks for slave 0175 extended_chunk_roi_os = bml_annot_intersect(roi_os, extended_chunks(chunk_i,:)); 0176 0177 0178 do_this_chunk = true; 0179 %cheking if this chunk was previously done 0180 if ~isempty(prev_sync_roi) 0181 cfg=[]; 0182 cfg.overlap=0.001; 0183 prev_chunk_i = bml_annot_filter(cfg,prev_sync_roi,master_chunk_roi_os); 0184 prev_chunk_i_master = prev_chunk_i(strcmp(prev_chunk_i.sync_type,'master') &... 0185 strcmp(prev_chunk_i.sync_channel,master_channel) & ... 0186 strcmp(prev_chunk_i.chantype,master_chantype),:); 0187 %cheking consistency of previous and current syncs 0188 if height(prev_chunk_i_master)==height(master_chunk_roi_os) && ... 0189 abs(min(prev_chunk_i_master.starts) - min(master_chunk_roi_os.starts)) < timetol && ... 0190 abs(max(prev_chunk_i_master.ends) - max(master_chunk_roi_os.ends)) < timetol 0191 0192 do_this_chunk = false; %will skip the calculations for this chunk 0193 fprintf('skipping chunk %i consistent with chunk %i in cfg.sync_roi \n',chunk_id, unique(prev_chunk_i_master.chunk_id)); 0194 0195 %adding row info for master 0196 row = prev_chunk_i_master(:,sync_roi_vars_out); 0197 row.chunk_id(:) = chunk_id; %replacing chunk info 0198 sync_roi = [sync_roi;row]; 0199 0200 %adding row info for slaves 0201 row = prev_chunk_i(ismember(prev_chunk_i.filetype,slave_filetypes) & ... 0202 ismember(prev_chunk_i.sync_channel,sync_channels.channel(ismember(sync_channels.filetype,slave_filetypes))) &... 0203 prev_chunk_i.chunk_id == unique(prev_chunk_i_master.chunk_id), ... 0204 sync_roi_vars_out); 0205 row.chunk_id(:) = chunk_id; %replacing chunk info 0206 sync_roi = [sync_roi;row]; 0207 end 0208 end 0209 0210 if do_this_chunk 0211 cfg=[]; %creating masters raw with sync channel for entire session 0212 cfg.channel = master_channel; 0213 cfg.chantype = master_chantype; 0214 cfg.roi = master_chunk_roi_os; 0215 cfg.ft_feedback=ft_feedback; 0216 cfg.dryrun = dryrun; 0217 cfg.discontinuous=discontinuous; 0218 [master, master_map] = bml_load_continuous(cfg); 0219 0220 if high_pass 0221 master.trial{1} = ft_preproc_highpassfilter(master.trial{1},... 0222 master.fsample, high_pass_freq, 4, 'but', 'twopass'); 0223 end 0224 0225 row = master_chunk_roi_os(:,sync_roi_vars); 0226 row.chantype = repmat({master_chantype},height(row),1); 0227 row.chunk_id = repmat(chunk_id,height(row),1); 0228 row.warpfactor = ones(height(row),1); 0229 row.sync_channel = master_channel; 0230 row.sync_type = {'master'}; 0231 sync_roi = [sync_roi;row]; 0232 0233 if praat && ~dryrun 0234 bml_praat(strcat('c',num2str(chunk_id),'_master_',master_filetype),master); 0235 end 0236 0237 for slave_i=1:length(slave_filetypes) 0238 filetype_chunk_roi_os=extended_chunk_roi_os(string(extended_chunk_roi_os.filetype)==slave_filetypes(slave_i),:); 0239 slave_channel = sync_channels.channel{strcmp(sync_channels.filetype,slave_filetypes(slave_i))}; 0240 slave_chantype = sync_channels.chantype{strcmp(sync_channels.filetype,slave_filetypes(slave_i))}; 0241 0242 cfg=[]; %creating slave raw with sync channel for entire session 0243 cfg.ft_feedback=ft_feedback; 0244 cfg.channel = slave_channel; 0245 cfg.chantype = slave_chantype; 0246 cfg.roi = filetype_chunk_roi_os; 0247 cfg.dryrun = dryrun; 0248 cfg.discontinuous=discontinuous; 0249 [slave, slave_map] = bml_load_continuous(cfg); 0250 0251 if high_pass && ~dryrun 0252 slave.trial{1} = ft_preproc_highpassfilter(slave.trial{1},... 0253 slave.fsample, 5, 4, 'but', 'twopass'); 0254 end 0255 0256 %envelope alingment and warping 0257 if ~dryrun 0258 cfg=[]; cfg.ft_feedback=ft_feedback; 0259 cfg.resample_freq=resample_freq; cfg.timewarp=timewarp; 0260 cfg.method='envelope'; cfg.env_freq=env_freq; cfg.scan=env_scan; 0261 cfg.penalty_wt0_min=env_penalty_wt0_min; cfg.penalty_ws1=env_penalty_ws1; 0262 wc_env = bml_timewarp(cfg,master,slave); 0263 slave.time{1} = bml_idx2time(wc_env, 1:length(slave.time{1})); 0264 else 0265 wc_env = []; 0266 wc_env.s1 = min(slave_map.raw1); 0267 wc_env.s2 = max(slave_map.raw2); 0268 wc_env.t1 = min(slave_map.t1); 0269 wc_env.t2 = max(slave_map.t2); 0270 wc_env.wt0=0; 0271 wc_env.ws1=1; 0272 end 0273 0274 if lpf && ~dryrun %low-pass frequency filter alignment and warping 0275 cfg=[]; cfg.ft_feedback=ft_feedback; 0276 cfg.resample_freq=resample_freq; cfg.timewarp=timewarp; 0277 cfg.method='low-pass-filter'; cfg.scan=lpf_scan; 0278 cfg.lpf_freq=min([master.fsample,slave.fsample,lpf_max_freq]); 0279 cfg.penalty_wt0_min=lpf_penalty_wt0_min; cfg.penalty_ws1=lpf_penalty_ws1; 0280 wc_lpf = bml_timewarp(cfg,master,slave); 0281 slave.time{1} = bml_idx2time(wc_lpf, 1:length(slave.time{1})); 0282 else 0283 wc_lpf = wc_env; 0284 wc_lpf.ws1 = 1; 0285 end 0286 0287 %saving sync info 0288 row = filetype_chunk_roi_os(:,sync_roi_vars); 0289 if height(row) ~= height(slave_map) 0290 row = row(ismember(row.name,slave_map.name),:); 0291 end 0292 row.s1 = slave_map.s1; 0293 row.s2 = slave_map.s2; 0294 row.t1 = bml_idx2time(wc_lpf,slave_map.raw1); 0295 row.t2 = bml_idx2time(wc_lpf,slave_map.raw2); 0296 row.starts = row.t1 - 0.5./row.Fs; 0297 row.ends = row.t2 + 0.5./row.Fs; 0298 row.chantype=repmat({slave_chantype},height(row),1); 0299 row.sync_channel = repmat({slave_channel},height(row),1); 0300 row.sync_type = repmat({'slave'},height(row),1); 0301 row.warpfactor = repmat(wc_env.ws1*wc_lpf.ws1,height(row),1); 0302 row.chunk_id = repmat(chunk_id,height(row),1); 0303 sync_roi = [sync_roi; row]; 0304 0305 if praat && ~dryrun 0306 slave_crop = bml_conform_to(master,slave); 0307 bml_praat(strcat('c',num2str(chunk_id),'_slave_',slave_filetypes(slave_i)),slave_crop); 0308 end 0309 end 0310 end 0311 end 0312 0313 sync_roi = bml_roi_table(sync_roi); 0314 0315 for slave_i=1:length(slave_filetypes) 0316 sync_roi_i = sync_roi(strcmp(sync_roi.filetype,slave_filetypes{slave_i}),:); 0317 chantype = unique(sync_roi_i.chantype); 0318 sync_channel = unique(sync_roi_i.sync_channel); 0319 fprintf('Summary for slave filetype %s, chantype %s, sync_channel %s \n.',... 0320 slave_filetypes{slave_i},chantype{1},sync_channel{1}); 0321 sync_roi_i(:,{'id','starts','ends','duration','name','session_id','warpfactor'}) 0322 end 0323 0324 0325