Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
47042fd
Merge branch 'master' of https://github.com/CosmoStat/shapepipe into …
tobias-liaudat Jun 1, 2021
d7610d2
Adding modified working RCA runner
tobias-liaudat Jun 10, 2021
46e881a
Adding vignet saving in PSFEx interpolator
tobias-liaudat Jun 10, 2021
2c1415d
Adding modified working psfex/RCA merge_starcat runner
tobias-liaudat Jun 10, 2021
279c4ae
Adding modified working plots runner for PSFEx/RCA
tobias-liaudat Jun 10, 2021
d996ba0
Adding validation-only runner as input module
tobias-liaudat Jun 10, 2021
79cbfae
Merge branch 'master' of https://github.com/CosmoStat/shapepipe into …
tobias-liaudat Jun 29, 2021
3016060
New PSF interpolation function.
tobias-liaudat Jul 2, 2021
66d398a
cfis mccd trying outliers
martinkilbinger Jul 7, 2021
3d3551e
Cancel outlier rejection if the threshold is too high
tobias-liaudat Jul 13, 2021
1c89953
Merge remote-tracking branch 'upstream/tobi_develop' into tobi_develop
tobias-liaudat Jul 13, 2021
aefa1e3
Added MCCD config with more complex model
martinkilbinger Jul 16, 2021
116c74f
merge_star_cat_mccd fix to tobi_develop branch
martinkilbinger Jul 21, 2021
eba3c1a
Merge remote-tracking branch 'upstream/develop' into tobi_develop
martinkilbinger Jul 23, 2021
9e4ab52
Merge remote-tracking branch 'upstream/develop' into tobi_develop
martinkilbinger Jul 30, 2021
7552d1b
run MCCD v1.1.1; modules use module_config_sec
martinkilbinger Aug 2, 2021
d4f5238
Fixed minor print format bug in mccd plot utilities
martinkilbinger Aug 3, 2021
5408e19
job_sp: fixed vos string comparison bug
martinkilbinger Aug 3, 2021
8564ec3
Merge branch 'mccd_oc2' of github.com:martinkilbinger/shapepipe into …
martinkilbinger Aug 3, 2021
94411b4
job_sp: fixed again ln config_dir
martinkilbinger Aug 3, 2021
9741293
job_sp: fixed again ln config_dir
martinkilbinger Aug 3, 2021
6e2d81f
galsim run gave error in locating last module; disabled galsim
martinkilbinger Aug 4, 2021
ace63e0
fixed run analysis script
martinkilbinger Aug 4, 2021
b47c55d
Merge remote-tracking branch 'origin/overwrite' into mccd_oc2
martinkilbinger Aug 4, 2021
d898b5d
setools __init__
martinkilbinger Aug 4, 2021
6d473d4
canfar_avail_results updated, can use local path as input; README upd…
martinkilbinger Aug 5, 2021
7a0c6ea
Merge branch 'mccd_oc2' of github.com:martinkilbinger/shapepipe into …
martinkilbinger Aug 5, 2021
b41254a
Merge remote-tracking branch 'upstream/develop' into mccd_oc2
martinkilbinger Aug 5, 2021
11e2f96
Merge remote-tracking branch 'upstream/develop' into mccd_oc2
martinkilbinger Aug 5, 2021
ecb21ae
final_car.param only ngmix; untar with output file{
martinkilbinger Aug 6, 2021
ba47524
setools bug fixed (again?)
martinkilbinger Aug 6, 2021
a818a14
SETools deleted (again?)
martinkilbinger Aug 6, 2021
44aaf74
Merge branch 'tobi_develop' of github.com:CosmoStat/shapepipe into Co…
martinkilbinger Aug 6, 2021
577d508
Merge branch 'CosmoStat-tobi_develop' into mccd_oc2
martinkilbinger Aug 6, 2021
8401bb8
canfar.md from simul
martinkilbinger Aug 6, 2021
7d13a67
canfar.md updated
martinkilbinger Aug 6, 2021
f9b912f
Update canfar.md
martinkilbinger Aug 6, 2021
4781d19
commented vos log output
martinkilbinger Aug 6, 2021
722db91
Merge branch 'mccd_oc2' of github.com:martinkilbinger/shapepipe into …
martinkilbinger Aug 6, 2021
3f76540
scripts updated to MCCD
martinkilbinger Aug 16, 2021
374856a
post_proc: sp run for merge star cat and PSF plots
martinkilbinger Aug 16, 2021
14609d6
added get_image init
martinkilbinger Aug 17, 2021
7b784cf
Merge remote-tracking branch 'upstream/develop' into mccd_oc2
martinkilbinger Aug 17, 2021
2e0fe9a
Reduced DETECT_MINAREA to 5
martinkilbinger Aug 17, 2021
a30cadb
Reduced DETECT_MINAREA to 5 for exp
martinkilbinger Aug 17, 2021
13e7743
job v1
martinkilbinger Aug 19, 2021
4e96469
exp processing BATCH=4 due to memory issues mccd/more stars
martinkilbinger Aug 30, 2021
95d5b8c
post_proc set SP_RUN
martinkilbinger Sep 2, 2021
b97dd70
re-aranged psf post-proc handling
martinkilbinger Sep 2, 2021
2ffd778
Merge branch 'develop' into v1
martinkilbinger Sep 7, 2021
77d704e
Updated post-proc config file
martinkilbinger Sep 9, 2021
b13a515
added 2021 tiles list, 7 patches
martinkilbinger Sep 10, 2021
ee1d9d6
testing symlink /tmp/vospace
martinkilbinger Sep 14, 2021
bd91096
minor update jupyter script
martinkilbinger Sep 17, 2021
e1b8cb1
Merge branch 'v1' of github.com:martinkilbinger/shapepipe into v1
martinkilbinger Sep 17, 2021
4d2c983
First Code Clean Up (#456)
sfarrens Sep 20, 2021
d1c6522
merged master; resolved conflicts
martinkilbinger Sep 20, 2021
e593cf5
sextractor runner bug fix: argument order
martinkilbinger Sep 20, 2021
14b3070
fixed index bugs in sextractor script
martinkilbinger Sep 21, 2021
5d7d5d6
bug fixed in aux script
martinkilbinger Sep 21, 2021
22a2662
module_config_sec for merge_sep_cats runner
martinkilbinger Sep 21, 2021
25b38b4
Merge branch 'develop' into v1
martinkilbinger Sep 21, 2021
e53a7fb
exp processing: BATCH_SIZE = 2 to run on candide
martinkilbinger Sep 22, 2021
ceb1e80
file_handler: check_duplicate also returs string with duplicate element
martinkilbinger Sep 22, 2021
f7ad970
Clean-up of job_sp.bash file and config files (avoiding problems when…
martinkilbinger Sep 30, 2021
ef6231e
config use get_last patch
martinkilbinger Oct 1, 2021
bd2d457
Bug fixed: psfex catalogues were written in '.' instead of module out…
martinkilbinger Oct 1, 2021
e696150
towards repeted calls of job_sp
martinkilbinger Oct 6, 2021
655879d
Merge remote-tracking branch 'upstream/develop' into mult_tiles
martinkilbinger Jan 31, 2022
d11d994
running MaMa
martinkilbinger Feb 1, 2022
39f94ff
new uf job script running 16
martinkilbinger Feb 3, 2022
a443d67
Merge remote-tracking branch 'upstream/develop' into mult_tiles
martinkilbinger Feb 9, 2022
a828172
bug in config file fixed
martinkilbinger Feb 9, 2022
af914cf
Merge remote-tracking branch 'upstream/develop' into mult_tiles
martinkilbinger Feb 10, 2022
e9f10d3
colors in job script
martinkilbinger Feb 10, 2022
fdb450e
config files
martinkilbinger Feb 10, 2022
5901143
modules was removed in develop
martinkilbinger Feb 10, 2022
4c4d301
modules was removed in develop
martinkilbinger Feb 10, 2022
a397389
Removed copied functions
martinkilbinger Feb 25, 2022
430a391
Added previous RunLog methods as stand-alone functions
martinkilbinger Feb 25, 2022
5fae675
added run_log_file as arg
martinkilbinger Feb 25, 2022
77b15a1
updated calls to get_last_dir
martinkilbinger Feb 25, 2022
f3096bc
Added 'run_log' to run_dirs dict
martinkilbinger Feb 25, 2022
a362be7
passing tests
martinkilbinger Feb 25, 2022
ca9e660
added missing imports
martinkilbinger Feb 25, 2022
fe4dfda
api doc
martinkilbinger Feb 25, 2022
fccff6b
api doc changed for types
martinkilbinger Feb 25, 2022
8f507d7
Apply suggestions from code review
sfarrens Feb 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 144 additions & 0 deletions example/cfis/config_GitFeGie_symlink.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# ShapePipe configuration file for: get images


## Default ShapePipe options
[DEFAULT]

# verbose mode (optional), default: True, print messages on terminal
VERBOSE = False

# Name of run (optional) default: shapepipe_run
RUN_NAME = run_sp_GitFeGie

# Add date and time to RUN_NAME, optional, default: False
RUN_DATETIME = True


## ShapePipe execution options
[EXECUTION]

# Module name, single string or comma-separated list of valid module runner names
MODULE = get_images_runner, find_exposures_runner, get_images_runner

# Parallel processing mode, SMP or MPI
MODE = SMP


## ShapePipe file handling options
[FILE]

# Log file master name, optional, default: shapepipe
LOG_NAME = log_sp

# Runner log file name, optional, default: shapepipe_runs
RUN_LOG_NAME = log_run_sp

# Input directory, containing input files, single string or list of names
INPUT_DIR = $SP_RUN

# Output directory
OUTPUT_DIR = $SP_RUN/output


## ShapePipe job handling options
[JOB]

# Batch size of parallel processing (optional), default is 1, i.e. run all jobs in serial
SMP_BATCH_SIZE = 1

# Timeout value (optional), default is None, i.e. no timeout limit applied
TIMEOUT = 96:00:00


## Module options

# Get tiles
[GET_IMAGES_RUNNER_RUN_1]

FILE_PATTERN = tile_numbers

FILE_EXT = .txt

# NUMBERING_SCHEME (optional) string with numbering pattern for input files
NUMBERING_SCHEME =

# Paths

# Input path where original images are stored. Can be local path or vos url.
# Single string or list of strings
INPUT_PATH = $HOME/astro/data/CFIS/tiles_DR2, $HOME/astro/data/CFIS/tiles_DR2

# Input file pattern including tile number as dummy template
INPUT_FILE_PATTERN = CFIS.000.000.r, CFIS.000.000.r.weight

# Input file extensions
INPUT_FILE_EXT = .fits, .fits.fz

# Input numbering scheme, python regexp
INPUT_NUMBERING = \d{3}\.\d{3}

# Output file pattern without number
OUTPUT_FILE_PATTERN = CFIS_image-, CFIS_weight-

# Copy/download method, one in 'vos', 'symlink'
RETRIEVE = symlink

# Copy command options, optional
RETRIEVE_OPTIONS = -L


[FIND_EXPOSURES_RUNNER]

INPUT_MODULE = get_images_runner_run_1

FILE_PATTERN = CFIS_image

FILE_EXT = .fits

# NUMBERING_SCHEME (optional) string with numbering pattern for input files
NUMBERING_SCHEME = -000-000


# Get exposures
[GET_IMAGES_RUNNER_RUN_2]

INPUT_MODULE = find_exposures_runner

FILE_PATTERN = exp_numbers

FILE_EXT = .txt

# NUMBERING_SCHEME (optional) string with numbering pattern for input files
NUMBERING_SCHEME = -000-000


# Paths

# Output path (optional, default is [FILE]:OUTPUT_DIR
# OUTPUT_PATH = input_images

# Input path where original images are stored. Can be local path or vos url.
# Single string or list of strings
INPUT_PATH = $HOME/astro/data/CFIS/pitcairn_DR2, $HOME/astro/data/CFIS/weights_DR2, $HOME/astro/data/CFIS/flags_DR2

# Input file pattern including tile number as dummy template
INPUT_FILE_PATTERN = 000000, 000000.weight, 000000.flag

# Input file extensions
INPUT_FILE_EXT = .fits.fz, .fits.fz, .fits.fz

# Input numbering scheme, python regexp
INPUT_NUMBERING = \d{6}

# Output file pattern without number
OUTPUT_FILE_PATTERN = image-, weight-, flag-

# Method to retrieve images, one in 'vos', 'symlink'
RETRIEVE = symlink

# If RETRIEVE=vos, number of attempts to download
# Optional, default=3
N_TRY = 3

# Retrieve command options, optional
RETRIEVE_OPTIONS = -L
149 changes: 149 additions & 0 deletions example/cfis/config_GitFeGie_vos.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# ShapePipe configuration file for: get images


## Default ShapePipe options
[DEFAULT]

# verbose mode (optional), default: True, print messages on terminal
VERBOSE = False

# Name of run (optional) default: shapepipe_run
RUN_NAME = run_sp_GitFeGie

# Add date and time to RUN_NAME, optional, default: False
RUN_DATETIME = True


## ShapePipe execution options
[EXECUTION]

# Module name, single string or comma-separated list of valid module runner names
MODULE = get_images_runner, find_exposures_runner, get_images_runner

# Parallel processing mode, SMP or MPI
MODE = SMP


## ShapePipe file handling options
[FILE]

# Log file master name, optional, default: shapepipe
LOG_NAME = log_sp

# Runner log file name, optional, default: shapepipe_runs
RUN_LOG_NAME = log_run_sp

# Input directory, containing input files, single string or list of names
INPUT_DIR = $SP_RUN

# Output directory
OUTPUT_DIR = $SP_RUN/output


## ShapePipe job handling options
[JOB]

# Batch size of parallel processing (optional), default is 1, i.e. run all jobs in serial
SMP_BATCH_SIZE = 1

# Timeout value (optional), default is None, i.e. no timeout limit applied
TIMEOUT = 96:00:00


## Module options

# Get tiles
[GET_IMAGES_RUNNER_RUN_1]

FILE_PATTERN = tile_numbers

FILE_EXT = .txt

# NUMBERING_SCHEME (optional) string with numbering pattern for input files
NUMBERING_SCHEME =

# Paths

# Input path where original images are stored. Can be local path or vos url.
# Single string or list of strings
INPUT_PATH = vos:cfis/tiles_DR3, vos:cfis/tiles_DR3

# Input file pattern including tile number as dummy template
INPUT_FILE_PATTERN = CFIS.000.000.r, CFIS.000.000.r.weight

# Input file extensions
INPUT_FILE_EXT = .fits, .fits.fz

# Input numbering scheme, python regexp
INPUT_NUMBERING = \d{3}\.\d{3}

# Output file pattern without number
OUTPUT_FILE_PATTERN = CFIS_image-, CFIS_weight-

# Copy/download method, one in 'vos', 'symlink'
RETRIEVE = vos

# Copy command options, optional
RETRIEVE_OPTIONS = --certfile=$VM_HOME/.ssl/cadcproxy.pem


[FIND_EXPOSURES_RUNNER]

INPUT_MODULE = get_images_runner_run_1

FILE_PATTERN = CFIS_image

FILE_EXT = .fits

# NUMBERING_SCHEME (optional) string with numbering pattern for input files
NUMBERING_SCHEME = -000-000


# Get exposures
[GET_IMAGES_RUNNER_RUN_2]

INPUT_DIR = last:find_exposures_runner

FILE_PATTERN = exp_numbers

FILE_EXT = .txt

# NUMBERING_SCHEME (optional) string with numbering pattern for input files
NUMBERING_SCHEME = -000-000


# Paths

# Output path (optional, default is [FILE]:OUTPUT_DIR
# OUTPUT_PATH = input_images

# Input path where original images are stored. Can be local path or vos url.
# Single string or list of strings
INPUT_PATH = vos:cfis/pitcairn, vos:cfis/weights, vos:cfis/flags
# LSB images:
#INPUT_PATH = vos:cfis/lsb_individual, vos:cfis/weights, vos:cfis/flags

# Input file pattern including tile number as dummy template
INPUT_FILE_PATTERN = 000000, 000000.weight, 000000.flag
# LSB images
#INPUT_FILE_PATTERN = 000000s, 000000p.weight, 000000p.flag

# Input file extensions
INPUT_FILE_EXT = .fits.fz, .fits.fz, .fits.fz

# Input numbering scheme, python regexp
INPUT_NUMBERING = \d{6}

# Output file pattern without number
OUTPUT_FILE_PATTERN = image-, weight-, flag-

# Method to retrieve images, one in 'vos', 'symlink'
RETRIEVE = vos

# If RETRIEVE=vos, number of attempts to download
# Optional, default=3
N_TRY = 3

# Retrieve command options, optional
RETRIEVE_OPTIONS = --certfile=$VM_HOME/.ssl/cadcproxy.pem

Loading