Plasma2D extraction from MultiIDSLoader on ITER failing

[munechika-koyo]

Extracting a Plasma2D from a MultiIDSLoader does not seem to work in my case:

In [2]: multi = tf.imas2tofu.MultiIDSLoader(user='public',tokamak='iter',shot=134000,run=28,ids=['core_sources', 'equilibrium'])
Getting ids   [occ]  tokamak  user    version  shot    run  refshot  refrun
------------  -----  -------  ------  -------  ------  ---  -------  ------
core_sources  [0]    iter     public  3        134000  28   -1       -1
equilibrium   [0]    "        "       "        "       "    "        "

In [4]: _dshort = {'core_sources': {'1drhotn':'source[identifier.name=radiation].profiles_1d[time].grid.rho_tor_norm',
   ...:                             '1deEnergy':'source[identifier.name=radiation].profiles_1d[time].electrons.energy'
   ...:                             },
   ...:            'equilibrium': {'2dpsi': 'time_slice[time].profiles_2d[0].psi',
   ...:                            '2dR': 'time_slice[time].profiles_2d[0].r',
   ...:                            '2dZ': 'time_slice[time].profiles_2d[0].z'}}

In [5]: multi.set_shortcuts(dshort=_dshort)

In [7]: plasma = multi.to_Plasma2D()

Then the following error message appears:

/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
No / several matching signals for:
    - source[]['identifier', 'name'] = bremsstrahlung
    - nb.of matches: 0
        In ids core_sources, signal 1dbrem not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
No / several matching signals for:
    - source[]['identifier', 'name'] = lineradiation
    - nb.of matches: 0
        In ids core_sources, signal 1dline not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'1dbrem'
        In ids core_sources, signal 1dprad not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
No / several matching signals for:
    - source[]['identifier', 'name'] = lineradiation
    - nb.of matches: 0
        In ids core_sources, signal 1dpsi not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'1dpsi'
        In ids core_sources, signal 1drhopn not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2dBT not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2dBR not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2dBZ not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'2dBT'
        In ids equilibrium, signal 2dB not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2djT not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2dmeshFaces not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2dmeshNodes not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal 2dphi not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1844: RuntimeWarning: invalid value encountered i                                                                      n greater
  out[ii][np.abs(out[ii]) > 1.e30] = np.nan
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'2dphi'
        In ids equilibrium, signal 2drhotn not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'2dmeshNodes'
        In ids equilibrium, signal 2dtheta not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal strike0R not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal strike0Z not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'strike0R'
        In ids equilibrium, signal strike0 not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal strike1R not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal strike1Z not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'strike1R'
        In ids equilibrium, signal strike1 not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal x0R not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal x0Z not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'x0R'
        In ids equilibrium, signal x0 not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal x1R not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
list index out of range
        In ids equilibrium, signal x1Z not loaded !
  warnings.warn(msg)
/home/ITER/munechk/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py:1912: UserWarning:
'x1R'
        In ids equilibrium, signal x1 not loaded !
  warnings.warn(msg)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-7-51881e127524> in <module>
----> 1 plasma = multi.to_Plasma2D()

~/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/imas2tofu/_core.py in to_Plasma2D(self, tlim, dsig, t0, Name, occ, config, out, plot,                                                                       plot_sig, plot_X, bck, dextra, nan, pos)
   2657         if out == object or plot == True:
   2658             import tofu.data as tfd
-> 2659             plasma = tfd.Plasma2D( **plasma )
   2660             if plot == True:
   2661                 plasma.plot(plot_sig, X=plot_X, bck=bck)

~/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/data/_core.py in __init__(self, dtime, dradius, d0d, d1d, d2d, dmesh, config, Id, Nam                                                                      e, Exp, shot, fromdict, SavePath, SavePath_Include)
   2271         del kwdargs['self']
   2272         # super()
-> 2273         super(Plasma2D,self).__init__(**kwdargs)
   2274
   2275     def _reset(self):

~/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/utils.py in __init__(self, fromdict, **kwdargs)
   1279             self._reset()
   1280             self._set_Id(**kwdargs)
-> 1281             self._init(**kwdargs)
   1282         self._Done = True
   1283

~/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/data/_core.py in _init(self, dtime, dradius, dmesh, d0d, d1d, d2d, config, **kwargs)
   2528         largs = self._get_largs_dgeom()
   2529         kwdgeom = self._extract_kwdargs(kwdargs, largs)
-> 2530         self._set_dindrefdatagroup(**kwdindrefdatagroup)
   2531         self.set_dgeom(**kwdgeom)
   2532         self._dstrip['strip'] = 0

~/.conda/envs/tofu2/lib/python3.6/site-packages/tofu/data/_core.py in _set_dindrefdatagroup(self, dtime, dradius, dmesh, d0d, d1d, d2d)
   2628                     assert len(lkt) == 1
   2629                     axist = dradius[k0]['depend'].index(lkt[0])
-> 2630                     size = data.shape[1-axist]
   2631                 dindref[k0] = {'size':size,
   2632                                'group':'radius'}

IndexError: tuple index out of range

The configuration:

• tofu 1.4.1

And the loaded modules are:

[ munechk hpc-login04 ~ ] module list
Currently Loaded Modulefiles:
  1) GCCcore/6.4.0                                       16) ncurses/6.0-GCCcore-6.4.0                           31) Tk/8.6.8-GCCcore-6.4.0
  2) binutils/2.28-GCCcore-6.4.0                         17) libreadline/7.0-GCCcore-6.4.0                       32) Tkinter/3.6.4-intel-2018a-Python-3.6.4
  3) GCC/6.4.0-2.28                                      18) Tcl/8.6.8-GCCcore-6.4.0                             33) Szip/2.1.1-GCCcore-6.4.0
  4) icc/2018.1.163-GCC-6.4.0-2.28                       19) SQLite/3.21.0-GCCcore-6.4.0                         34) HDF5/1.10.1-intel-2018a
  5) ifort/2018.1.163-GCC-6.4.0-2.28                     20) XZ/5.2.3-GCCcore-6.4.0                              35) libxml2/2.9.7-GCCcore-6.4.0
  6) iccifort/2018.1.163-GCC-6.4.0-2.28                  21) GMP/6.1.2-GCCcore-6.4.0                             36) MDSplus/7.46.1-intel-2018a
  7) impi/2018.1.163-iccifort-2018.1.163-GCC-6.4.0-2.28  22) libffi/3.2.1-GCCcore-6.4.0                          37) MDSplus-Java/7.46.1-intel-2018a-Java-1.8.0_162
  8) iimpi/2018a                                         23) Python/3.6.4-intel-2018a                            38) Blitz++/0.10-GCCcore-6.4.0
  9) imkl/2018.1.163-iimpi-2018a                         24) libyaml/0.1.7-GCCcore-6.4.0                         39) libMemcached/1.0.18-GCCcore-6.4.0
 10) intel/2018a                                         25) PyYAML/3.12-intel-2018a-Python-3.6.4                40) Boost/1.66.0-GCCcore-6.4.0-no_mpi
 11) numactl/2.0.11-GCCcore-6.4.0                        26) libpng/1.6.34-GCCcore-6.4.0                         41) UDA/2.2.6-GCCcore-6.4.0
 12) PGI/18.4-GCC-6.4.0-2.28                             27) freetype/2.9-GCCcore-6.4.0                          42) ant/1.10.1-Java-1.8.0_162
 13) Java/1.8.0_162(1.8)                                 28) expat/2.2.5-GCCcore-6.4.0                           43) IDStools/1.2.2-Python-3.6.4
 14) bzip2/1.0.6-GCCcore-6.4.0                           29) fontconfig/2.12.6-GCCcore-6.4.0                     44) IMAS/3.24.0-4.1.5
 15) zlib/1.2.11-GCCcore-6.4.0                           30) X11/20180131-GCCcore-6.4.0                          45) Anaconda3/5.0.1

[Didou09]

Error due to the fact tofu is trying to extract some radial profiles.
It assumes that radial profiles are always 2d, with time shape (nt, nr) where:

• nt = number of time steps
• nr = number of radial points

But in this case, there is only one time step, and the extracted array is flat (1d of shape (nr,)), instead of being of shape (1, nr).

=> solution: will implement a safety check, with a more explicit error message

[Didou09]

Should be fixed by PR #280 (available in 1.4.2 as soon as the Pull Request is validated and merged).
See the PR for more details.

Please note, @munechika-koyo , that you should:

• define the R and Z grid with shortcuts containing 'mesh' so tofu will recognize these fields as defining the rectangular mesh (I suggest defining shortcuts '2dmeshR' and '2dmeshZ' as below).
• Provide the keyword argument shapeRZ = ('R', 'Z') or ('Z', 'R'). This is to tell tofu in which order the data is stored in the 2d data arrays (because in your example R and Z have the same size, so it's not possible to guess the order from the data shape).

An example is provided below based on your needs:

In [1]: import tofu as tf
/home/ITER/vezined/ToFu_All/tofu/tofu/__init__.py:95: UserWarning: 
The following subpackages are not available:
    - tofu.mag
  => see tofu.dsub[<subpackage>] for details.
  warnings.warn(msg)

In [2]: tf.__version__
Out[2]: '1.4.2-alpha2-28-g806afc5'

In [3]: multi = tf.imas2tofu.MultiIDSLoader(user='public',tokamak='iter',shot=134000,run=28,ids=['core_sources', 'equilibrium'])
Getting ids   [occ]  tokamak  user    version  shot    run  refshot  refrun
------------  -----  -------  ------  -------  ------  ---  -------  ------
core_sources  [0]    iter     public  3        134000  28   -1       -1    
equilibrium   [0]    "        "       "        "       "    "        "     

In [4]: _dshort = {'core_sources': {'1drhotn':'source[identifier.name=radiation].profiles_1d[time].grid.rho_tor_norm',
   ...:    ...:                             '1deEnergy':'source[identifier.name=radiation].profiles_1d[time].electrons.energy'
   ...:    ...:                             },
   ...:    ...:            'equilibrium': {'2dpsi': 'time_slice[time].profiles_2d[0].psi',
   ...:    ...:                            '2dmeshR': 'time_slice[time].profiles_2d[0].r',
   ...:    ...:                            '2dmeshZ': 'time_slice[time].profiles_2d[0].z'}}
   ...:                            

In [5]: multi.set_shortcuts(dshort=_dshort)

In [6]: plasma = multi.to_Plasma2D(shapeRZ=('R', 'Z'))
/home/ITER/vezined/ToFu_All/tofu/tofu/imas2tofu/_core.py:484: RuntimeWarning: invalid value encountered in sqrt
  return np.sqrt( (psi - psi0[:,None]) / (psisep[:,None] - psi0[:,None]) )
/home/ITER/vezined/ToFu_All/tofu/tofu/imas2tofu/_core.py:1846: RuntimeWarning: invalid value encountered in greater
  out[ii][np.abs(out[ii]) > 1.e30] = np.nan
/home/ITER/vezined/ToFu_All/tofu/tofu/imas2tofu/_core.py:1962: UserWarning: The following data could not be retrieved:
	- equilibrium:
		2dB  : '2dBT'
		2dBR  : list index out of range
		2dBT  : list index out of range
		2dBZ  : list index out of range
		2djT  : list index out of range
		2dmeshFaces  : list index out of range
		2dmeshNodes  : list index out of range
		2dphi  : list index out of range
		2drhotn  : '2dphi'
		2dtheta  : '2dmeshNodes'
		strike0  : 'strike0R'
		strike0R  : list index out of range
		strike0Z  : list index out of range
		strike1  : 'strike1R'
		strike1R  : list index out of range
		strike1Z  : list index out of range
		x0  : 'x0R'
		x0R  : list index out of range
		x0Z  : list index out of range
		x1  : 'x1R'
		x1R  : list index out of range
		x1Z  : list index out of range
	- core_sources:
		1dbrem  : No / several matching signals for:     - source[]['identifier', 'name'] = bremsstrahlung     - nb.of matches: 0
		1dline  : No / several matching signals for:     - source[]['identifier', 'name'] = lineradiation     - nb.of matches: 0
		1dprad  : '1dbrem'
		1dpsi  : No / several matching signals for:     - source[]['identifier', 'name'] = lineradiation     - nb.of matches: 0
		1drhopn  : '1dpsi'
  warnings.warn(msg)

In [7]: npts = 100

In [8]: pts = np.array([np.linspace(4, 8, npts), np.zeros((npts)), np.zeros((npts,))])

In [9]: val, t = plasma.interp_pts2profile(pts, quant='2drhopn')

In [10]: val
Out[10]: 
array([[1.00000001, 1.00000001, 1.00000001, 1.00000001, 1.00000001,
        1.00000001, 1.00000001, 0.99519379, 0.9823119 , 0.9693463 ,
        0.95623018, 0.94737822, 0.93391692, 0.92022385, 0.90629477,
        0.89212509, 0.87770741, 0.86303376, 0.84809764, 0.83799228,
        0.82261002, 0.8069563 , 0.79102977, 0.77483118, 0.75836272,
        0.74162666, 0.72462484, 0.71314399, 0.69570641, 0.67801283,
        0.66006675, 0.64187223, 0.62343606, 0.60476769, 0.58587839,
        0.57316944, 0.55394466, 0.53454563, 0.51499928, 0.4953385 ,
        0.47560473, 0.45585194, 0.44270668, 0.42308562, 0.40366962,
        0.3845765 , 0.36591491, 0.34778222, 0.33029443, 0.31358574,
        0.30295342, 0.28788942, 0.27405435, 0.26165837, 0.25093208,
        0.2421167 , 0.23544763, 0.23113242, 0.2296427 , 0.22955887,
        0.23207063, 0.23711526, 0.2445543 , 0.25419447, 0.26581205,
        0.27917483, 0.28894178, 0.30472475, 0.32169771, 0.33968939,
        0.35854621, 0.37813249, 0.39833154, 0.41905435, 0.43309765,
        0.45441556, 0.47594585, 0.497604  , 0.51932443, 0.54105608,
        0.56275732, 0.58439388, 0.59876878, 0.62024028, 0.64158581,
        0.66278883, 0.68383636, 0.7047192 , 0.72542845, 0.74595545,
        0.7595356 , 0.77974593, 0.79976103, 0.81957919, 0.83920331,
        0.85864043, 0.87790024, 0.89699693, 0.9096461 , 0.92851068]])