In [1]:
%matplotlib inline
import pandas as pd
import socket
host = socket.getfqdn()

from core import  load, zoom, calc, save,plots,monitor
In [2]:
#reload funcs after updating ./core/*.py
import importlib
importlib.reload(load)
importlib.reload(zoom)
importlib.reload(calc)
importlib.reload(save)
importlib.reload(plots)
importlib.reload(monitor)
Out[2]:
<module 'core.monitor' from '/ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/core/monitor.py'>

If you submit the job with job scheduler, above¶

below are list of enviroment variable one can pass

%env local='2"¶

local : if True run dask local cluster, if not true, put number of workers setted in the 'local' if no 'local ' given, local will be setted automatically to 'True'

%env ychunk='2'¶

%env tchunk='2'¶

controls chunk. 'False' sets no modification from original netcdf file's chunk.¶

ychunk=10 will group the original netcdf file to 10 by 10¶

tchunk=1 will chunk the time coordinate one by one¶

%env file_exp=¶

'file_exp': Which 'experiment' name is it?¶

. this corresopnds to intake catalog name without path and .yaml¶

%env year=¶

for Validation, this correspoinds to path/year/month 's year¶

for monitoring, this corresponids to 'date' having * means do all files in the monitoring directory¶

setting it as 0[0-9] &1[0-9]& *[2-3][0-9], the job can be separated in three lots.¶

%env month=¶

for monitoring this corresponds to file path path-XIOS.{month}/¶

#

%env control=FWC_SSH¶

name of control file to be used for computation/plots/save/ & how it is called from Monitor.sh¶

Monitor.sh calls M_MLD_2D

and AWTD.sh, Fluxnet.sh, Siconc.sh, IceClim.sh, FWC_SSH.sh

  • AWTD.sh M_AWTMD

  • Fluxnet.sh M_Fluxnet

  • Siconc.sh M_Ice_quantities
  • IceClim.sh M_IceClim M_IceConce M_IceThick

FWC_SSH.sh M_FWC_2D M_FWC_integrals M_FWC_SSH M_SSH_anomaly

Integrals.sh M_Mean_temp_velo M_Mooring M_Sectionx M_Sectiony

%env save= proceed saving? True or False , Default is setted as True¶

%env plot= proceed plotting? True or False , Default is setted as True¶

%env calc= proceed computation? or just load computed result? True or False , Default is setted as True¶

%env save=False¶

%env lazy=False¶

For debugging this cell can help¶

%env file_exp=SEDNA_DELTA_MONITOR %env year=2012 %env month=01

0[1-2]¶

%env ychunk=10 %env ychunk=False %env save=False %env plot=True %env calc=True # %env lazy=False

False¶

%env control=M_Fluxnet

M_Sectiony ok with ychunk=False local=True lazy=False¶

In [3]:
%%time
# 'savefig': Do we save output in html? or not. keep it true. 
savefig=True
client,cluster,control,catalog_url,month,year,daskreport,outputpath = load.set_control(host)
!mkdir -p $outputpath
!mkdir -p $daskreport
client

local True
using host= irene5284.c-irene.mg1.tgcc.ccc.cea.fr starting dask cluster on local= True workers 16
10000000000
False
tgcc local cluster starting
This code is running on  irene5284.c-irene.mg1.tgcc.ccc.cea.fr using  SEDNA_DELTA_MONITOR file experiment, read from  ../lib/SEDNA_DELTA_MONITOR.yaml  on year= 2012  on month= 01  outputpath= ../results/SEDNA_DELTA_MONITOR/ daskreport= ../results/dask/6419275irene5284.c-irene.mg1.tgcc.ccc.cea.fr_SEDNA_DELTA_MONITOR_01M_Mean_temp_velo/
CPU times: user 3.92 s, sys: 750 ms, total: 4.67 s
Wall time: 1min 39s
Out[3]:

Client

Client-bf2df7b3-13da-11ed-b731-080038b9459d

Connection method: Cluster object Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

0d7302bb

Dashboard: http://127.0.0.1:8787/status Workers: 64
Total threads: 256 Total memory: 251.06 GiB
Status: running Using processes: True

Scheduler Info

Scheduler

Scheduler-aed10307-733a-4e63-9453-846a6be2a04e

Comm: tcp://127.0.0.1:43467 Workers: 64
Dashboard: http://127.0.0.1:8787/status Total threads: 256
Started: 1 minute ago Total memory: 251.06 GiB

Workers

Worker: 0
Comm: tcp://127.0.0.1:38235 Total threads: 4
Dashboard: http://127.0.0.1:43831/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40396
Local directory: /tmp/dask-worker-space/worker-uevkzb4i

Worker: 1
Comm: tcp://127.0.0.1:37095 Total threads: 4
Dashboard: http://127.0.0.1:41482/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34203
Local directory: /tmp/dask-worker-space/worker-l5m9uc_h

Worker: 2
Comm: tcp://127.0.0.1:40615 Total threads: 4
Dashboard: http://127.0.0.1:45348/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39357
Local directory: /tmp/dask-worker-space/worker-m2n6y_4b

Worker: 3
Comm: tcp://127.0.0.1:40431 Total threads: 4
Dashboard: http://127.0.0.1:44795/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45539
Local directory: /tmp/dask-worker-space/worker-rdr9g__s

Worker: 4
Comm: tcp://127.0.0.1:43159 Total threads: 4
Dashboard: http://127.0.0.1:45130/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38826
Local directory: /tmp/dask-worker-space/worker-s8gsjg_z

Worker: 5
Comm: tcp://127.0.0.1:40590 Total threads: 4
Dashboard: http://127.0.0.1:36895/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44701
Local directory: /tmp/dask-worker-space/worker-7buip29i

Worker: 6
Comm: tcp://127.0.0.1:40077 Total threads: 4
Dashboard: http://127.0.0.1:46717/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41434
Local directory: /tmp/dask-worker-space/worker-tos6g3t0

Worker: 7
Comm: tcp://127.0.0.1:45532 Total threads: 4
Dashboard: http://127.0.0.1:41831/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43191
Local directory: /tmp/dask-worker-space/worker-oo7e44rm

Worker: 8
Comm: tcp://127.0.0.1:37099 Total threads: 4
Dashboard: http://127.0.0.1:43012/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:33351
Local directory: /tmp/dask-worker-space/worker-mctejd8g

Worker: 9
Comm: tcp://127.0.0.1:33289 Total threads: 4
Dashboard: http://127.0.0.1:39973/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41022
Local directory: /tmp/dask-worker-space/worker-yva8f2jl

Worker: 10
Comm: tcp://127.0.0.1:44440 Total threads: 4
Dashboard: http://127.0.0.1:37790/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42683
Local directory: /tmp/dask-worker-space/worker-4dbrz9ye

Worker: 11
Comm: tcp://127.0.0.1:36285 Total threads: 4
Dashboard: http://127.0.0.1:45934/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39537
Local directory: /tmp/dask-worker-space/worker-aso1t1ec

Worker: 12
Comm: tcp://127.0.0.1:34503 Total threads: 4
Dashboard: http://127.0.0.1:42584/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37892
Local directory: /tmp/dask-worker-space/worker-baod6koa

Worker: 13
Comm: tcp://127.0.0.1:36852 Total threads: 4
Dashboard: http://127.0.0.1:42286/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:46187
Local directory: /tmp/dask-worker-space/worker-rpuq2p9q

Worker: 14
Comm: tcp://127.0.0.1:39573 Total threads: 4
Dashboard: http://127.0.0.1:40544/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41376
Local directory: /tmp/dask-worker-space/worker-273yugor

Worker: 15
Comm: tcp://127.0.0.1:46555 Total threads: 4
Dashboard: http://127.0.0.1:44945/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43405
Local directory: /tmp/dask-worker-space/worker-pfdjbnu6

Worker: 16
Comm: tcp://127.0.0.1:41892 Total threads: 4
Dashboard: http://127.0.0.1:46273/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36884
Local directory: /tmp/dask-worker-space/worker-zzrbzm2j

Worker: 17
Comm: tcp://127.0.0.1:41621 Total threads: 4
Dashboard: http://127.0.0.1:46283/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:32814
Local directory: /tmp/dask-worker-space/worker-n3lxzzk0

Worker: 18
Comm: tcp://127.0.0.1:38972 Total threads: 4
Dashboard: http://127.0.0.1:44534/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37763
Local directory: /tmp/dask-worker-space/worker-3pxv6b24

Worker: 19
Comm: tcp://127.0.0.1:34184 Total threads: 4
Dashboard: http://127.0.0.1:32951/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35924
Local directory: /tmp/dask-worker-space/worker-hcabx6ru

Worker: 20
Comm: tcp://127.0.0.1:34872 Total threads: 4
Dashboard: http://127.0.0.1:42851/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:33685
Local directory: /tmp/dask-worker-space/worker-u_heb13d

Worker: 21
Comm: tcp://127.0.0.1:36383 Total threads: 4
Dashboard: http://127.0.0.1:45740/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37639
Local directory: /tmp/dask-worker-space/worker-c4rmdfiw

Worker: 22
Comm: tcp://127.0.0.1:38836 Total threads: 4
Dashboard: http://127.0.0.1:46323/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40589
Local directory: /tmp/dask-worker-space/worker-ehac7j92

Worker: 23
Comm: tcp://127.0.0.1:32849 Total threads: 4
Dashboard: http://127.0.0.1:43442/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45452
Local directory: /tmp/dask-worker-space/worker-d4a9fc3_

Worker: 24
Comm: tcp://127.0.0.1:41920 Total threads: 4
Dashboard: http://127.0.0.1:39954/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45393
Local directory: /tmp/dask-worker-space/worker-9ko8kgjz

Worker: 25
Comm: tcp://127.0.0.1:35870 Total threads: 4
Dashboard: http://127.0.0.1:40605/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34245
Local directory: /tmp/dask-worker-space/worker-fx05f446

Worker: 26
Comm: tcp://127.0.0.1:42322 Total threads: 4
Dashboard: http://127.0.0.1:44418/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41681
Local directory: /tmp/dask-worker-space/worker-ncimavj3

Worker: 27
Comm: tcp://127.0.0.1:37839 Total threads: 4
Dashboard: http://127.0.0.1:34355/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39884
Local directory: /tmp/dask-worker-space/worker-byp80vf9

Worker: 28
Comm: tcp://127.0.0.1:39153 Total threads: 4
Dashboard: http://127.0.0.1:37590/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35195
Local directory: /tmp/dask-worker-space/worker-04t8ypdw

Worker: 29
Comm: tcp://127.0.0.1:43685 Total threads: 4
Dashboard: http://127.0.0.1:37696/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39586
Local directory: /tmp/dask-worker-space/worker-67xbp85v

Worker: 30
Comm: tcp://127.0.0.1:45774 Total threads: 4
Dashboard: http://127.0.0.1:40014/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40346
Local directory: /tmp/dask-worker-space/worker-3ywjx_84

Worker: 31
Comm: tcp://127.0.0.1:38914 Total threads: 4
Dashboard: http://127.0.0.1:35300/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42504
Local directory: /tmp/dask-worker-space/worker-29ggryj3

Worker: 32
Comm: tcp://127.0.0.1:41966 Total threads: 4
Dashboard: http://127.0.0.1:43573/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:32798
Local directory: /tmp/dask-worker-space/worker-e67jw53t

Worker: 33
Comm: tcp://127.0.0.1:45777 Total threads: 4
Dashboard: http://127.0.0.1:34934/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43590
Local directory: /tmp/dask-worker-space/worker-x1vtuwpo

Worker: 34
Comm: tcp://127.0.0.1:34596 Total threads: 4
Dashboard: http://127.0.0.1:37104/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39918
Local directory: /tmp/dask-worker-space/worker-tzubstkz

Worker: 35
Comm: tcp://127.0.0.1:41794 Total threads: 4
Dashboard: http://127.0.0.1:44253/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42044
Local directory: /tmp/dask-worker-space/worker-8mq9omz4

Worker: 36
Comm: tcp://127.0.0.1:42429 Total threads: 4
Dashboard: http://127.0.0.1:41958/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35608
Local directory: /tmp/dask-worker-space/worker-9nlk2d_j

Worker: 37
Comm: tcp://127.0.0.1:40225 Total threads: 4
Dashboard: http://127.0.0.1:36939/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45827
Local directory: /tmp/dask-worker-space/worker-z3g2pnps

Worker: 38
Comm: tcp://127.0.0.1:43615 Total threads: 4
Dashboard: http://127.0.0.1:37888/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:32779
Local directory: /tmp/dask-worker-space/worker-bry6o06q

Worker: 39
Comm: tcp://127.0.0.1:40967 Total threads: 4
Dashboard: http://127.0.0.1:42222/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40008
Local directory: /tmp/dask-worker-space/worker-d96sjwm3

Worker: 40
Comm: tcp://127.0.0.1:36237 Total threads: 4
Dashboard: http://127.0.0.1:42356/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36120
Local directory: /tmp/dask-worker-space/worker-8thsapw9

Worker: 41
Comm: tcp://127.0.0.1:43398 Total threads: 4
Dashboard: http://127.0.0.1:40323/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:46290
Local directory: /tmp/dask-worker-space/worker-lritkzlg

Worker: 42
Comm: tcp://127.0.0.1:34131 Total threads: 4
Dashboard: http://127.0.0.1:40415/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45742
Local directory: /tmp/dask-worker-space/worker-0io0vtxn

Worker: 43
Comm: tcp://127.0.0.1:43906 Total threads: 4
Dashboard: http://127.0.0.1:42337/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38985
Local directory: /tmp/dask-worker-space/worker-eimj9h9z

Worker: 44
Comm: tcp://127.0.0.1:44197 Total threads: 4
Dashboard: http://127.0.0.1:46543/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34907
Local directory: /tmp/dask-worker-space/worker-ihuhjapj

Worker: 45
Comm: tcp://127.0.0.1:38764 Total threads: 4
Dashboard: http://127.0.0.1:34531/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44599
Local directory: /tmp/dask-worker-space/worker-5zo_ej9s

Worker: 46
Comm: tcp://127.0.0.1:44102 Total threads: 4
Dashboard: http://127.0.0.1:43998/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43920
Local directory: /tmp/dask-worker-space/worker-n1_efbh9

Worker: 47
Comm: tcp://127.0.0.1:34212 Total threads: 4
Dashboard: http://127.0.0.1:42509/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38247
Local directory: /tmp/dask-worker-space/worker-ymkj753i

Worker: 48
Comm: tcp://127.0.0.1:46825 Total threads: 4
Dashboard: http://127.0.0.1:42704/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44643
Local directory: /tmp/dask-worker-space/worker-e13dt2x0

Worker: 49
Comm: tcp://127.0.0.1:40021 Total threads: 4
Dashboard: http://127.0.0.1:37335/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34698
Local directory: /tmp/dask-worker-space/worker-o7ytxyjx

Worker: 50
Comm: tcp://127.0.0.1:37703 Total threads: 4
Dashboard: http://127.0.0.1:45642/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38269
Local directory: /tmp/dask-worker-space/worker-v8ny21m0

Worker: 51
Comm: tcp://127.0.0.1:43330 Total threads: 4
Dashboard: http://127.0.0.1:33369/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36312
Local directory: /tmp/dask-worker-space/worker-xqhdqhi0

Worker: 52
Comm: tcp://127.0.0.1:38913 Total threads: 4
Dashboard: http://127.0.0.1:33114/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:46840
Local directory: /tmp/dask-worker-space/worker-e9xqzuik

Worker: 53
Comm: tcp://127.0.0.1:41643 Total threads: 4
Dashboard: http://127.0.0.1:35683/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42155
Local directory: /tmp/dask-worker-space/worker-okyr31c3

Worker: 54
Comm: tcp://127.0.0.1:43941 Total threads: 4
Dashboard: http://127.0.0.1:34767/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40816
Local directory: /tmp/dask-worker-space/worker-gb8ae72b

Worker: 55
Comm: tcp://127.0.0.1:37093 Total threads: 4
Dashboard: http://127.0.0.1:46452/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35601
Local directory: /tmp/dask-worker-space/worker-eas7qoxk

Worker: 56
Comm: tcp://127.0.0.1:37592 Total threads: 4
Dashboard: http://127.0.0.1:46062/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35510
Local directory: /tmp/dask-worker-space/worker-1cmhtwij

Worker: 57
Comm: tcp://127.0.0.1:35207 Total threads: 4
Dashboard: http://127.0.0.1:37052/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41917
Local directory: /tmp/dask-worker-space/worker-0fnqd_uf

Worker: 58
Comm: tcp://127.0.0.1:45230 Total threads: 4
Dashboard: http://127.0.0.1:42724/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37042
Local directory: /tmp/dask-worker-space/worker-h_ppsr51

Worker: 59
Comm: tcp://127.0.0.1:38798 Total threads: 4
Dashboard: http://127.0.0.1:33171/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:33011
Local directory: /tmp/dask-worker-space/worker-it5z9mik

Worker: 60
Comm: tcp://127.0.0.1:44248 Total threads: 4
Dashboard: http://127.0.0.1:44114/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40694
Local directory: /tmp/dask-worker-space/worker-16mguce2

Worker: 61
Comm: tcp://127.0.0.1:40798 Total threads: 4
Dashboard: http://127.0.0.1:42992/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41528
Local directory: /tmp/dask-worker-space/worker-1sv3ps68

Worker: 62
Comm: tcp://127.0.0.1:37257 Total threads: 4
Dashboard: http://127.0.0.1:41845/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:46222
Local directory: /tmp/dask-worker-space/worker-_wu74tq2

Worker: 63
Comm: tcp://127.0.0.1:40362 Total threads: 4
Dashboard: http://127.0.0.1:44510/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:33727
Local directory: /tmp/dask-worker-space/worker-e6p4goz1

read plotting information from a csv file¶

In [4]:
df=load.controlfile(control)
#Take out 'later' tagged computations
#df=df[~df['Value'].str.contains('later')]
df
Out[4]:
Value Inputs Equation Zone Plot Colourmap MinMax Unit Oldname Unnamed: 10
Mean Temp & Velocity gridV.vomecrty,gridT.votemper,param.mask,param... calc.Mean_temp_velo(data) FramS_Small Mean_temp_velo_integrals None ((0,4),(0,10)) (°C,cm.s^-1) I-5

Computation starts here¶

Each computation consists of

  1. Load NEMO data set
  2. Zoom data set
  3. Compute (or load computed data set)
  4. Save
  5. Plot
  6. Close
In [5]:
%%time
import os
calcswitch=os.environ.get('calc', 'True') 
lazy=os.environ.get('lazy','False' )
loaddata=((df.Inputs != '').any()) 
print('calcswitch=',calcswitch,'df.Inputs != nothing',loaddata, 'lazy=',lazy)
data = load.datas(catalog_url,df.Inputs,month,year,daskreport,lazy=lazy) if ((calcswitch=='True' )*loaddata) else 0 
data

calcswitch= True df.Inputs != nothing True lazy= False
../lib/SEDNA_DELTA_MONITOR.yaml
using param_xios reading  ../lib/SEDNA_DELTA_MONITOR.yaml
using param_xios reading  <bound method DataSourceBase.describe of sources:
  param_xios:
    args:
      combine: nested
      concat_dim: y
      urlpath: /ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc
      xarray_kwargs:
        compat: override
        coords: minimal
        data_vars: minimal
        parallel: true
    description: SEDNA NEMO parameters from MPI output  nav_lon lat fails
    driver: intake_xarray.netcdf.NetCDFSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/
>
{'name': 'param_xios', 'container': 'xarray', 'plugin': ['netcdf'], 'driver': ['netcdf'], 'description': 'SEDNA NEMO parameters from MPI output  nav_lon lat fails', 'direct_access': 'forbid', 'user_parameters': [{'name': 'path', 'description': 'file coordinate', 'type': 'str', 'default': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/MESH/SEDNA_mesh_mask_Tgt_20210423_tsh10m_L1/param'}], 'metadata': {}, 'args': {'urlpath': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc', 'combine': 'nested', 'concat_dim': 'y'}}
0 read gridT ['votemper']
lazy= False
using load_data_xios_kerchunk reading  gridT
using load_data_xios_kerchunk reading  <bound method DataSourceBase.describe of sources:
  data_xios_kerchunk:
    args:
      consolidated: false
      storage_options:
        fo: file:////ccc/cont003/home/ra5563/ra5563/catalogue/DELTA/201201/gridT_0[0-5][0-9][0-9].json
        target_protocol: file
      urlpath: reference://
    description: CREG025 NEMO outputs from different xios server in kerchunk format
    driver: intake_xarray.xzarr.ZarrSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/
>
      took 23.885221242904663 seconds
0 merging gridT ['votemper']
1 read gridV ['vomecrty']
lazy= False
using load_data_xios_kerchunk reading  gridV
using load_data_xios_kerchunk reading  <bound method DataSourceBase.describe of sources:
  data_xios_kerchunk:
    args:
      consolidated: false
      storage_options:
        fo: file:////ccc/cont003/home/ra5563/ra5563/catalogue/DELTA/201201/gridV_0[0-5][0-9][0-9].json
        target_protocol: file
      urlpath: reference://
    description: CREG025 NEMO outputs from different xios server in kerchunk format
    driver: intake_xarray.xzarr.ZarrSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/
>
      took 22.873196363449097 seconds
1 merging gridV ['vomecrty']
      took 0.871453046798706 seconds
param nav_lat will be included in data
param mask2d will be included in data
param depth will be included in data
param mask will be included in data
param nav_lon will be included in data
ychunk= 5 calldatas_y_rechunk
sum_num (13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12)
start rechunking with (65, 65, 62, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 48)
end of y_rechunk
before rechunking t item (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
start rechunking t with 1
end of t_rechunk
CPU times: user 48.8 s, sys: 12.3 s, total: 1min 1s
Wall time: 1min 46s
Out[5]:
<xarray.Dataset>
Dimensions:        (t: 31, z: 150, y: 6540, x: 6560)
Coordinates:
    time_centered  (t) object dask.array<chunksize=(1,), meta=np.ndarray>
  * t              (t) object 2012-01-01 12:00:00 ... 2012-01-31 12:00:00
  * y              (y) int64 1 2 3 4 5 6 7 ... 6535 6536 6537 6538 6539 6540
  * x              (x) int64 1 2 3 4 5 6 7 ... 6555 6556 6557 6558 6559 6560
  * z              (z) int64 1 2 3 4 5 6 7 8 ... 143 144 145 146 147 148 149 150
    nav_lat        (y, x) float32 dask.array<chunksize=(65, 6560), meta=np.ndarray>
    mask2d         (y, x) bool dask.array<chunksize=(65, 6560), meta=np.ndarray>
    depth          (z, y, x) float32 dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
    mask           (z, y, x) bool dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
    nav_lon        (y, x) float32 dask.array<chunksize=(65, 6560), meta=np.ndarray>
Data variables:
    votemper       (t, z, y, x) float32 dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
    vomecrty       (t, z, y, x) float32 dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
Attributes: (12/26)
    CASE:                    DELTA
    CONFIG:                  SEDNA
    Conventions:             CF-1.6
    DOMAIN_dimensions_ids:   [2, 3]
    DOMAIN_halo_size_end:    [0, 0]
    DOMAIN_halo_size_start:  [0, 0]
    ...                      ...
    nj:                      13
    output_frequency:        1d
    start_date:              20090101
    timeStamp:               2022-Jan-17 19:00:13 GMT
    title:                   ocean T grid variables
    uuid:                    4a651a9f-119a-41ea-a59c-075a9c8a31df
xarray.Dataset
    • t: 31
    • z: 150
    • y: 6540
    • x: 6560
    • time_centered
      (t)
      object
      dask.array<chunksize=(1,), meta=np.ndarray>
      bounds :
      time_centered_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      Array Chunk
      Bytes 248 B 8 B
      Shape (31,) (1,)
      Count 189 Tasks 31 Chunks
      Type object numpy.ndarray
      		31 1
    • t
      (t)
      object
      2012-01-01 12:00:00 ... 2012-01-...
      axis :
      T
      bounds :
      time_counter_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      array([cftime.DatetimeNoLeap(2012, 1, 1, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 2, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 3, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 4, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 5, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 6, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 7, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 8, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 9, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 10, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 11, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 12, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 13, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 14, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 17, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 18, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 19, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 20, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 21, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 22, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 23, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 24, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 25, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 26, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 27, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 28, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 29, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 30, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 31, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
    • y
      (y)
      int64
      1 2 3 4 5 ... 6537 6538 6539 6540
      array([   1,    2,    3, ..., 6538, 6539, 6540])
    • x
      (x)
      int64
      1 2 3 4 5 ... 6557 6558 6559 6560
      array([   1,    2,    3, ..., 6558, 6559, 6560])
    • z
      (z)
      int64
      1 2 3 4 5 6 ... 146 147 148 149 150
      array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,
        15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,  28,
        29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,  42,
        43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,  56,
        57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,  70,
        71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,  84,
        85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
        99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
       113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126,
       127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
       141, 142, 143, 144, 145, 146, 147, 148, 149, 150])
    • nav_lat
      (y, x)
      float32
      dask.array<chunksize=(65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 163.66 MiB 1.63 MiB
      Shape (6540, 6560) (65, 6560)
      Count 1741 Tasks 109 Chunks
      Type float32 numpy.ndarray
      		6560 6540
    • mask2d
      (y, x)
      bool
      dask.array<chunksize=(65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 40.91 MiB 416.41 kiB
      Shape (6540, 6560) (65, 6560)
      Count 1741 Tasks 109 Chunks
      Type bool numpy.ndarray
      		6560 6540
    • depth
      (z, y, x)
      float32
      dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 23.97 GiB 243.99 MiB
      Shape (150, 6540, 6560) (150, 65, 6560)
      Count 1741 Tasks 109 Chunks
      Type float32 numpy.ndarray
      		6560 6540 150
    • mask
      (z, y, x)
      bool
      dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 5.99 GiB 61.00 MiB
      Shape (150, 6540, 6560) (150, 65, 6560)
      Count 1741 Tasks 109 Chunks
      Type bool numpy.ndarray
      		6560 6540 150
    • nav_lon
      (y, x)
      float32
      dask.array<chunksize=(65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 163.66 MiB 1.63 MiB
      Shape (6540, 6560) (65, 6560)
      Count 1741 Tasks 109 Chunks
      Type float32 numpy.ndarray
      		6560 6540
    • votemper
      (t, z, y, x)
      float32
      dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      temperature
      online_operation :
      average
      standard_name :
      sea_water_potential_temperature
      units :
      degC
      Array Chunk
      Bytes 743.18 GiB 243.99 MiB
      Shape (31, 150, 6540, 6560) (1, 150, 65, 6560)
      Count 37651 Tasks 3379 Chunks
      Type float32 numpy.ndarray
      		31 1 6560 6540 150
    • vomecrty
      (t, z, y, x)
      float32
      dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      ocean current along j-axis
      online_operation :
      average
      standard_name :
      sea_water_y_velocity
      units :
      m/s
      Array Chunk
      Bytes 743.18 GiB 243.99 MiB
      Shape (31, 150, 6540, 6560) (1, 150, 65, 6560)
      Count 37651 Tasks 3379 Chunks
      Type float32 numpy.ndarray
      		31 1 6560 6540 150
  • CASE :
    DELTA
    CONFIG :
    SEDNA
    Conventions :
    CF-1.6
    DOMAIN_dimensions_ids :
    [2, 3]
    DOMAIN_halo_size_end :
    [0, 0]
    DOMAIN_halo_size_start :
    [0, 0]
    DOMAIN_number :
    0
    DOMAIN_number_total :
    544
    DOMAIN_position_first :
    [1, 1]
    DOMAIN_position_last :
    [6560, 13]
    DOMAIN_size_global :
    [6560, 6540]
    DOMAIN_size_local :
    [6560, 13]
    DOMAIN_type :
    box
    NCO :
    netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)
    description :
    ocean T grid variables
    history :
    Tue Jan 18 17:26:40 2022: ncks -4 -L 1 SEDNA-DELTA_1d_gridT_201201-201201_NOZIP_0000.nc /ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d/2012/01/SEDNA-DELTA_1d_gridT_201201-201201_0000.nc Tue Jan 18 17:26:25 2022: ncrcat -n 31,2,1 SEDNA-DELTA_1d_gridT_0000_01.nc SEDNA-DELTA_1d_gridT_201201-201201_NOZIP_0000.nc
    ibegin :
    0
    jbegin :
    0
    name :
    /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUNS/SEDNA-DELTA-XIOS.46/SEDNA-DELTA_1d_gridT
    ni :
    6560
    nj :
    13
    output_frequency :
    1d
    start_date :
    20090101
    timeStamp :
    2022-Jan-17 19:00:13 GMT
    title :
    ocean T grid variables
    uuid :
    4a651a9f-119a-41ea-a59c-075a9c8a31df
In [6]:
%%time
monitor.auto(df,data,savefig,daskreport,outputpath,file_exp='SEDNA'
            )

#calc= True
#save= True
#plot= False
Value='Mean Temp & Velocity'
Zone='FramS_Small'
Plot='Mean_temp_velo_integrals'
cmap='None'
clabel='(°C,cm.s^-1)'
clim= ((0, 4), (0, 10))
outputpath='../results/SEDNA_DELTA_MONITOR/'
nc_outputpath='../nc_results/SEDNA_DELTA_MONITOR/'
filename='SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity'
data=monitor.optimize_dataset(data)
#2 Zooming Data
data= zoom.FramS_Small(data)
data=monitor.optimize_dataset(data)

<xarray.Dataset>
Dimensions:        (t: 31, z: 150, x: 601)
Coordinates:
    time_centered  (t) object dask.array<chunksize=(1,), meta=np.ndarray>
  * t              (t) object 2012-01-01 12:00:00 ... 2012-01-31 12:00:00
    y              int64 2609
  * x              (x) int64 3734 3735 3736 3737 3738 ... 4331 4332 4333 4334
  * z              (z) int64 1 2 3 4 5 6 7 8 ... 143 144 145 146 147 148 149 150
    nav_lat        (x) float32 dask.array<chunksize=(601,), meta=np.ndarray>
    mask2d         (x) bool dask.array<chunksize=(601,), meta=np.ndarray>
    depth          (z, x) float32 dask.array<chunksize=(150, 601), meta=np.ndarray>
    mask           (z, x) bool dask.array<chunksize=(150, 601), meta=np.ndarray>
    nav_lon        (x) float32 dask.array<chunksize=(601,), meta=np.ndarray>
Data variables:
    votemper       (t, z, x) float32 dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
    vomecrty       (t, z, x) float32 dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
Attributes: (12/26)
    CASE:                    DELTA
    CONFIG:                  SEDNA
    Conventions:             CF-1.6
    DOMAIN_dimensions_ids:   [2, 3]
    DOMAIN_halo_size_end:    [0, 0]
    DOMAIN_halo_size_start:  [0, 0]
    ...                      ...
    nj:                      13
    output_frequency:        1d
    start_date:              20090101
    timeStamp:               2022-Jan-17 19:00:13 GMT
    title:                   ocean T grid variables
    uuid:                    4a651a9f-119a-41ea-a59c-075a9c8a31df
xarray.Dataset
    • t: 31
    • z: 150
    • x: 601
    • time_centered
      (t)
      object
      dask.array<chunksize=(1,), meta=np.ndarray>
      bounds :
      time_centered_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      Array Chunk
      Bytes 248 B 8 B
      Shape (31,) (1,)
      Count 189 Tasks 31 Chunks
      Type object numpy.ndarray
      		31 1
    • t
      (t)
      object
      2012-01-01 12:00:00 ... 2012-01-...
      axis :
      T
      bounds :
      time_counter_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      array([cftime.DatetimeNoLeap(2012, 1, 1, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 2, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 3, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 4, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 5, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 6, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 7, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 8, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 9, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 10, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 11, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 12, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 13, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 14, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 17, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 18, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 19, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 20, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 21, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 22, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 23, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 24, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 25, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 26, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 27, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 28, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 29, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 30, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 31, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
    • y
      ()
      int64
      2609
      array(2609)
    • x
      (x)
      int64
      3734 3735 3736 ... 4332 4333 4334
      array([3734, 3735, 3736, ..., 4332, 4333, 4334])
    • z
      (z)
      int64
      1 2 3 4 5 6 ... 146 147 148 149 150
      array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,
        15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,  28,
        29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,  42,
        43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,  56,
        57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,  70,
        71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,  84,
        85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
        99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
       113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126,
       127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
       141, 142, 143, 144, 145, 146, 147, 148, 149, 150])
    • nav_lat
      (x)
      float32
      dask.array<chunksize=(601,), meta=np.ndarray>
      Array Chunk
      Bytes 2.35 kiB 2.35 kiB
      Shape (601,) (601,)
      Count 1742 Tasks 1 Chunks
      Type float32 numpy.ndarray
      		601 1
    • mask2d
      (x)
      bool
      dask.array<chunksize=(601,), meta=np.ndarray>
      Array Chunk
      Bytes 601 B 601 B
      Shape (601,) (601,)
      Count 1742 Tasks 1 Chunks
      Type bool numpy.ndarray
      		601 1
    • depth
      (z, x)
      float32
      dask.array<chunksize=(150, 601), meta=np.ndarray>
      Array Chunk
      Bytes 352.15 kiB 352.15 kiB
      Shape (150, 601) (150, 601)
      Count 1742 Tasks 1 Chunks
      Type float32 numpy.ndarray
      		601 150
    • mask
      (z, x)
      bool
      dask.array<chunksize=(150, 601), meta=np.ndarray>
      Array Chunk
      Bytes 88.04 kiB 88.04 kiB
      Shape (150, 601) (150, 601)
      Count 1742 Tasks 1 Chunks
      Type bool numpy.ndarray
      		601 150
    • nav_lon
      (x)
      float32
      dask.array<chunksize=(601,), meta=np.ndarray>
      Array Chunk
      Bytes 2.35 kiB 2.35 kiB
      Shape (601,) (601,)
      Count 1742 Tasks 1 Chunks
      Type float32 numpy.ndarray
      		601 1
    • votemper
      (t, z, x)
      float32
      dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      temperature
      online_operation :
      average
      standard_name :
      sea_water_potential_temperature
      units :
      degC
      Array Chunk
      Bytes 10.66 MiB 352.15 kiB
      Shape (31, 150, 601) (1, 150, 601)
      Count 459 Tasks 31 Chunks
      Type float32 numpy.ndarray
      		601 150 31
    • vomecrty
      (t, z, x)
      float32
      dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      ocean current along j-axis
      online_operation :
      average
      standard_name :
      sea_water_y_velocity
      units :
      m/s
      Array Chunk
      Bytes 10.66 MiB 352.15 kiB
      Shape (31, 150, 601) (1, 150, 601)
      Count 459 Tasks 31 Chunks
      Type float32 numpy.ndarray
      		601 150 31
  • CASE :
    DELTA
    CONFIG :
    SEDNA
    Conventions :
    CF-1.6
    DOMAIN_dimensions_ids :
    [2, 3]
    DOMAIN_halo_size_end :
    [0, 0]
    DOMAIN_halo_size_start :
    [0, 0]
    DOMAIN_number :
    0
    DOMAIN_number_total :
    544
    DOMAIN_position_first :
    [1, 1]
    DOMAIN_position_last :
    [6560, 13]
    DOMAIN_size_global :
    [6560, 6540]
    DOMAIN_size_local :
    [6560, 13]
    DOMAIN_type :
    box
    NCO :
    netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)
    description :
    ocean T grid variables
    history :
    Tue Jan 18 17:26:40 2022: ncks -4 -L 1 SEDNA-DELTA_1d_gridT_201201-201201_NOZIP_0000.nc /ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d/2012/01/SEDNA-DELTA_1d_gridT_201201-201201_0000.nc Tue Jan 18 17:26:25 2022: ncrcat -n 31,2,1 SEDNA-DELTA_1d_gridT_0000_01.nc SEDNA-DELTA_1d_gridT_201201-201201_NOZIP_0000.nc
    ibegin :
    0
    jbegin :
    0
    name :
    /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUNS/SEDNA-DELTA-XIOS.46/SEDNA-DELTA_1d_gridT
    ni :
    6560
    nj :
    13
    output_frequency :
    1d
    start_date :
    20090101
    timeStamp :
    2022-Jan-17 19:00:13 GMT
    title :
    ocean T grid variables
    uuid :
    4a651a9f-119a-41ea-a59c-075a9c8a31df
#3 Start computing 
data= calc.Mean_temp_velo(data)
monitor.optimize_dataset(data)
add optimise here once otimise can recognise

<xarray.Dataset>
Dimensions:          (t: 31)
Coordinates:
    time_centered    (t) object dask.array<chunksize=(1,), meta=np.ndarray>
  * t                (t) object 2012-01-01 12:00:00 ... 2012-01-31 12:00:00
    y                int64 2609
Data variables:
    Mean Tempreture  (t) float32 dask.array<chunksize=(1,), meta=np.ndarray>
    Mean Velocity    (t) float32 dask.array<chunksize=(1,), meta=np.ndarray>
xarray.Dataset
    • t: 31
    • time_centered
      (t)
      object
      dask.array<chunksize=(1,), meta=np.ndarray>
      bounds :
      time_centered_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      Array Chunk
      Bytes 248 B 8 B
      Shape (31,) (1,)
      Count 189 Tasks 31 Chunks
      Type object numpy.ndarray
      		31 1
    • t
      (t)
      object
      2012-01-01 12:00:00 ... 2012-01-...
      axis :
      T
      bounds :
      time_counter_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      array([cftime.DatetimeNoLeap(2012, 1, 1, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 2, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 3, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 4, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 5, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 6, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 7, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 8, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 9, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 10, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 11, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 12, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 13, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 14, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 17, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 18, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 19, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 20, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 21, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 22, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 23, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 24, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 25, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 26, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 27, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 28, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 29, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 30, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 1, 31, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
    • y
      ()
      int64
      2609
      array(2609)
    • Mean Tempreture
      (t)
      float32
      dask.array<chunksize=(1,), meta=np.ndarray>
      Array Chunk
      Bytes 124 B 4 B
      Shape (31,) (1,)
      Count 2295 Tasks 31 Chunks
      Type float32 numpy.ndarray
      		31 1
    • Mean Velocity
      (t)
      float32
      dask.array<chunksize=(1,), meta=np.ndarray>
      Array Chunk
      Bytes 124 B 4 B
      Shape (31,) (1,)
      Count 2326 Tasks 31 Chunks
      Type float32 numpy.ndarray
      		31 1 
#4 Saving  SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity
data=save.datas(data,plot=Plot,path=nc_outputpath,filename=filename)
start saving data
saving data in a  csv file ../nc_results/SEDNA_DELTA_MONITOR/SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity2012-01-01_2012-01-31.nc
save computed data at ../nc_results/SEDNA_DELTA_MONITOR/SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity2012-01-01_2012-01-31.nc completed
CPU times: user 6.49 s, sys: 1.01 s, total: 7.5 s
Wall time: 10.8 s