In [1]:
%matplotlib inline
import pandas as pd
import socket
host = socket.getfqdn()

from core import  load, zoom, calc, save,plots,monitor
In [2]:
#reload funcs after updating ./core/*.py
import importlib
importlib.reload(load)
importlib.reload(zoom)
importlib.reload(calc)
importlib.reload(save)
importlib.reload(plots)
importlib.reload(monitor)
Out[2]:
<module 'core.monitor' from '/ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/core/monitor.py'>

If you submit the job with job scheduler, above¶

below are list of enviroment variable one can pass

%env local='2"¶

local : if True run dask local cluster, if not true, put number of workers setted in the 'local' if no 'local ' given, local will be setted automatically to 'True'

%env ychunk='2'¶

%env tchunk='2'¶

controls chunk. 'False' sets no modification from original netcdf file's chunk.¶

ychunk=10 will group the original netcdf file to 10 by 10¶

tchunk=1 will chunk the time coordinate one by one¶

%env file_exp=¶

'file_exp': Which 'experiment' name is it?¶

. this corresopnds to intake catalog name without path and .yaml¶

%env year=¶

for Validation, this correspoinds to path/year/month 's year¶

for monitoring, this corresponids to 'date' having * means do all files in the monitoring directory¶

setting it as 0[0-9] &1[0-9]& *[2-3][0-9], the job can be separated in three lots.¶

%env month=¶

for monitoring this corresponds to file path path-XIOS.{month}/¶

#

%env control=FWC_SSH¶

name of control file to be used for computation/plots/save/ & how it is called from Monitor.sh¶

Monitor.sh calls M_MLD_2D

and AWTD.sh, Fluxnet.sh, Siconc.sh, IceClim.sh, FWC_SSH.sh

  • AWTD.sh M_AWTMD

  • Fluxnet.sh M_Fluxnet

  • Siconc.sh M_Ice_quantities
  • IceClim.sh M_IceClim M_IceConce M_IceThick

FWC_SSH.sh M_FWC_2D M_FWC_integrals M_FWC_SSH M_SSH_anomaly

Integrals.sh M_Mean_temp_velo M_Mooring M_Sectionx M_Sectiony

%env save= proceed saving? True or False , Default is setted as True¶

%env plot= proceed plotting? True or False , Default is setted as True¶

%env calc= proceed computation? or just load computed result? True or False , Default is setted as True¶

%env save=False¶

%env lazy=False¶

For debugging this cell can help¶

%env file_exp=SEDNA_DELTA_MONITOR %env year=2012 %env month=01

0[1-2]¶

%env ychunk=10 %env ychunk=False %env save=False %env plot=True %env calc=True # %env lazy=False

False¶

%env control=M_Fluxnet

M_Sectiony ok with ychunk=False local=True lazy=False¶

In [3]:
%%time
# 'savefig': Do we save output in html? or not. keep it true. 
savefig=True
client,cluster,control,catalog_url,month,year,daskreport,outputpath = load.set_control(host)
!mkdir -p $outputpath
!mkdir -p $daskreport
client

local True
using host= irene5424.c-irene.mg1.tgcc.ccc.cea.fr starting dask cluster on local= True workers 16
10000000000
False
tgcc local cluster starting
This code is running on  irene5424.c-irene.mg1.tgcc.ccc.cea.fr using  SEDNA_DELTA_MONITOR file experiment, read from  ../lib/SEDNA_DELTA_MONITOR.yaml  on year= 2012  on month= 02  outputpath= ../results/SEDNA_DELTA_MONITOR/ daskreport= ../results/dask/6419277irene5424.c-irene.mg1.tgcc.ccc.cea.fr_SEDNA_DELTA_MONITOR_02M_Mean_temp_velo/
CPU times: user 3.58 s, sys: 706 ms, total: 4.29 s
Wall time: 1min 39s
Out[3]:

Client

Client-bf04666e-13da-11ed-8598-080038b94031

Connection method: Cluster object Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

fa6b85fd

Dashboard: http://127.0.0.1:8787/status Workers: 64
Total threads: 256 Total memory: 251.06 GiB
Status: running Using processes: True

Scheduler Info

Scheduler

Scheduler-0621d0ef-bd69-4fbe-894d-90787f9f292a

Comm: tcp://127.0.0.1:40067 Workers: 64
Dashboard: http://127.0.0.1:8787/status Total threads: 256
Started: 1 minute ago Total memory: 251.06 GiB

Workers

Worker: 0
Comm: tcp://127.0.0.1:36465 Total threads: 4
Dashboard: http://127.0.0.1:43825/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43240
Local directory: /tmp/dask-worker-space/worker-h9n3yvj6

Worker: 1
Comm: tcp://127.0.0.1:42715 Total threads: 4
Dashboard: http://127.0.0.1:41008/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41509
Local directory: /tmp/dask-worker-space/worker-ke_cr937

Worker: 2
Comm: tcp://127.0.0.1:42504 Total threads: 4
Dashboard: http://127.0.0.1:40246/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45218
Local directory: /tmp/dask-worker-space/worker-51v5lqr9

Worker: 3
Comm: tcp://127.0.0.1:45844 Total threads: 4
Dashboard: http://127.0.0.1:39886/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38163
Local directory: /tmp/dask-worker-space/worker-1on2abf9

Worker: 4
Comm: tcp://127.0.0.1:33381 Total threads: 4
Dashboard: http://127.0.0.1:34201/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36391
Local directory: /tmp/dask-worker-space/worker-dtft71i0

Worker: 5
Comm: tcp://127.0.0.1:44551 Total threads: 4
Dashboard: http://127.0.0.1:34628/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43001
Local directory: /tmp/dask-worker-space/worker-ibbtcjvk

Worker: 6
Comm: tcp://127.0.0.1:34889 Total threads: 4
Dashboard: http://127.0.0.1:35568/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35555
Local directory: /tmp/dask-worker-space/worker-fb8pklht

Worker: 7
Comm: tcp://127.0.0.1:35867 Total threads: 4
Dashboard: http://127.0.0.1:39734/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43640
Local directory: /tmp/dask-worker-space/worker-l1etuopf

Worker: 8
Comm: tcp://127.0.0.1:34578 Total threads: 4
Dashboard: http://127.0.0.1:39964/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37268
Local directory: /tmp/dask-worker-space/worker-49sosyw7

Worker: 9
Comm: tcp://127.0.0.1:44052 Total threads: 4
Dashboard: http://127.0.0.1:43267/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39110
Local directory: /tmp/dask-worker-space/worker-ar5io9mh

Worker: 10
Comm: tcp://127.0.0.1:36923 Total threads: 4
Dashboard: http://127.0.0.1:42385/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44644
Local directory: /tmp/dask-worker-space/worker-0lm1nmq1

Worker: 11
Comm: tcp://127.0.0.1:33646 Total threads: 4
Dashboard: http://127.0.0.1:35235/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34980
Local directory: /tmp/dask-worker-space/worker-3fvo_i5m

Worker: 12
Comm: tcp://127.0.0.1:41868 Total threads: 4
Dashboard: http://127.0.0.1:33618/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44164
Local directory: /tmp/dask-worker-space/worker-ki3uzlor

Worker: 13
Comm: tcp://127.0.0.1:35640 Total threads: 4
Dashboard: http://127.0.0.1:34079/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35063
Local directory: /tmp/dask-worker-space/worker-p9fhcyzd

Worker: 14
Comm: tcp://127.0.0.1:44699 Total threads: 4
Dashboard: http://127.0.0.1:35227/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37766
Local directory: /tmp/dask-worker-space/worker-_l4ct9af

Worker: 15
Comm: tcp://127.0.0.1:43542 Total threads: 4
Dashboard: http://127.0.0.1:44828/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41992
Local directory: /tmp/dask-worker-space/worker-g954rjci

Worker: 16
Comm: tcp://127.0.0.1:33167 Total threads: 4
Dashboard: http://127.0.0.1:45199/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36714
Local directory: /tmp/dask-worker-space/worker-o0s4ga6j

Worker: 17
Comm: tcp://127.0.0.1:42018 Total threads: 4
Dashboard: http://127.0.0.1:42878/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39062
Local directory: /tmp/dask-worker-space/worker-oux2a_28

Worker: 18
Comm: tcp://127.0.0.1:36481 Total threads: 4
Dashboard: http://127.0.0.1:36792/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38712
Local directory: /tmp/dask-worker-space/worker-bz3m_fic

Worker: 19
Comm: tcp://127.0.0.1:32897 Total threads: 4
Dashboard: http://127.0.0.1:44796/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39992
Local directory: /tmp/dask-worker-space/worker-7d2hi8cl

Worker: 20
Comm: tcp://127.0.0.1:35201 Total threads: 4
Dashboard: http://127.0.0.1:33734/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45336
Local directory: /tmp/dask-worker-space/worker-bsuajgj_

Worker: 21
Comm: tcp://127.0.0.1:41220 Total threads: 4
Dashboard: http://127.0.0.1:34202/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43664
Local directory: /tmp/dask-worker-space/worker-pc_ozsml

Worker: 22
Comm: tcp://127.0.0.1:41455 Total threads: 4
Dashboard: http://127.0.0.1:46522/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36117
Local directory: /tmp/dask-worker-space/worker-8v1b39fz

Worker: 23
Comm: tcp://127.0.0.1:42437 Total threads: 4
Dashboard: http://127.0.0.1:34847/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35088
Local directory: /tmp/dask-worker-space/worker-u6avpfq1

Worker: 24
Comm: tcp://127.0.0.1:43446 Total threads: 4
Dashboard: http://127.0.0.1:43225/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:38229
Local directory: /tmp/dask-worker-space/worker-kvtjs2dd

Worker: 25
Comm: tcp://127.0.0.1:43438 Total threads: 4
Dashboard: http://127.0.0.1:35360/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39432
Local directory: /tmp/dask-worker-space/worker-nvuj4gfv

Worker: 26
Comm: tcp://127.0.0.1:36805 Total threads: 4
Dashboard: http://127.0.0.1:38970/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43283
Local directory: /tmp/dask-worker-space/worker-u5bw8bpl

Worker: 27
Comm: tcp://127.0.0.1:45929 Total threads: 4
Dashboard: http://127.0.0.1:34311/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40202
Local directory: /tmp/dask-worker-space/worker-kvpsyg8g

Worker: 28
Comm: tcp://127.0.0.1:43477 Total threads: 4
Dashboard: http://127.0.0.1:33056/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39155
Local directory: /tmp/dask-worker-space/worker-n4_h2l4a

Worker: 29
Comm: tcp://127.0.0.1:45583 Total threads: 4
Dashboard: http://127.0.0.1:37750/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34421
Local directory: /tmp/dask-worker-space/worker-qjx_rpqq

Worker: 30
Comm: tcp://127.0.0.1:36357 Total threads: 4
Dashboard: http://127.0.0.1:46066/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:45025
Local directory: /tmp/dask-worker-space/worker-7vc_e1w1

Worker: 31
Comm: tcp://127.0.0.1:35737 Total threads: 4
Dashboard: http://127.0.0.1:43250/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36066
Local directory: /tmp/dask-worker-space/worker-gheg4qmq

Worker: 32
Comm: tcp://127.0.0.1:34823 Total threads: 4
Dashboard: http://127.0.0.1:33891/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41901
Local directory: /tmp/dask-worker-space/worker-z663ed3v

Worker: 33
Comm: tcp://127.0.0.1:36887 Total threads: 4
Dashboard: http://127.0.0.1:37269/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44436
Local directory: /tmp/dask-worker-space/worker-kauvt96q

Worker: 34
Comm: tcp://127.0.0.1:43486 Total threads: 4
Dashboard: http://127.0.0.1:38798/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41151
Local directory: /tmp/dask-worker-space/worker-4a092fpc

Worker: 35
Comm: tcp://127.0.0.1:38329 Total threads: 4
Dashboard: http://127.0.0.1:33631/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:33415
Local directory: /tmp/dask-worker-space/worker-1y0b2q7u

Worker: 36
Comm: tcp://127.0.0.1:45581 Total threads: 4
Dashboard: http://127.0.0.1:37764/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40727
Local directory: /tmp/dask-worker-space/worker-8jtwlvov

Worker: 37
Comm: tcp://127.0.0.1:34200 Total threads: 4
Dashboard: http://127.0.0.1:40790/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35309
Local directory: /tmp/dask-worker-space/worker-gyezoacp

Worker: 38
Comm: tcp://127.0.0.1:42623 Total threads: 4
Dashboard: http://127.0.0.1:35097/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36218
Local directory: /tmp/dask-worker-space/worker-66hljsn0

Worker: 39
Comm: tcp://127.0.0.1:38286 Total threads: 4
Dashboard: http://127.0.0.1:36312/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41014
Local directory: /tmp/dask-worker-space/worker-je5j71x2

Worker: 40
Comm: tcp://127.0.0.1:37997 Total threads: 4
Dashboard: http://127.0.0.1:41708/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37449
Local directory: /tmp/dask-worker-space/worker-esvz9znu

Worker: 41
Comm: tcp://127.0.0.1:33762 Total threads: 4
Dashboard: http://127.0.0.1:44490/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44449
Local directory: /tmp/dask-worker-space/worker-szcs8g3a

Worker: 42
Comm: tcp://127.0.0.1:33253 Total threads: 4
Dashboard: http://127.0.0.1:38164/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42348
Local directory: /tmp/dask-worker-space/worker-od375khn

Worker: 43
Comm: tcp://127.0.0.1:38352 Total threads: 4
Dashboard: http://127.0.0.1:43107/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41836
Local directory: /tmp/dask-worker-space/worker-rmxfdasa

Worker: 44
Comm: tcp://127.0.0.1:44732 Total threads: 4
Dashboard: http://127.0.0.1:36565/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44867
Local directory: /tmp/dask-worker-space/worker-z730qc7u

Worker: 45
Comm: tcp://127.0.0.1:42121 Total threads: 4
Dashboard: http://127.0.0.1:35387/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43737
Local directory: /tmp/dask-worker-space/worker-thd__mh9

Worker: 46
Comm: tcp://127.0.0.1:36003 Total threads: 4
Dashboard: http://127.0.0.1:36041/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:37250
Local directory: /tmp/dask-worker-space/worker-052im4hb

Worker: 47
Comm: tcp://127.0.0.1:36800 Total threads: 4
Dashboard: http://127.0.0.1:36496/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:44556
Local directory: /tmp/dask-worker-space/worker-xrk9vmio

Worker: 48
Comm: tcp://127.0.0.1:41949 Total threads: 4
Dashboard: http://127.0.0.1:35707/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40963
Local directory: /tmp/dask-worker-space/worker-zt8afflg

Worker: 49
Comm: tcp://127.0.0.1:40184 Total threads: 4
Dashboard: http://127.0.0.1:43537/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42671
Local directory: /tmp/dask-worker-space/worker-8_mivvsp

Worker: 50
Comm: tcp://127.0.0.1:36500 Total threads: 4
Dashboard: http://127.0.0.1:33377/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35615
Local directory: /tmp/dask-worker-space/worker-xbutyv77

Worker: 51
Comm: tcp://127.0.0.1:45258 Total threads: 4
Dashboard: http://127.0.0.1:43786/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:39831
Local directory: /tmp/dask-worker-space/worker-7y2mmhzp

Worker: 52
Comm: tcp://127.0.0.1:36045 Total threads: 4
Dashboard: http://127.0.0.1:40713/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:42908
Local directory: /tmp/dask-worker-space/worker-5li9q766

Worker: 53
Comm: tcp://127.0.0.1:35605 Total threads: 4
Dashboard: http://127.0.0.1:44819/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36605
Local directory: /tmp/dask-worker-space/worker-d0lnsom6

Worker: 54
Comm: tcp://127.0.0.1:44560 Total threads: 4
Dashboard: http://127.0.0.1:38837/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41506
Local directory: /tmp/dask-worker-space/worker-lpsl0adu

Worker: 55
Comm: tcp://127.0.0.1:38316 Total threads: 4
Dashboard: http://127.0.0.1:33362/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:40319
Local directory: /tmp/dask-worker-space/worker-nktyeaas

Worker: 56
Comm: tcp://127.0.0.1:45312 Total threads: 4
Dashboard: http://127.0.0.1:46337/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:36699
Local directory: /tmp/dask-worker-space/worker-83epjmhm

Worker: 57
Comm: tcp://127.0.0.1:34943 Total threads: 4
Dashboard: http://127.0.0.1:40897/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:43028
Local directory: /tmp/dask-worker-space/worker-24uey8f7

Worker: 58
Comm: tcp://127.0.0.1:34607 Total threads: 4
Dashboard: http://127.0.0.1:44669/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41362
Local directory: /tmp/dask-worker-space/worker-5uzmf_9z

Worker: 59
Comm: tcp://127.0.0.1:37812 Total threads: 4
Dashboard: http://127.0.0.1:41416/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:46864
Local directory: /tmp/dask-worker-space/worker-sgcow18i

Worker: 60
Comm: tcp://127.0.0.1:33939 Total threads: 4
Dashboard: http://127.0.0.1:43064/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34289
Local directory: /tmp/dask-worker-space/worker-ihqe84_j

Worker: 61
Comm: tcp://127.0.0.1:40976 Total threads: 4
Dashboard: http://127.0.0.1:39402/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:41264
Local directory: /tmp/dask-worker-space/worker-ql9fxzzk

Worker: 62
Comm: tcp://127.0.0.1:37659 Total threads: 4
Dashboard: http://127.0.0.1:42993/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:34640
Local directory: /tmp/dask-worker-space/worker-hkguth5x

Worker: 63
Comm: tcp://127.0.0.1:44388 Total threads: 4
Dashboard: http://127.0.0.1:42788/status Memory: 3.92 GiB
Nanny: tcp://127.0.0.1:35951
Local directory: /tmp/dask-worker-space/worker-r_h_pi0c

read plotting information from a csv file¶

In [4]:
df=load.controlfile(control)
#Take out 'later' tagged computations
#df=df[~df['Value'].str.contains('later')]
df
Out[4]:
Value Inputs Equation Zone Plot Colourmap MinMax Unit Oldname Unnamed: 10
Mean Temp & Velocity gridV.vomecrty,gridT.votemper,param.mask,param... calc.Mean_temp_velo(data) FramS_Small Mean_temp_velo_integrals None ((0,4),(0,10)) (°C,cm.s^-1) I-5

Computation starts here¶

Each computation consists of

  1. Load NEMO data set
  2. Zoom data set
  3. Compute (or load computed data set)
  4. Save
  5. Plot
  6. Close
In [5]:
%%time
import os
calcswitch=os.environ.get('calc', 'True') 
lazy=os.environ.get('lazy','False' )
loaddata=((df.Inputs != '').any()) 
print('calcswitch=',calcswitch,'df.Inputs != nothing',loaddata, 'lazy=',lazy)
data = load.datas(catalog_url,df.Inputs,month,year,daskreport,lazy=lazy) if ((calcswitch=='True' )*loaddata) else 0 
data

calcswitch= True df.Inputs != nothing True lazy= False
../lib/SEDNA_DELTA_MONITOR.yaml
using param_xios reading  ../lib/SEDNA_DELTA_MONITOR.yaml
using param_xios reading  <bound method DataSourceBase.describe of sources:
  param_xios:
    args:
      combine: nested
      concat_dim: y
      urlpath: /ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc
      xarray_kwargs:
        compat: override
        coords: minimal
        data_vars: minimal
        parallel: true
    description: SEDNA NEMO parameters from MPI output  nav_lon lat fails
    driver: intake_xarray.netcdf.NetCDFSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/
>
{'name': 'param_xios', 'container': 'xarray', 'plugin': ['netcdf'], 'driver': ['netcdf'], 'description': 'SEDNA NEMO parameters from MPI output  nav_lon lat fails', 'direct_access': 'forbid', 'user_parameters': [{'name': 'path', 'description': 'file coordinate', 'type': 'str', 'default': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/MESH/SEDNA_mesh_mask_Tgt_20210423_tsh10m_L1/param'}], 'metadata': {}, 'args': {'urlpath': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc', 'combine': 'nested', 'concat_dim': 'y'}}
0 read gridT ['votemper']
lazy= False
using load_data_xios_kerchunk reading  gridT
using load_data_xios_kerchunk reading  <bound method DataSourceBase.describe of sources:
  data_xios_kerchunk:
    args:
      consolidated: false
      storage_options:
        fo: file:////ccc/cont003/home/ra5563/ra5563/catalogue/DELTA/201202/gridT_0[0-5][0-9][0-9].json
        target_protocol: file
      urlpath: reference://
    description: CREG025 NEMO outputs from different xios server in kerchunk format
    driver: intake_xarray.xzarr.ZarrSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/
>
      took 24.443889617919922 seconds
0 merging gridT ['votemper']
1 read gridV ['vomecrty']
lazy= False
using load_data_xios_kerchunk reading  gridV
using load_data_xios_kerchunk reading  <bound method DataSourceBase.describe of sources:
  data_xios_kerchunk:
    args:
      consolidated: false
      storage_options:
        fo: file:////ccc/cont003/home/ra5563/ra5563/catalogue/DELTA/201202/gridV_0[0-5][0-9][0-9].json
        target_protocol: file
      urlpath: reference://
    description: CREG025 NEMO outputs from different xios server in kerchunk format
    driver: intake_xarray.xzarr.ZarrSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/
>
      took 22.58785080909729 seconds
1 merging gridV ['vomecrty']
      took 0.8485901355743408 seconds
param depth will be included in data
param nav_lat will be included in data
param mask2d will be included in data
param mask will be included in data
param nav_lon will be included in data
ychunk= 5 calldatas_y_rechunk
sum_num (13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12)
start rechunking with (65, 65, 62, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 48)
end of y_rechunk
before rechunking t item (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
start rechunking t with 1
end of t_rechunk
CPU times: user 49.1 s, sys: 13.3 s, total: 1min 2s
Wall time: 1min 45s
Out[5]:
<xarray.Dataset>
Dimensions:        (t: 28, z: 150, y: 6540, x: 6560)
Coordinates:
    time_centered  (t) object dask.array<chunksize=(1,), meta=np.ndarray>
  * t              (t) object 2012-02-01 12:00:00 ... 2012-02-28 12:00:00
  * y              (y) int64 1 2 3 4 5 6 7 ... 6535 6536 6537 6538 6539 6540
  * x              (x) int64 1 2 3 4 5 6 7 ... 6555 6556 6557 6558 6559 6560
  * z              (z) int64 1 2 3 4 5 6 7 8 ... 143 144 145 146 147 148 149 150
    depth          (z, y, x) float32 dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
    nav_lat        (y, x) float32 dask.array<chunksize=(65, 6560), meta=np.ndarray>
    mask2d         (y, x) bool dask.array<chunksize=(65, 6560), meta=np.ndarray>
    mask           (z, y, x) bool dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
    nav_lon        (y, x) float32 dask.array<chunksize=(65, 6560), meta=np.ndarray>
Data variables:
    votemper       (t, z, y, x) float32 dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
    vomecrty       (t, z, y, x) float32 dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
Attributes: (12/26)
    CASE:                    DELTA
    CONFIG:                  SEDNA
    Conventions:             CF-1.6
    DOMAIN_dimensions_ids:   [2, 3]
    DOMAIN_halo_size_end:    [0, 0]
    DOMAIN_halo_size_start:  [0, 0]
    ...                      ...
    nj:                      13
    output_frequency:        1d
    start_date:              20090101
    timeStamp:               2022-Jan-18 16:51:23 GMT
    title:                   ocean T grid variables
    uuid:                    1d86c3a3-deb0-4097-82f9-bacf8b39e958
xarray.Dataset
    • t: 28
    • z: 150
    • y: 6540
    • x: 6560
    • time_centered
      (t)
      object
      dask.array<chunksize=(1,), meta=np.ndarray>
      bounds :
      time_centered_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      Array Chunk
      Bytes 224 B 8 B
      Shape (28,) (1,)
      Count 171 Tasks 28 Chunks
      Type object numpy.ndarray
      		28 1
    • t
      (t)
      object
      2012-02-01 12:00:00 ... 2012-02-...
      axis :
      T
      bounds :
      time_counter_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      array([cftime.DatetimeNoLeap(2012, 2, 1, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 2, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 3, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 4, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 5, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 6, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 7, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 8, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 9, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 10, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 11, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 12, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 13, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 14, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 17, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 18, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 19, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 20, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 21, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 22, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 23, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 24, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 25, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 26, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 27, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 28, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
    • y
      (y)
      int64
      1 2 3 4 5 ... 6537 6538 6539 6540
      array([   1,    2,    3, ..., 6538, 6539, 6540])
    • x
      (x)
      int64
      1 2 3 4 5 ... 6557 6558 6559 6560
      array([   1,    2,    3, ..., 6558, 6559, 6560])
    • z
      (z)
      int64
      1 2 3 4 5 6 ... 146 147 148 149 150
      array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,
        15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,  28,
        29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,  42,
        43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,  56,
        57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,  70,
        71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,  84,
        85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
        99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
       113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126,
       127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
       141, 142, 143, 144, 145, 146, 147, 148, 149, 150])
    • depth
      (z, y, x)
      float32
      dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 23.97 GiB 243.99 MiB
      Shape (150, 6540, 6560) (150, 65, 6560)
      Count 1741 Tasks 109 Chunks
      Type float32 numpy.ndarray
      		6560 6540 150
    • nav_lat
      (y, x)
      float32
      dask.array<chunksize=(65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 163.66 MiB 1.63 MiB
      Shape (6540, 6560) (65, 6560)
      Count 1741 Tasks 109 Chunks
      Type float32 numpy.ndarray
      		6560 6540
    • mask2d
      (y, x)
      bool
      dask.array<chunksize=(65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 40.91 MiB 416.41 kiB
      Shape (6540, 6560) (65, 6560)
      Count 1741 Tasks 109 Chunks
      Type bool numpy.ndarray
      		6560 6540
    • mask
      (z, y, x)
      bool
      dask.array<chunksize=(150, 65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 5.99 GiB 61.00 MiB
      Shape (150, 6540, 6560) (150, 65, 6560)
      Count 1741 Tasks 109 Chunks
      Type bool numpy.ndarray
      		6560 6540 150
    • nav_lon
      (y, x)
      float32
      dask.array<chunksize=(65, 6560), meta=np.ndarray>
      Array Chunk
      Bytes 163.66 MiB 1.63 MiB
      Shape (6540, 6560) (65, 6560)
      Count 1741 Tasks 109 Chunks
      Type float32 numpy.ndarray
      		6560 6540
    • votemper
      (t, z, y, x)
      float32
      dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      temperature
      online_operation :
      average
      standard_name :
      sea_water_potential_temperature
      units :
      degC
      Array Chunk
      Bytes 671.26 GiB 243.99 MiB
      Shape (28, 150, 6540, 6560) (1, 150, 65, 6560)
      Count 34060 Tasks 3052 Chunks
      Type float32 numpy.ndarray
      		28 1 6560 6540 150
    • vomecrty
      (t, z, y, x)
      float32
      dask.array<chunksize=(1, 150, 65, 6560), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      ocean current along j-axis
      online_operation :
      average
      standard_name :
      sea_water_y_velocity
      units :
      m/s
      Array Chunk
      Bytes 671.26 GiB 243.99 MiB
      Shape (28, 150, 6540, 6560) (1, 150, 65, 6560)
      Count 34060 Tasks 3052 Chunks
      Type float32 numpy.ndarray
      		28 1 6560 6540 150
  • CASE :
    DELTA
    CONFIG :
    SEDNA
    Conventions :
    CF-1.6
    DOMAIN_dimensions_ids :
    [2, 3]
    DOMAIN_halo_size_end :
    [0, 0]
    DOMAIN_halo_size_start :
    [0, 0]
    DOMAIN_number :
    0
    DOMAIN_number_total :
    544
    DOMAIN_position_first :
    [1, 1]
    DOMAIN_position_last :
    [6560, 13]
    DOMAIN_size_global :
    [6560, 6540]
    DOMAIN_size_local :
    [6560, 13]
    DOMAIN_type :
    box
    NCO :
    netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)
    description :
    ocean T grid variables
    history :
    Wed Jan 19 12:40:53 2022: ncks -4 -L 1 SEDNA-DELTA_1d_gridT_201202-201202_NOZIP_0000.nc /ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d/2012/02/SEDNA-DELTA_1d_gridT_201202-201202_0000.nc Wed Jan 19 12:40:32 2022: ncrcat -n 28,2,1 SEDNA-DELTA_1d_gridT_0000_01.nc SEDNA-DELTA_1d_gridT_201202-201202_NOZIP_0000.nc
    ibegin :
    0
    jbegin :
    0
    name :
    /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUNS/SEDNA-DELTA-XIOS.47/SEDNA-DELTA_1d_gridT
    ni :
    6560
    nj :
    13
    output_frequency :
    1d
    start_date :
    20090101
    timeStamp :
    2022-Jan-18 16:51:23 GMT
    title :
    ocean T grid variables
    uuid :
    1d86c3a3-deb0-4097-82f9-bacf8b39e958
In [6]:
%%time
monitor.auto(df,data,savefig,daskreport,outputpath,file_exp='SEDNA'
            )

#calc= True
#save= True
#plot= False
Value='Mean Temp & Velocity'
Zone='FramS_Small'
Plot='Mean_temp_velo_integrals'
cmap='None'
clabel='(°C,cm.s^-1)'
clim= ((0, 4), (0, 10))
outputpath='../results/SEDNA_DELTA_MONITOR/'
nc_outputpath='../nc_results/SEDNA_DELTA_MONITOR/'
filename='SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity'
data=monitor.optimize_dataset(data)
#2 Zooming Data
data= zoom.FramS_Small(data)
data=monitor.optimize_dataset(data)

<xarray.Dataset>
Dimensions:        (t: 28, z: 150, x: 601)
Coordinates:
    time_centered  (t) object dask.array<chunksize=(1,), meta=np.ndarray>
  * t              (t) object 2012-02-01 12:00:00 ... 2012-02-28 12:00:00
    y              int64 2609
  * x              (x) int64 3734 3735 3736 3737 3738 ... 4331 4332 4333 4334
  * z              (z) int64 1 2 3 4 5 6 7 8 ... 143 144 145 146 147 148 149 150
    depth          (z, x) float32 dask.array<chunksize=(150, 601), meta=np.ndarray>
    nav_lat        (x) float32 dask.array<chunksize=(601,), meta=np.ndarray>
    mask2d         (x) bool dask.array<chunksize=(601,), meta=np.ndarray>
    mask           (z, x) bool dask.array<chunksize=(150, 601), meta=np.ndarray>
    nav_lon        (x) float32 dask.array<chunksize=(601,), meta=np.ndarray>
Data variables:
    votemper       (t, z, x) float32 dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
    vomecrty       (t, z, x) float32 dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
Attributes: (12/26)
    CASE:                    DELTA
    CONFIG:                  SEDNA
    Conventions:             CF-1.6
    DOMAIN_dimensions_ids:   [2, 3]
    DOMAIN_halo_size_end:    [0, 0]
    DOMAIN_halo_size_start:  [0, 0]
    ...                      ...
    nj:                      13
    output_frequency:        1d
    start_date:              20090101
    timeStamp:               2022-Jan-18 16:51:23 GMT
    title:                   ocean T grid variables
    uuid:                    1d86c3a3-deb0-4097-82f9-bacf8b39e958
xarray.Dataset
    • t: 28
    • z: 150
    • x: 601
    • time_centered
      (t)
      object
      dask.array<chunksize=(1,), meta=np.ndarray>
      bounds :
      time_centered_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      Array Chunk
      Bytes 224 B 8 B
      Shape (28,) (1,)
      Count 171 Tasks 28 Chunks
      Type object numpy.ndarray
      		28 1
    • t
      (t)
      object
      2012-02-01 12:00:00 ... 2012-02-...
      axis :
      T
      bounds :
      time_counter_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      array([cftime.DatetimeNoLeap(2012, 2, 1, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 2, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 3, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 4, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 5, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 6, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 7, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 8, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 9, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 10, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 11, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 12, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 13, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 14, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 17, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 18, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 19, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 20, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 21, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 22, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 23, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 24, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 25, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 26, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 27, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 28, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
    • y
      ()
      int64
      2609
      array(2609)
    • x
      (x)
      int64
      3734 3735 3736 ... 4332 4333 4334
      array([3734, 3735, 3736, ..., 4332, 4333, 4334])
    • z
      (z)
      int64
      1 2 3 4 5 6 ... 146 147 148 149 150
      array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,  14,
        15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,  27,  28,
        29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,  40,  41,  42,
        43,  44,  45,  46,  47,  48,  49,  50,  51,  52,  53,  54,  55,  56,
        57,  58,  59,  60,  61,  62,  63,  64,  65,  66,  67,  68,  69,  70,
        71,  72,  73,  74,  75,  76,  77,  78,  79,  80,  81,  82,  83,  84,
        85,  86,  87,  88,  89,  90,  91,  92,  93,  94,  95,  96,  97,  98,
        99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112,
       113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126,
       127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
       141, 142, 143, 144, 145, 146, 147, 148, 149, 150])
    • depth
      (z, x)
      float32
      dask.array<chunksize=(150, 601), meta=np.ndarray>
      Array Chunk
      Bytes 352.15 kiB 352.15 kiB
      Shape (150, 601) (150, 601)
      Count 1742 Tasks 1 Chunks
      Type float32 numpy.ndarray
      		601 150
    • nav_lat
      (x)
      float32
      dask.array<chunksize=(601,), meta=np.ndarray>
      Array Chunk
      Bytes 2.35 kiB 2.35 kiB
      Shape (601,) (601,)
      Count 1742 Tasks 1 Chunks
      Type float32 numpy.ndarray
      		601 1
    • mask2d
      (x)
      bool
      dask.array<chunksize=(601,), meta=np.ndarray>
      Array Chunk
      Bytes 601 B 601 B
      Shape (601,) (601,)
      Count 1742 Tasks 1 Chunks
      Type bool numpy.ndarray
      		601 1
    • mask
      (z, x)
      bool
      dask.array<chunksize=(150, 601), meta=np.ndarray>
      Array Chunk
      Bytes 88.04 kiB 88.04 kiB
      Shape (150, 601) (150, 601)
      Count 1742 Tasks 1 Chunks
      Type bool numpy.ndarray
      		601 150
    • nav_lon
      (x)
      float32
      dask.array<chunksize=(601,), meta=np.ndarray>
      Array Chunk
      Bytes 2.35 kiB 2.35 kiB
      Shape (601,) (601,)
      Count 1742 Tasks 1 Chunks
      Type float32 numpy.ndarray
      		601 1
    • votemper
      (t, z, x)
      float32
      dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      temperature
      online_operation :
      average
      standard_name :
      sea_water_potential_temperature
      units :
      degC
      Array Chunk
      Bytes 9.63 MiB 352.15 kiB
      Shape (28, 150, 601) (1, 150, 601)
      Count 420 Tasks 28 Chunks
      Type float32 numpy.ndarray
      		601 150 28
    • vomecrty
      (t, z, x)
      float32
      dask.array<chunksize=(1, 150, 601), meta=np.ndarray>
      cell_methods :
      time: mean (interval: 40 s)
      interval_operation :
      40 s
      interval_write :
      1 d
      long_name :
      ocean current along j-axis
      online_operation :
      average
      standard_name :
      sea_water_y_velocity
      units :
      m/s
      Array Chunk
      Bytes 9.63 MiB 352.15 kiB
      Shape (28, 150, 601) (1, 150, 601)
      Count 420 Tasks 28 Chunks
      Type float32 numpy.ndarray
      		601 150 28
  • CASE :
    DELTA
    CONFIG :
    SEDNA
    Conventions :
    CF-1.6
    DOMAIN_dimensions_ids :
    [2, 3]
    DOMAIN_halo_size_end :
    [0, 0]
    DOMAIN_halo_size_start :
    [0, 0]
    DOMAIN_number :
    0
    DOMAIN_number_total :
    544
    DOMAIN_position_first :
    [1, 1]
    DOMAIN_position_last :
    [6560, 13]
    DOMAIN_size_global :
    [6560, 6540]
    DOMAIN_size_local :
    [6560, 13]
    DOMAIN_type :
    box
    NCO :
    netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)
    description :
    ocean T grid variables
    history :
    Wed Jan 19 12:40:53 2022: ncks -4 -L 1 SEDNA-DELTA_1d_gridT_201202-201202_NOZIP_0000.nc /ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d/2012/02/SEDNA-DELTA_1d_gridT_201202-201202_0000.nc Wed Jan 19 12:40:32 2022: ncrcat -n 28,2,1 SEDNA-DELTA_1d_gridT_0000_01.nc SEDNA-DELTA_1d_gridT_201202-201202_NOZIP_0000.nc
    ibegin :
    0
    jbegin :
    0
    name :
    /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUNS/SEDNA-DELTA-XIOS.47/SEDNA-DELTA_1d_gridT
    ni :
    6560
    nj :
    13
    output_frequency :
    1d
    start_date :
    20090101
    timeStamp :
    2022-Jan-18 16:51:23 GMT
    title :
    ocean T grid variables
    uuid :
    1d86c3a3-deb0-4097-82f9-bacf8b39e958
#3 Start computing 
data= calc.Mean_temp_velo(data)
monitor.optimize_dataset(data)
add optimise here once otimise can recognise

<xarray.Dataset>
Dimensions:          (t: 28)
Coordinates:
    time_centered    (t) object dask.array<chunksize=(1,), meta=np.ndarray>
  * t                (t) object 2012-02-01 12:00:00 ... 2012-02-28 12:00:00
    y                int64 2609
Data variables:
    Mean Tempreture  (t) float32 dask.array<chunksize=(1,), meta=np.ndarray>
    Mean Velocity    (t) float32 dask.array<chunksize=(1,), meta=np.ndarray>
xarray.Dataset
    • t: 28
    • time_centered
      (t)
      object
      dask.array<chunksize=(1,), meta=np.ndarray>
      bounds :
      time_centered_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      Array Chunk
      Bytes 224 B 8 B
      Shape (28,) (1,)
      Count 171 Tasks 28 Chunks
      Type object numpy.ndarray
      		28 1
    • t
      (t)
      object
      2012-02-01 12:00:00 ... 2012-02-...
      axis :
      T
      bounds :
      time_counter_bounds
      long_name :
      Time axis
      standard_name :
      time
      time_origin :
      1900-01-01 00:00:00
      array([cftime.DatetimeNoLeap(2012, 2, 1, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 2, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 3, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 4, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 5, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 6, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 7, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 8, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 9, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 10, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 11, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 12, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 13, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 14, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 15, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 16, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 17, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 18, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 19, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 20, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 21, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 22, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 23, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 24, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 25, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 26, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 27, 12, 0, 0, 0, has_year_zero=True),
       cftime.DatetimeNoLeap(2012, 2, 28, 12, 0, 0, 0, has_year_zero=True)],
      dtype=object)
    • y
      ()
      int64
      2609
      array(2609)
    • Mean Tempreture
      (t)
      float32
      dask.array<chunksize=(1,), meta=np.ndarray>
      Array Chunk
      Bytes 112 B 4 B
      Shape (28,) (1,)
      Count 2247 Tasks 28 Chunks
      Type float32 numpy.ndarray
      		28 1
    • Mean Velocity
      (t)
      float32
      dask.array<chunksize=(1,), meta=np.ndarray>
      Array Chunk
      Bytes 112 B 4 B
      Shape (28,) (1,)
      Count 2275 Tasks 28 Chunks
      Type float32 numpy.ndarray
      		28 1 
#4 Saving  SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity
data=save.datas(data,plot=Plot,path=nc_outputpath,filename=filename)
start saving data
saving data in a  csv file ../nc_results/SEDNA_DELTA_MONITOR/SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity2012-02-01_2012-02-28.nc
save computed data at ../nc_results/SEDNA_DELTA_MONITOR/SEDNA_Mean_temp_velo_integrals_FramS_Small_Mean_Temp_&_Velocity2012-02-01_2012-02-28.nc completed
CPU times: user 6.97 s, sys: 887 ms, total: 7.86 s
Wall time: 11.1 s