%matplotlib inline
import pandas as pd
import socket
host = socket.getfqdn()

from core import  load, zoom, calc, save,plots,monitor

#reload funcs after updating ./core/*.py
import importlib
importlib.reload(load)
importlib.reload(zoom)
importlib.reload(calc)
importlib.reload(save)
importlib.reload(plots)
importlib.reload(monitor)

<module 'core.monitor' from '/ccc/work/cont003/gen7420/talandel/TOOLS/monitor-sedna/notebook/core/monitor.py'>

If you submit the job with job scheduler; below are list of enviroment variable one can pass

• %env local='2"

local : if True run dask local cluster, if not true, put number of workers setted in the 'local' if no 'local ' given, local will be setted automatically to 'True'

• %env ychunk='2', #%env tchunk='2'

  controls chunk. 'False' sets no modification from original netcdf file's chunk.
  ychunk=10 will group the original netcdf file to 10 by 10 tchunk=1 will chunk the time coordinate one by one

• %env file_exp=

  'file_exp': Which 'experiment' name is it? this corresopnds to intake catalog name without path and .yaml

• #%env year=

  for Validation, this correspoinds to path/year/month 's year for monitoring, this corresponids to 'date' having means do all files in the monitoring directory setting it as 0[0-9] &1[0-9]& [2-3][0-9], the job can be separated in three lots. For DELTA experiment, year corresponds to really 'year'

• %env month=

  for monitoring this corresponds to file path path-XIOS.{month}/

For DELTA experiment, year corresponds to really 'month'

• #%env save=

proceed saving? True or False , Default is setted as True

• #%env plot=

proceed plotting? True or False , Default is setted as True

• #%env calc=

proceed computation? or just load computed result? True or False , Default is setted as True

• #%env save=False

save output file used for plotting

• #%env lazy=False

using kerchunked file -> False, not using kerhcunk -> True

• #%env control=FWC_SSH

name of control file to be used for computation/plots/save/ We have number of M_xxx.csv

Monitor.sh automate all of above enviroment settings.

Monitor.sh calls M_MLD_2D

and AWTD.sh, Fluxnet.sh, Siconc.sh, IceClim.sh, FWC_SSH.sh, Integrals.sh , Sections.sh

• AWTD.sh

M_AWTMD

• Fluxnet.sh

M_Fluxnet

• Siconc.sh

M_Ice_quantities

• IceClim.sh

M_IceClim M_IceConce M_IceThick

• FWC_SSH.sh

M_FWC_2D M_FWC_integrals M_FWC_SSH M_SSH_anomaly

• Integrals.sh

M_Mean_temp_velo M_Mooring

• Sections.sh

M_Sectionx M_Sectiony

# For debugging this cell can help turn this cell to 'code' and modify variables manually. %env file_exp=SEDNA_DELTA_MONITOR %env year=2012 %env month=01 #0[1-2] %env ychunk=10 %env ychunk=False %env save=False %env plot=True %env calc=True # %env lazy=False #False %env control=M_Fluxnet #M_Sectiony ok with ychunk=False local=True lazy=False

%%time
# 'savefig': Do we save output in html? or not. keep it true. 
savefig=True
client,cluster,control,catalog_url,month,year,daskreport,outputpath = load.set_control(host)
!mkdir -p $outputpath
!mkdir -p $daskreport
client

local True
using host= irene6170.c-irene.mg1.tgcc.ccc.cea.fr starting dask cluster on local= True workers 16
10000000000
rome local cluster starting
This code is running on  irene6170.c-irene.mg1.tgcc.ccc.cea.fr using  SEDNA_DELTA_MONITOR file experiment, read from  ../lib/SEDNA_DELTA_MONITOR.yaml  on year= 2015  on month= 10  outputpath= ../results/SEDNA_DELTA_MONITOR/ daskreport= ../results/dask/7449013irene6170.c-irene.mg1.tgcc.ccc.cea.fr_SEDNA_DELTA_MONITOR_10M_SSH_anomaly/
CPU times: user 420 ms, sys: 91.3 ms, total: 511 ms
Wall time: 11.2 s

Client

Client-5f2ce1b6-6f42-11ed-8e16-080038b94977

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

dc7e7723

Dashboard: http://127.0.0.1:8787/status	Workers: 16
Total threads: 128	Total memory: 221.88 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-a857a2e0-5640-48ba-97ae-be0581864309

Comm: tcp://127.0.0.1:41665	Workers: 16
Dashboard: http://127.0.0.1:8787/status	Total threads: 128
Started: Just now	Total memory: 221.88 GiB

Workers

Worker: 0

Comm: tcp://127.0.0.1:38083	Total threads: 8
Dashboard: http://127.0.0.1:37960/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:40450
Local directory: /tmp/dask-worker-space/worker-bv9z8q4j

Worker: 1

Comm: tcp://127.0.0.1:41171	Total threads: 8
Dashboard: http://127.0.0.1:44770/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:32962
Local directory: /tmp/dask-worker-space/worker-2ozzu5od

Worker: 2

Comm: tcp://127.0.0.1:36154	Total threads: 8
Dashboard: http://127.0.0.1:39246/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:45961
Local directory: /tmp/dask-worker-space/worker-lofdigs3

Worker: 3

Comm: tcp://127.0.0.1:34579	Total threads: 8
Dashboard: http://127.0.0.1:45821/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:39882
Local directory: /tmp/dask-worker-space/worker-dlq2cmrv

Worker: 4

Comm: tcp://127.0.0.1:34456	Total threads: 8
Dashboard: http://127.0.0.1:39805/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:43587
Local directory: /tmp/dask-worker-space/worker-gu90um5e

Worker: 5

Comm: tcp://127.0.0.1:42100	Total threads: 8
Dashboard: http://127.0.0.1:33599/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:40105
Local directory: /tmp/dask-worker-space/worker-rkfq5swr

Worker: 6

Comm: tcp://127.0.0.1:33510	Total threads: 8
Dashboard: http://127.0.0.1:39167/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:39277
Local directory: /tmp/dask-worker-space/worker-v8e16fat

Worker: 7

Comm: tcp://127.0.0.1:35198	Total threads: 8
Dashboard: http://127.0.0.1:42483/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:44689
Local directory: /tmp/dask-worker-space/worker-edk8n4j0

Worker: 8

Comm: tcp://127.0.0.1:46796	Total threads: 8
Dashboard: http://127.0.0.1:33009/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:41077
Local directory: /tmp/dask-worker-space/worker-xx4zydxh

Worker: 9

Comm: tcp://127.0.0.1:34117	Total threads: 8
Dashboard: http://127.0.0.1:35094/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:34660
Local directory: /tmp/dask-worker-space/worker-pxo45wn6

Worker: 10

Comm: tcp://127.0.0.1:35781	Total threads: 8
Dashboard: http://127.0.0.1:40773/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:42942
Local directory: /tmp/dask-worker-space/worker-jloqtf9u

Worker: 11

Comm: tcp://127.0.0.1:45540	Total threads: 8
Dashboard: http://127.0.0.1:45543/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:33734
Local directory: /tmp/dask-worker-space/worker-6fig19al

Worker: 12

Comm: tcp://127.0.0.1:35901	Total threads: 8
Dashboard: http://127.0.0.1:44218/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:34307
Local directory: /tmp/dask-worker-space/worker-kw7nsjne

Worker: 13

Comm: tcp://127.0.0.1:41547	Total threads: 8
Dashboard: http://127.0.0.1:37431/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:33368
Local directory: /tmp/dask-worker-space/worker-g5ntah50

Worker: 14

Comm: tcp://127.0.0.1:36394	Total threads: 8
Dashboard: http://127.0.0.1:41397/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:39669
Local directory: /tmp/dask-worker-space/worker-034593uk

Worker: 15

Comm: tcp://127.0.0.1:35435	Total threads: 8
Dashboard: http://127.0.0.1:33708/status	Memory: 13.87 GiB
Nanny: tcp://127.0.0.1:36244
Local directory: /tmp/dask-worker-space/worker-_z1r1444

read plotting information from a csv file

df=load.controlfile(control)
#Take out 'later' tagged computations
#df=df[~df['Value'].str.contains('later')]
df

	Value	Inputs	Equation	Zone	Plot	Colourmap	MinMax	Unit	Oldname	Unnamed: 10
	SSH_anomaly	gridT-2D.ssh	calc.SSH_anomaly(data)	ALL	maps	Spectral_r	(-0.35,0.35)	m	M-2

Computation starts here

Each computation consists of

1. Load NEMO data set
2. Zoom data set
3. Compute (or load computed data set)
4. Save
5. Plot
6. Close

%%time
import os
calcswitch=os.environ.get('calc', 'True') 
lazy=os.environ.get('lazy','False' )
loaddata=((df.Inputs != '').any()) 
print('calcswitch=',calcswitch,'df.Inputs != nothing',loaddata, 'lazy=',lazy)
data = load.datas(catalog_url,df.Inputs,month,year,daskreport,lazy=lazy) if ((calcswitch=='True' )*loaddata) else 0 
data

calcswitch= True df.Inputs != nothing True lazy= True
../lib/SEDNA_DELTA_MONITOR.yaml
using param_xios reading  ../lib/SEDNA_DELTA_MONITOR.yaml
using param_xios reading  <bound method DataSourceBase.describe of sources:
  param_xios:
    args:
      combine: nested
      concat_dim: y
      urlpath: /ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc
      xarray_kwargs:
        compat: override
        coords: minimal
        data_vars: minimal
        parallel: true
    description: SEDNA NEMO parameters from MPI output  nav_lon lat fails
    driver: intake_xarray.netcdf.NetCDFSource
    metadata:
      catalog_dir: /ccc/work/cont003/gen7420/talandel/TOOLS/monitor-sedna/notebook/../lib/
>
{'name': 'param_xios', 'container': 'xarray', 'plugin': ['netcdf'], 'driver': ['netcdf'], 'description': 'SEDNA NEMO parameters from MPI output  nav_lon lat fails', 'direct_access': 'forbid', 'user_parameters': [{'name': 'path', 'description': 'file coordinate', 'type': 'str', 'default': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/MESH/SEDNA_mesh_mask_Tgt_20210423_tsh10m_L1/param'}], 'metadata': {}, 'args': {'urlpath': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc', 'combine': 'nested', 'concat_dim': 'y'}}
0 read gridT-2D ['ssh']
lazy= True
using load_data_xios reading  gridT-2D
using load_data_xios reading  {'name': 'data_xios', 'container': 'xarray', 'plugin': ['netcdf'], 'driver': ['netcdf'], 'description': 'SEDNA NEMO outputs from different xios server', 'direct_access': 'forbid', 'user_parameters': [{'name': 'path', 'description': 'name of config', 'type': 'str', 'default': '/ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d'}, {'name': 'fileexp', 'description': 'name of config', 'type': 'str', 'default': 'SEDNA-DELTA'}, {'name': 'month', 'description': 'running number 2 digit', 'type': 'str', 'default': '02'}, {'name': 'freq', 'description': '1d or 1m', 'type': 'str', 'default': '1d'}, {'name': 'year', 'description': 'last digits of yearmonthdate.', 'type': 'str', 'default': '2012'}, {'name': 'file', 'description': 'file name', 'type': 'str', 'default': 'icemod'}, {'name': 'eio', 'description': 'xios mpi number', 'type': 'str', 'default': '0[0-5][0-9][0-9]'}], 'metadata': {}, 'args': {'urlpath': '{{path}}/{{year}}/{{month}}/*{{file}}_*_{{eio}}.nc', 'combine': 'nested', 'concat_dim': 'y'}}
      took 53.59534931182861 seconds
0 merging gridT-2D ['ssh']
param nav_lon will be included in data
param mask2d will be included in data
param nav_lat will be included in data
ychunk= 10 calldatas_y_rechunk
sum_num (13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12)
start rechunking with (130, 122, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 48)
end of y_rechunk
CPU times: user 9.89 s, sys: 2.41 s, total: 12.3 s
Wall time: 1min 7s

<xarray.Dataset>
Dimensions:  (t: 31, y: 6540, x: 6560)
Coordinates:
    nav_lat  (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray>
    nav_lon  (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray>
  * t        (t) object 2015-10-01 12:00:00 ... 2015-10-31 12:00:00
  * y        (y) int64 1 2 3 4 5 6 7 8 ... 6534 6535 6536 6537 6538 6539 6540
  * x        (x) int64 1 2 3 4 5 6 7 8 ... 6554 6555 6556 6557 6558 6559 6560
    mask2d   (y, x) bool dask.array<chunksize=(130, 6560), meta=np.ndarray>
Data variables:
    ssh      (t, y, x) float32 dask.array<chunksize=(31, 130, 6560), meta=np.ndarray>
Attributes: (12/26)
    name:                    /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUN...
    description:             ocean T grid variables
    title:                   ocean T grid variables
    Conventions:             CF-1.6
    timeStamp:               2022-Aug-18 08:24:52 GMT
    uuid:                    33c4cf83-ff69-4680-af4f-c19dd7faf10e
    ...                      ...
    start_date:              20090101
    output_frequency:        1d
    CONFIG:                  SEDNA
    CASE:                    DELTA
    history:                 Fri Aug 19 13:43:38 2022: ncks -4 -L 1 SEDNA-DEL...
    NCO:                     netCDF Operators version 4.9.1 (Homepage = http:...

xarray.Dataset

• Dimensions:
  • t: 31
  • y: 6540
  • x: 6560
• Coordinates: (6)
  • nav_lat
    (y, x)
    float32
    dask.array<chunksize=(130, 6560), meta=np.ndarray>

    Array Chunk Bytes 163.66 MiB 3.25 MiB Shape (6540, 6560) (130, 6560) Count 1687 Tasks 55 Chunks Type float32 numpy.ndarray

  • nav_lon
    (y, x)
    float32
    dask.array<chunksize=(130, 6560), meta=np.ndarray>

    Array Chunk Bytes 163.66 MiB 3.25 MiB Shape (6540, 6560) (130, 6560) Count 1687 Tasks 55 Chunks Type float32 numpy.ndarray

  • t
    (t)
    object
    2015-10-01 12:00:00 ... 2015-10-...

    axis :

    T

    standard_name :

    time

    long_name :

    Time axis

    time_origin :

    1900-01-01 00:00:00

    bounds :

    time_counter_bounds

    array([cftime.DatetimeNoLeap(2015, 10, 1, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 2, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 3, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 4, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 5, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 6, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 7, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 8, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 9, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 10, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 11, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 12, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 13, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 14, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 15, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 17, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 18, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 19, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 20, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 21, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 22, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 23, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 24, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 25, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 26, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 27, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 28, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 29, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 30, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 31, 12, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • y
    (y)
    int64
    1 2 3 4 5 ... 6537 6538 6539 6540

    array([   1,    2,    3, ..., 6538, 6539, 6540])

  • x
    (x)
    int64
    1 2 3 4 5 ... 6557 6558 6559 6560

    array([   1,    2,    3, ..., 6558, 6559, 6560])

  • mask2d
    (y, x)
    bool
    dask.array<chunksize=(130, 6560), meta=np.ndarray>

    Array Chunk Bytes 40.91 MiB 832.81 kiB Shape (6540, 6560) (130, 6560) Count 1687 Tasks 55 Chunks Type bool numpy.ndarray

• Data variables: (1)
  • ssh
    (t, y, x)
    float32
    dask.array<chunksize=(31, 130, 6560), meta=np.ndarray>

    standard_name :

    sea_surface_height_above_geoid

    long_name :

    sea surface height

    units :

    m

    online_operation :

    average

    interval_operation :

    40 s

    interval_write :

    1 d

    cell_methods :

    time: mean (interval: 40 s)

    Array Chunk Bytes 4.95 GiB 100.85 MiB Shape (31, 6540, 6560) (31, 130, 6560) Count 1687 Tasks 55 Chunks Type float32 numpy.ndarray

• Attributes: (26)

  name :

  /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUNS/SEDNA-DELTA-XIOS.92/SEDNA-DELTA_1d_gridT-2D

  description :

  ocean T grid variables

  title :

  ocean T grid variables

  Conventions :

  CF-1.6

  timeStamp :

  2022-Aug-18 08:24:52 GMT

  uuid :

  33c4cf83-ff69-4680-af4f-c19dd7faf10e

  ibegin :

  0

  ni :

  6560

  jbegin :

  0

  nj :

  13

  DOMAIN_number_total :

  544

  DOMAIN_number :

  0

  DOMAIN_dimensions_ids :

  [2 3]

  DOMAIN_size_global :

  [6560 6540]

  DOMAIN_size_local :

  [6560 13]

  DOMAIN_position_first :

  [1 1]

  DOMAIN_position_last :

  [6560 13]

  DOMAIN_halo_size_start :

  [0 0]

  DOMAIN_halo_size_end :

  [0 0]

  DOMAIN_type :

  box

  start_date :

  20090101

  output_frequency :

  1d

  CONFIG :

  SEDNA

  CASE :

  DELTA

  history :

  Fri Aug 19 13:43:38 2022: ncks -4 -L 1 SEDNA-DELTA_1d_gridT-2D_201510-201510_0000.nc /ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d/2015/10/SEDNA-DELTA_1d_gridT-2D_201510-201510_0000.nc

  NCO :

  netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)

%%time
monitor.auto(df,data,savefig,daskreport,outputpath,file_exp='SEDNA'
            )

#calc= True
#save= True
#plot= False
Value='SSH_anomaly'
Zone='ALL'
Plot='maps'
cmap='Spectral_r'
clabel='m'
clim= (-0.35, 0.35)
outputpath='../results/SEDNA_DELTA_MONITOR/'
nc_outputpath='../nc_results/SEDNA_DELTA_MONITOR/'
filename='SEDNA_maps_ALL_SSH_anomaly'
data=monitor.optimize_dataset(data)
#3 Start computing 
data= calc.SSH_anomaly(data)
monitor.optimize_dataset(data)
add optimise here once otimise can recognise

<xarray.Dataset>
Dimensions:      (t: 31, y: 6540, x: 6560)
Coordinates:
  * t            (t) object 2015-10-01 12:00:00 ... 2015-10-31 12:00:00
    nav_lat      (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray>
    nav_lon      (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray>
  * y            (y) int64 1 2 3 4 5 6 7 ... 6534 6535 6536 6537 6538 6539 6540
  * x            (x) int64 1 2 3 4 5 6 7 ... 6554 6555 6556 6557 6558 6559 6560
    mask2d       (y, x) bool dask.array<chunksize=(130, 6560), meta=np.ndarray>
Data variables:
    SSH_anomaly  (t, y, x) float32 dask.array<chunksize=(31, 130, 6560), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • t: 31
  • y: 6540
  • x: 6560
• Coordinates: (6)
  • t
    (t)
    object
    2015-10-01 12:00:00 ... 2015-10-...

    axis :

    T

    standard_name :

    time

    long_name :

    Time axis

    time_origin :

    1900-01-01 00:00:00

    bounds :

    time_counter_bounds

    array([cftime.DatetimeNoLeap(2015, 10, 1, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 2, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 3, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 4, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 5, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 6, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 7, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 8, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 9, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 10, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 11, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 12, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 13, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 14, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 15, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 17, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 18, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 19, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 20, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 21, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 22, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 23, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 24, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 25, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 26, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 27, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 28, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 29, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 30, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 10, 31, 12, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • nav_lat
    (y, x)
    float32
    dask.array<chunksize=(130, 6560), meta=np.ndarray>

    Array Chunk Bytes 163.66 MiB 3.25 MiB Shape (6540, 6560) (130, 6560) Count 1687 Tasks 55 Chunks Type float32 numpy.ndarray

  • nav_lon
    (y, x)
    float32
    dask.array<chunksize=(130, 6560), meta=np.ndarray>

    Array Chunk Bytes 163.66 MiB 3.25 MiB Shape (6540, 6560) (130, 6560) Count 1687 Tasks 55 Chunks Type float32 numpy.ndarray

  • y
    (y)
    int64
    1 2 3 4 5 ... 6537 6538 6539 6540

    array([   1,    2,    3, ..., 6538, 6539, 6540])

  • x
    (x)
    int64
    1 2 3 4 5 ... 6557 6558 6559 6560

    array([   1,    2,    3, ..., 6558, 6559, 6560])

  • mask2d
    (y, x)
    bool
    dask.array<chunksize=(130, 6560), meta=np.ndarray>

    Array Chunk Bytes 40.91 MiB 832.81 kiB Shape (6540, 6560) (130, 6560) Count 1687 Tasks 55 Chunks Type bool numpy.ndarray

• Data variables: (1)
  • SSH_anomaly
    (t, y, x)
    float32
    dask.array<chunksize=(31, 130, 6560), meta=np.ndarray>

    Array Chunk Bytes 4.95 GiB 100.85 MiB Shape (31, 6540, 6560) (31, 130, 6560) Count 3485 Tasks 55 Chunks Type float32 numpy.ndarray

• Attributes: (0)

#4 Saving  SEDNA_maps_ALL_SSH_anomaly
data=save.datas(data,plot=Plot,path=nc_outputpath,filename=filename)
start saving data
saving data in a file
t (31,)
0
slice(0, 31, None)
CPU times: user 8.78 s, sys: 8.11 s, total: 16.9 s
Wall time: 24.1 s