%matplotlib inline
import pandas as pd
import socket
host = socket.getfqdn()

from core import  load, zoom, calc, save,plots,monitor

#reload funcs after updating ./core/*.py
import importlib
importlib.reload(load)
importlib.reload(zoom)
importlib.reload(calc)
importlib.reload(save)
importlib.reload(plots)
importlib.reload(monitor)

<module 'core.monitor' from '/ccc/work/cont003/gen7420/talandel/TOOLS/monitor-sedna/notebook/core/monitor.py'>

If you submit the job with job scheduler; below are list of enviroment variable one can pass

• %env local='2"

local : if True run dask local cluster, if not true, put number of workers setted in the 'local' if no 'local ' given, local will be setted automatically to 'True'

• %env ychunk='2', #%env tchunk='2'

  controls chunk. 'False' sets no modification from original netcdf file's chunk.
  ychunk=10 will group the original netcdf file to 10 by 10 tchunk=1 will chunk the time coordinate one by one

• %env file_exp=

  'file_exp': Which 'experiment' name is it? this corresopnds to intake catalog name without path and .yaml

• #%env year=

  for Validation, this correspoinds to path/year/month 's year for monitoring, this corresponids to 'date' having means do all files in the monitoring directory setting it as 0[0-9] &1[0-9]& [2-3][0-9], the job can be separated in three lots. For DELTA experiment, year corresponds to really 'year'

• %env month=

  for monitoring this corresponds to file path path-XIOS.{month}/

For DELTA experiment, year corresponds to really 'month'

• #%env save=

proceed saving? True or False , Default is setted as True

• #%env plot=

proceed plotting? True or False , Default is setted as True

• #%env calc=

proceed computation? or just load computed result? True or False , Default is setted as True

• #%env save=False

save output file used for plotting

• #%env lazy=False

using kerchunked file -> False, not using kerhcunk -> True

• #%env control=FWC_SSH

name of control file to be used for computation/plots/save/ We have number of M_xxx.csv

Monitor.sh automate all of above enviroment settings.

Monitor.sh calls M_MLD_2D

and AWTD.sh, Fluxnet.sh, Siconc.sh, IceClim.sh, FWC_SSH.sh, Integrals.sh , Sections.sh

• AWTD.sh

M_AWTMD

• Fluxnet.sh

M_Fluxnet

• Siconc.sh

M_Ice_quantities

• IceClim.sh

M_IceClim M_IceConce M_IceThick

• FWC_SSH.sh

M_FWC_2D M_FWC_integrals M_FWC_SSH M_SSH_anomaly

• Integrals.sh

M_Mean_temp_velo M_Mooring

• Sections.sh

M_Sectionx M_Sectiony

# For debugging this cell can help turn this cell to 'code' and modify variables manually. %env file_exp=SEDNA_DELTA_MONITOR %env year=2012 %env month=01 #0[1-2] %env ychunk=10 %env ychunk=False %env save=False %env plot=True %env calc=True # %env lazy=False #False %env control=M_Fluxnet #M_Sectiony ok with ychunk=False local=True lazy=False

%%time
# 'savefig': Do we save output in html? or not. keep it true. 
savefig=True
client,cluster,control,catalog_url,month,year,daskreport,outputpath = load.set_control(host)
!mkdir -p $outputpath
!mkdir -p $daskreport
client

local True
using host= irene8002.c-irene.mg1.tgcc.ccc.cea.fr starting dask cluster on local= True workers 16
10000000000
rome local cluster starting
This code is running on  irene8002.c-irene.mg1.tgcc.ccc.cea.fr using  SEDNA_DELTA_MONITOR file experiment, read from  ../lib/SEDNA_DELTA_MONITOR.yaml  on year= 2015  on month= 12  outputpath= ../results/SEDNA_DELTA_MONITOR/ daskreport= ../results/dask/7402419irene8002.c-irene.mg1.tgcc.ccc.cea.fr_SEDNA_DELTA_MONITOR_12M_Sectionx/
CPU times: user 472 ms, sys: 194 ms, total: 666 ms
Wall time: 10.6 s

Client

Client-ee9fff88-6da9-11ed-bd7f-080038bfdcae

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

62561db3

Dashboard: http://127.0.0.1:8787/status	Workers: 16
Total threads: 128	Total memory: 2.86 TiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-692b52dd-5986-409c-b900-b956bbf2d000

Comm: tcp://127.0.0.1:38845	Workers: 16
Dashboard: http://127.0.0.1:8787/status	Total threads: 128
Started: Just now	Total memory: 2.86 TiB

Workers

Worker: 0

Comm: tcp://127.0.0.1:36574	Total threads: 8
Dashboard: http://127.0.0.1:38218/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:37179
Local directory: /tmp/dask-worker-space/worker-i2x76ra1

Worker: 1

Comm: tcp://127.0.0.1:40011	Total threads: 8
Dashboard: http://127.0.0.1:35804/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:45770
Local directory: /tmp/dask-worker-space/worker-_go62pg3

Worker: 2

Comm: tcp://127.0.0.1:39015	Total threads: 8
Dashboard: http://127.0.0.1:34033/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:45773
Local directory: /tmp/dask-worker-space/worker-muj3vi9v

Worker: 3

Comm: tcp://127.0.0.1:39262	Total threads: 8
Dashboard: http://127.0.0.1:34417/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:42164
Local directory: /tmp/dask-worker-space/worker-yfn1cifg

Worker: 4

Comm: tcp://127.0.0.1:36971	Total threads: 8
Dashboard: http://127.0.0.1:45904/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:40972
Local directory: /tmp/dask-worker-space/worker-rmnfin1c

Worker: 5

Comm: tcp://127.0.0.1:41408	Total threads: 8
Dashboard: http://127.0.0.1:39782/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:43019
Local directory: /tmp/dask-worker-space/worker-ri48yhb8

Worker: 6

Comm: tcp://127.0.0.1:33140	Total threads: 8
Dashboard: http://127.0.0.1:36697/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:34475
Local directory: /tmp/dask-worker-space/worker-sjmvpcek

Worker: 7

Comm: tcp://127.0.0.1:46180	Total threads: 8
Dashboard: http://127.0.0.1:39832/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:42298
Local directory: /tmp/dask-worker-space/worker-josl7nz3

Worker: 8

Comm: tcp://127.0.0.1:38973	Total threads: 8
Dashboard: http://127.0.0.1:39693/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:44323
Local directory: /tmp/dask-worker-space/worker-hryd71z8

Worker: 9

Comm: tcp://127.0.0.1:45841	Total threads: 8
Dashboard: http://127.0.0.1:34229/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:35434
Local directory: /tmp/dask-worker-space/worker-aupy1tu9

Worker: 10

Comm: tcp://127.0.0.1:43570	Total threads: 8
Dashboard: http://127.0.0.1:42101/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:44959
Local directory: /tmp/dask-worker-space/worker-0qllr6ac

Worker: 11

Comm: tcp://127.0.0.1:37856	Total threads: 8
Dashboard: http://127.0.0.1:34588/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:44131
Local directory: /tmp/dask-worker-space/worker-4x0qo3hj

Worker: 12

Comm: tcp://127.0.0.1:39415	Total threads: 8
Dashboard: http://127.0.0.1:38356/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:45795
Local directory: /tmp/dask-worker-space/worker-sf581kj4

Worker: 13

Comm: tcp://127.0.0.1:34853	Total threads: 8
Dashboard: http://127.0.0.1:40421/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:36587
Local directory: /tmp/dask-worker-space/worker-0y63g7gy

Worker: 14

Comm: tcp://127.0.0.1:34224	Total threads: 8
Dashboard: http://127.0.0.1:37410/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:43836
Local directory: /tmp/dask-worker-space/worker-ep3qjhrw

Worker: 15

Comm: tcp://127.0.0.1:42020	Total threads: 8
Dashboard: http://127.0.0.1:39257/status	Memory: 183.20 GiB
Nanny: tcp://127.0.0.1:37756
Local directory: /tmp/dask-worker-space/worker-8xo26six

read plotting information from a csv file

df=load.controlfile(control)
#Take out 'later' tagged computations
#df=df[~df['Value'].str.contains('later')]
df

	Value	Inputs	Equation	Zone	Plot	Colourmap	MinMax	Unit	Oldname	Unnamed: 10
	Section	gridT.votemper,gridS.vosaline,gridU.vozocrtx,p...	data.chunk({'y':-1}).unify_chunks().persist()	BFGS	section	None	{'vosaline': (28,35), 'votemper': (-2,2), 'voz...	None	S-2

Computation starts here

Each computation consists of

1. Load NEMO data set
2. Zoom data set
3. Compute (or load computed data set)
4. Save
5. Plot
6. Close

%%time
import os
calcswitch=os.environ.get('calc', 'True') 
lazy=os.environ.get('lazy','False' )
loaddata=((df.Inputs != '').any()) 
print('calcswitch=',calcswitch,'df.Inputs != nothing',loaddata, 'lazy=',lazy)
data = load.datas(catalog_url,df.Inputs,month,year,daskreport,lazy=lazy) if ((calcswitch=='True' )*loaddata) else 0 
data

calcswitch= False df.Inputs != nothing True lazy= False
CPU times: user 0 ns, sys: 427 µs, total: 427 µs
Wall time: 435 µs

0

%%time
monitor.auto(df,data,savefig,daskreport,outputpath,file_exp='SEDNA'
            )

#calc= False
#save= False
#plot= True
Value='Section'
Zone='BFGS'
Plot='section'
cmap='None'
clabel='None'
clim= {'vosaline': (28, 35), 'votemper': (-2, 2), 'vozocrtx': (-0.05, 0.05)}
outputpath='../results/SEDNA_DELTA_MONITOR/'
nc_outputpath='../nc_results/SEDNA_DELTA_MONITOR/'
filename='SEDNA_section_BFGS_Section'
#3 no computing , loading starts
data=save.load_data(plot=Plot,path=nc_outputpath,filename=filename)
start loading data
filename= ../nc_results/SEDNA_DELTA_MONITOR/SEDNA_section_BFGS_Section/t_*.nc dim t
load computed data completed

<xarray.Dataset>
Dimensions:   (t: 365, z: 95, y: 2384)
Coordinates:
  * t         (t) object 2015-01-01 12:00:00 ... 2015-12-31 12:00:00
  * y         (y) int64 3494 3495 3496 3497 3498 ... 5909 5910 5911 5912 5913
    x         int64 ...
  * z         (z) int64 1 2 3 4 5 6 7 8 9 10 ... 86 87 88 89 90 91 92 93 94 95
    nav_lon   (y) float32 dask.array<chunksize=(2384,), meta=np.ndarray>
    nav_lat   (y) float32 dask.array<chunksize=(2384,), meta=np.ndarray>
    mask2d    (y) bool dask.array<chunksize=(2384,), meta=np.ndarray>
    depth     (z, y) float32 dask.array<chunksize=(95, 2384), meta=np.ndarray>
    mask      (z, y) bool dask.array<chunksize=(95, 2384), meta=np.ndarray>
Data variables:
    vosaline  (t, z, y) float32 dask.array<chunksize=(1, 95, 2384), meta=np.ndarray>
    votemper  (t, z, y) float32 dask.array<chunksize=(1, 95, 2384), meta=np.ndarray>
    vozocrtx  (t, z, y) float32 dask.array<chunksize=(1, 95, 2384), meta=np.ndarray>
Attributes: (12/26)
    CASE:                    DELTA
    CONFIG:                  SEDNA
    Conventions:             CF-1.6
    DOMAIN_dimensions_ids:   [2 3]
    DOMAIN_halo_size_end:    [0 0]
    DOMAIN_halo_size_start:  [0 0]
    ...                      ...
    nj:                      13
    output_frequency:        1d
    start_date:              20090101
    timeStamp:               2022-Jul-21 16:35:22 GMT
    title:                   ocean T grid variables
    uuid:                    9aef3543-35d6-4da0-a58a-f2c75b69d3a7

xarray.Dataset

• Dimensions:
  • t: 365
  • z: 95
  • y: 2384
• Coordinates: (9)
  • t
    (t)
    object
    2015-01-01 12:00:00 ... 2015-12-...

    axis :

    T

    bounds :

    time_counter_bounds

    long_name :

    Time axis

    standard_name :

    time

    time_origin :

    1900-01-01 00:00:00

    array([cftime.DatetimeNoLeap(2015, 1, 1, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 1, 2, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 1, 3, 12, 0, 0, 0, has_year_zero=True), ...,
           cftime.DatetimeNoLeap(2015, 12, 29, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 12, 30, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2015, 12, 31, 12, 0, 0, 0, has_year_zero=True)],
          dtype=object)

  • y
    (y)
    int64
    3494 3495 3496 ... 5911 5912 5913

    array([3494, 3495, 3496, ..., 5911, 5912, 5913])

  • x
    ()
    int64
    ...

    array(2281)

  • z
    (z)
    int64
    1 2 3 4 5 6 7 ... 90 91 92 93 94 95

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
           19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
           37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54,
           55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72,
           73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
           91, 92, 93, 94, 95])

  • nav_lon
    (y)
    float32
    dask.array<chunksize=(2384,), meta=np.ndarray>

    Array Chunk Bytes 9.31 kiB 9.31 kiB Shape (2384,) (2384,) Count 2 Tasks 1 Chunks Type float32 numpy.ndarray

  • nav_lat
    (y)
    float32
    dask.array<chunksize=(2384,), meta=np.ndarray>

    Array Chunk Bytes 9.31 kiB 9.31 kiB Shape (2384,) (2384,) Count 2 Tasks 1 Chunks Type float32 numpy.ndarray

  • mask2d
    (y)
    bool
    dask.array<chunksize=(2384,), meta=np.ndarray>

    Array Chunk Bytes 2.33 kiB 2.33 kiB Shape (2384,) (2384,) Count 2 Tasks 1 Chunks Type bool numpy.ndarray

  • depth
    (z, y)
    float32
    dask.array<chunksize=(95, 2384), meta=np.ndarray>

    Array Chunk Bytes 884.69 kiB 884.69 kiB Shape (95, 2384) (95, 2384) Count 2 Tasks 1 Chunks Type float32 numpy.ndarray

  • mask
    (z, y)
    bool
    dask.array<chunksize=(95, 2384), meta=np.ndarray>

    Array Chunk Bytes 221.17 kiB 221.17 kiB Shape (95, 2384) (95, 2384) Count 2 Tasks 1 Chunks Type bool numpy.ndarray

• Data variables: (3)
  • vosaline
    (t, z, y)
    float32
    dask.array<chunksize=(1, 95, 2384), meta=np.ndarray>

    cell_methods :

    time: mean (interval: 40 s)

    interval_operation :

    40 s

    interval_write :

    1 d

    long_name :

    salinity

    online_operation :

    average

    standard_name :

    sea_water_practical_salinity

    units :

    1e-3

    Array Chunk Bytes 315.34 MiB 884.69 kiB Shape (365, 95, 2384) (1, 95, 2384) Count 1095 Tasks 365 Chunks Type float32 numpy.ndarray

  • votemper
    (t, z, y)
    float32
    dask.array<chunksize=(1, 95, 2384), meta=np.ndarray>

    cell_methods :

    time: mean (interval: 40 s)

    interval_operation :

    40 s

    interval_write :

    1 d

    long_name :

    temperature

    online_operation :

    average

    standard_name :

    sea_water_potential_temperature

    units :

    degC

    Array Chunk Bytes 315.34 MiB 884.69 kiB Shape (365, 95, 2384) (1, 95, 2384) Count 1095 Tasks 365 Chunks Type float32 numpy.ndarray

  • vozocrtx
    (t, z, y)
    float32
    dask.array<chunksize=(1, 95, 2384), meta=np.ndarray>

    cell_methods :

    time: mean (interval: 40 s)

    interval_operation :

    40 s

    interval_write :

    1 d

    long_name :

    ocean current along i-axis

    online_operation :

    average

    standard_name :

    sea_water_x_velocity

    units :

    m/s

    Array Chunk Bytes 315.34 MiB 884.69 kiB Shape (365, 95, 2384) (1, 95, 2384) Count 1095 Tasks 365 Chunks Type float32 numpy.ndarray

• Attributes: (26)

  CASE :

  DELTA

  CONFIG :

  SEDNA

  Conventions :

  CF-1.6

  DOMAIN_dimensions_ids :

  [2 3]

  DOMAIN_halo_size_end :

  [0 0]

  DOMAIN_halo_size_start :

  [0 0]

  DOMAIN_number :

  0

  DOMAIN_number_total :

  544

  DOMAIN_position_first :

  [1 1]

  DOMAIN_position_last :

  [6560 13]

  DOMAIN_size_global :

  [6560 6540]

  DOMAIN_size_local :

  [6560 13]

  DOMAIN_type :

  box

  NCO :

  netCDF Operators version 4.9.1 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco)

  description :

  ocean T grid variables

  history :

  Fri Jul 22 15:59:58 2022: ncks -4 -L 1 SEDNA-DELTA_1d_gridS_201501-201501_NOZIP_0000.nc /ccc/scratch/cont003/gen7420/talandel/SEDNA/SEDNA-DELTA-S/SPLIT/1d/2015/01/SEDNA-DELTA_1d_gridS_201501-201501_0000.nc Fri Jul 22 15:59:29 2022: ncrcat -n 31,2,1 SEDNA-DELTA_1d_gridS_0000_01.nc SEDNA-DELTA_1d_gridS_201501-201501_NOZIP_0000.nc

  ibegin :

  0

  jbegin :

  0

  name :

  /ccc/scratch/cont003/ra5563/talandel/ONGOING-RUNS/SEDNA-DELTA-XIOS.83/SEDNA-DELTA_1d_gridS

  ni :

  6560

  nj :

  13

  output_frequency :

  1d

  start_date :

  20090101

  timeStamp :

  2022-Jul-21 16:35:22 GMT

  title :

  ocean T grid variables

  uuid :

  9aef3543-35d6-4da0-a58a-f2c75b69d3a7

#5 Plotting
filename= plots.section(data,path=outputpath,filename=filename,save=savefig,cmap=cmap,clim=clim,clabel=clabel)
../results/SEDNA_DELTA_MONITOR/SEDNA_section_BFGS_Section_20150101-20151231.html starts plotting
../results/SEDNA_DELTA_MONITOR/SEDNA_section_BFGS_Section_20150101-20151231.html created

CPU times: user 4min 57s, sys: 52.7 s, total: 5min 49s
Wall time: 9min 38s