%matplotlib inline
import pandas as pd
import socket
host = socket.getfqdn()

from core import  load, zoom, calc, save,plots,monitor

Traceback (most recent call last):

  File /ccc/cont003/home/ra5563/ra5563/monitor/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3398 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Input In [1] in <cell line: 6>
    from core import  load, zoom, calc, save,plots,monitor

  File /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/core/__init__.py:9 in <module>
    from . import load

  File /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/core/load.py:126
    elif 'c-irene.mg1.tgcc.ccc.cea.fr' in host:
    ^
SyntaxError: invalid syntax

#reload funcs after updating ./core/*.py
import importlib
importlib.reload(load)
importlib.reload(zoom)
importlib.reload(calc)
importlib.reload(save)
importlib.reload(plots)
importlib.reload(monitor)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [2], in <cell line: 3>()
      1 #reload funcs after updating ./core/*.py
      2 import importlib
----> 3 importlib.reload(load)
      4 importlib.reload(zoom)
      5 importlib.reload(calc)

NameError: name 'load' is not defined

If you submit the job with job scheduler, above

below are list of enviroment variable one can pass

%env local='2"

local : if True run dask local cluster, if not true, put number of workers setted in the 'local' if no 'local ' given, local will be setted automatically to 'True'

%env ychunk='2'

%env tchunk='2'

controls chunk. 'False' sets no modification from original netcdf file's chunk.

ychunk=10 will group the original netcdf file to 10 by 10

tchunk=1 will chunk the time coordinate one by one

%env file_exp=

'file_exp': Which 'experiment' name is it?

. this corresopnds to intake catalog name without path and .yaml

%env year=

for Validation, this correspoinds to path/year/month 's year

for monitoring, this corresponids to 'date' having * means do all files in the monitoring directory

setting it as 0[0-9] &1[0-9]& *[2-3][0-9], the job can be separated in three lots.

%env month=

for monitoring this corresponds to file path path-XIOS.{month}/

#

%env control=FWC_SSH

name of control file to be used for computation/plots/save/

AWTD.sh M_AWTMD

Ice_quant_flux.sh M_Fluxnet M_Ice_quantities

FWC_SSH.sh M_FWC_2D M_FWC_integrals M_FWC_SSH M_SSH_anomaly

IceClim.sh M_IceClim M_IceConce M_IceThick

M_Mean_temp_velo M_MLD_2D M_Mooring M_Sectionx M_Sectiony

%env save= proceed saving? True or False , Default is setted as True

%env plot= proceed plotting? True or False , Default is setted as True

%env calc= proceed computation? or just load computed result? True or False , Default is setted as True

%env save=False

%env lazy=False

For debugging this cell can help

%env file_exp=SEDNA_DELTA_MONITOR %env year=2012 %env month=01

0[1-2]

%env ychunk=10 %env ychunk=False %env save=False %env plot=True %env calc=True # %env lazy=False

False

%env control=M_Fluxnet

M_Sectiony ok with ychunk=False local=True lazy=False

%%time
# 'savefig': Do we save output in html? or not. keep it true. 
savefig=True
client,cluster,control,catalog_url,month,year,daskreport,outputpath = load.set_control(host)
!mkdir -p $outputpath
!mkdir -p $daskreport
client

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
File <timed exec>:3, in <module>

NameError: name 'load' is not defined

read plotting information from a csv file

df=load.controlfile(control)
#Take out 'later' tagged computations
#df=df[~df['Value'].str.contains('later')]
df

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 df=load.controlfile(control)
      2 #Take out 'later' tagged computations
      3 #df=df[~df['Value'].str.contains('later')]
      4 df

NameError: name 'load' is not defined

Computation starts here

Each computation consists of

1. Load NEMO data set
2. Zoom data set
3. Compute (or load computed data set)
4. Save
5. Plot
6. Close

%%time
import os
calcswitch=os.environ.get('calc', 'True') 
lazy=os.environ.get('lazy','False' )
loaddata=((df.Inputs != '').any()) 
print('calcswitch=',calcswitch,'df.Inputs != nothing',loaddata, 'lazy=',lazy)
data = load.datas(catalog_url,df.Inputs,month,year,daskreport,lazy=lazy) if ((calcswitch=='True' )*loaddata) else 0 
data

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
File <timed exec>:4, in <module>

NameError: name 'df' is not defined

%%time
monitor.auto(df,data,savefig,daskreport,outputpath,file_exp='SEDNA'
            )

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
File <timed eval>:1, in <module>

NameError: name 'monitor' is not defined