%matplotlib inline
import pandas as pd
import socket
host = socket.getfqdn()
from core import load, zoom, calc, save,plots,monitor
#reload funcs after updating ./core/*.py
import importlib
importlib.reload(load)
importlib.reload(zoom)
importlib.reload(calc)
importlib.reload(save)
importlib.reload(plots)
importlib.reload(monitor)
<module 'core.monitor' from '/ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/core/monitor.py'>
If you submit the job with job scheduler; below are list of enviroment variable one can pass
local : if True run dask local cluster, if not true, put number of workers setted in the 'local' if no 'local ' given, local will be setted automatically to 'True'
%env ychunk='2', #%env tchunk='2'
controls chunk. 'False' sets no modification from original netcdf file's chunk.
ychunk=10 will group the original netcdf file to 10 by 10
tchunk=1 will chunk the time coordinate one by one
%env file_exp=
'file_exp': Which 'experiment' name is it? this corresopnds to intake catalog name without path and .yaml
#%env year=
for Validation, this correspoinds to path/year/month 's year for monitoring, this corresponids to 'date' having means do all files in the monitoring directory setting it as 0[0-9] &1[0-9]& [2-3][0-9], the job can be separated in three lots. For DELTA experiment, year corresponds to really 'year'
%env month=
for monitoring this corresponds to file path path-XIOS.{month}/
For DELTA experiment, year corresponds to really 'month'
proceed saving? True or False , Default is setted as True
proceed plotting? True or False , Default is setted as True
proceed computation? or just load computed result? True or False , Default is setted as True
save output file used for plotting
using kerchunked file -> False, not using kerhcunk -> True
name of control file to be used for computation/plots/save/ We have number of M_xxx.csv
Monitor.sh calls M_MLD_2D
and AWTD.sh, Fluxnet.sh, Siconc.sh, IceClim.sh, FWC_SSH.sh, Integrals.sh , Sections.sh
M_AWTMD
M_Fluxnet
M_Ice_quantities
M_IceClim M_IceConce M_IceThick
M_FWC_2D M_FWC_integrals M_FWC_SSH M_SSH_anomaly
M_Mean_temp_velo M_Mooring
M_Sectionx M_Sectiony
%%time
# 'savefig': Do we save output in html? or not. keep it true.
savefig=True
client,cluster,control,catalog_url,month,year,daskreport,outputpath = load.set_control(host)
!mkdir -p $outputpath
!mkdir -p $daskreport
client
local 20 local false ,workers 20 using host= irene6113.c-irene.mg1.tgcc.ccc.cea.fr starting dask cluster on local= False workers 20 10000000000 mg1: rome dask jobqueue rome, slurmcluster starting This code is running on irene6113.c-irene.mg1.tgcc.ccc.cea.fr using SEDNA_DELTA_MONITOR file experiment, read from ../lib/SEDNA_DELTA_MONITOR.yaml on year= 2012 on month= 0[3-4] outputpath= ../results/SEDNA_DELTA_MONITOR/ daskreport= ../results/dask/6470458irene6113.c-irene.mg1.tgcc.ccc.cea.fr_SEDNA_DELTA_MONITOR_0[3-4]M_MLD_2D/ CPU times: user 2.06 s, sys: 494 ms, total: 2.55 s Wall time: 2min 20s
Client-dfe34feb-18f6-11ed-aae6-080038b940f3
Connection method: Cluster object | Cluster type: dask_jobqueue.SLURMCluster |
Dashboard: http://10.131.14.172:8787/status |
338c76d7
Dashboard: http://10.131.14.172:8787/status | Workers: 28 |
Total threads: 252 | Total memory: 428.40 GiB |
Scheduler-d18baa01-c3c3-4ae7-9621-9b440530642d
Comm: tcp://10.131.14.172:46835 | Workers: 28 |
Dashboard: http://10.131.14.172:8787/status | Total threads: 252 |
Started: 2 minutes ago | Total memory: 428.40 GiB |
Comm: tcp://10.131.5.69:37537 | Total threads: 9 |
Dashboard: http://10.131.5.69:41378/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:38870 | |
Local directory: /tmp/dask-worker-space/worker-3h37v2no |
Comm: tcp://10.131.5.69:44971 | Total threads: 9 |
Dashboard: http://10.131.5.69:41224/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:37366 | |
Local directory: /tmp/dask-worker-space/worker-4qdg47gk |
Comm: tcp://10.131.5.69:37677 | Total threads: 9 |
Dashboard: http://10.131.5.69:37422/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:34535 | |
Local directory: /tmp/dask-worker-space/worker-tkh3tbxv |
Comm: tcp://10.131.5.69:40901 | Total threads: 9 |
Dashboard: http://10.131.5.69:44781/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:43270 | |
Local directory: /tmp/dask-worker-space/worker-sd9_igj5 |
Comm: tcp://10.131.5.69:32834 | Total threads: 9 |
Dashboard: http://10.131.5.69:39155/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:33284 | |
Local directory: /tmp/dask-worker-space/worker-q0m0_zhi |
Comm: tcp://10.131.5.69:37864 | Total threads: 9 |
Dashboard: http://10.131.5.69:36554/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:42992 | |
Local directory: /tmp/dask-worker-space/worker-2fap_lt5 |
Comm: tcp://10.131.5.69:40744 | Total threads: 9 |
Dashboard: http://10.131.5.69:44982/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:36258 | |
Local directory: /tmp/dask-worker-space/worker-5hew4hy6 |
Comm: tcp://10.131.5.69:46830 | Total threads: 9 |
Dashboard: http://10.131.5.69:43542/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:43578 | |
Local directory: /tmp/dask-worker-space/worker-xixa56u3 |
Comm: tcp://10.131.5.69:45294 | Total threads: 9 |
Dashboard: http://10.131.5.69:37229/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:42300 | |
Local directory: /tmp/dask-worker-space/worker-or8fcp8b |
Comm: tcp://10.131.5.69:33489 | Total threads: 9 |
Dashboard: http://10.131.5.69:44149/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:43312 | |
Local directory: /tmp/dask-worker-space/worker-7g988o_q |
Comm: tcp://10.131.5.69:40737 | Total threads: 9 |
Dashboard: http://10.131.5.69:46717/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:42698 | |
Local directory: /tmp/dask-worker-space/worker-bgdqr9nk |
Comm: tcp://10.131.5.69:45732 | Total threads: 9 |
Dashboard: http://10.131.5.69:33305/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:44042 | |
Local directory: /tmp/dask-worker-space/worker-_01tqtdd |
Comm: tcp://10.131.5.69:40774 | Total threads: 9 |
Dashboard: http://10.131.5.69:43522/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:34699 | |
Local directory: /tmp/dask-worker-space/worker-p4h1496g |
Comm: tcp://10.131.5.69:36172 | Total threads: 9 |
Dashboard: http://10.131.5.69:46237/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.69:44960 | |
Local directory: /tmp/dask-worker-space/worker-a9n5j9eu |
Comm: tcp://10.131.5.68:45922 | Total threads: 9 |
Dashboard: http://10.131.5.68:33582/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:38200 | |
Local directory: /tmp/dask-worker-space/worker-lh32zt_m |
Comm: tcp://10.131.5.68:43780 | Total threads: 9 |
Dashboard: http://10.131.5.68:42486/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:36045 | |
Local directory: /tmp/dask-worker-space/worker-cr0z4v15 |
Comm: tcp://10.131.5.68:41343 | Total threads: 9 |
Dashboard: http://10.131.5.68:34808/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:35709 | |
Local directory: /tmp/dask-worker-space/worker-003aqsjb |
Comm: tcp://10.131.5.68:34591 | Total threads: 9 |
Dashboard: http://10.131.5.68:38717/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:41217 | |
Local directory: /tmp/dask-worker-space/worker-kg86hrf7 |
Comm: tcp://10.131.5.68:43906 | Total threads: 9 |
Dashboard: http://10.131.5.68:39270/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:33489 | |
Local directory: /tmp/dask-worker-space/worker-x07kyxhx |
Comm: tcp://10.131.5.68:40097 | Total threads: 9 |
Dashboard: http://10.131.5.68:37956/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:43555 | |
Local directory: /tmp/dask-worker-space/worker-chjvr6iy |
Comm: tcp://10.131.5.68:43201 | Total threads: 9 |
Dashboard: http://10.131.5.68:44993/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:44594 | |
Local directory: /tmp/dask-worker-space/worker-pd6qeq4h |
Comm: tcp://10.131.5.68:44799 | Total threads: 9 |
Dashboard: http://10.131.5.68:38059/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:42923 | |
Local directory: /tmp/dask-worker-space/worker-8cf5r_1u |
Comm: tcp://10.131.5.68:44956 | Total threads: 9 |
Dashboard: http://10.131.5.68:40650/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:43277 | |
Local directory: /tmp/dask-worker-space/worker-iwfhs6qu |
Comm: tcp://10.131.5.68:40494 | Total threads: 9 |
Dashboard: http://10.131.5.68:45447/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:40740 | |
Local directory: /tmp/dask-worker-space/worker-imughd8c |
Comm: tcp://10.131.5.68:35472 | Total threads: 9 |
Dashboard: http://10.131.5.68:37283/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:34958 | |
Local directory: /tmp/dask-worker-space/worker-037xifbh |
Comm: tcp://10.131.5.68:38018 | Total threads: 9 |
Dashboard: http://10.131.5.68:40111/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:38573 | |
Local directory: /tmp/dask-worker-space/worker-o5ms0azy |
Comm: tcp://10.131.5.68:35006 | Total threads: 9 |
Dashboard: http://10.131.5.68:37265/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:40934 | |
Local directory: /tmp/dask-worker-space/worker-9phjt_st |
Comm: tcp://10.131.5.68:39180 | Total threads: 9 |
Dashboard: http://10.131.5.68:39430/status | Memory: 15.30 GiB |
Nanny: tcp://10.131.5.68:37740 | |
Local directory: /tmp/dask-worker-space/worker-kq7el8uz |
df=load.controlfile(control)
#Take out 'later' tagged computations
#df=df[~df['Value'].str.contains('later')]
df
Value | Inputs | Equation | Zone | Plot | Colourmap | MinMax | Unit | Oldname | Unnamed: 10 | |
---|---|---|---|---|---|---|---|---|---|---|
MLD_2D | gridT-2D.samldr1_1 | data.samldr1_1 | ALL | maps | Blues | (0,80) | m | M-5 |
Each computation consists of
%%time
import os
calcswitch=os.environ.get('calc', 'True')
lazy=os.environ.get('lazy','False' )
loaddata=((df.Inputs != '').any())
print('calcswitch=',calcswitch,'df.Inputs != nothing',loaddata, 'lazy=',lazy)
data = load.datas(catalog_url,df.Inputs,month,year,daskreport,lazy=lazy) if ((calcswitch=='True' )*loaddata) else 0
data
calcswitch= True df.Inputs != nothing True lazy= False ../lib/SEDNA_DELTA_MONITOR.yaml using param_xios reading ../lib/SEDNA_DELTA_MONITOR.yaml using param_xios reading <bound method DataSourceBase.describe of sources: param_xios: args: combine: nested concat_dim: y urlpath: /ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc xarray_kwargs: compat: override coords: minimal data_vars: minimal parallel: true description: SEDNA NEMO parameters from MPI output nav_lon lat fails driver: intake_xarray.netcdf.NetCDFSource metadata: catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/ > {'name': 'param_xios', 'container': 'xarray', 'plugin': ['netcdf'], 'driver': ['netcdf'], 'description': 'SEDNA NEMO parameters from MPI output nav_lon lat fails', 'direct_access': 'forbid', 'user_parameters': [{'name': 'path', 'description': 'file coordinate', 'type': 'str', 'default': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/MESH/SEDNA_mesh_mask_Tgt_20210423_tsh10m_L1/param'}], 'metadata': {}, 'args': {'urlpath': '/ccc/work/cont003/gen7420/odakatin/CONFIGS/SEDNA/SEDNA-I/SEDNA_Domain_cfg_Tgt_20210423_tsh10m_L1/param_f32/x_*.nc', 'combine': 'nested', 'concat_dim': 'y'}} 0 read gridT-2D ['samldr1_1'] lazy= False using load_data_xios_kerchunk reading gridT-2D using load_data_xios_kerchunk reading <bound method DataSourceBase.describe of sources: data_xios_kerchunk: args: consolidated: false storage_options: fo: file:////ccc/cont003/home/ra5563/ra5563/catalogue/DELTA/20120[3-4]/gridT-2D_0[0-5][0-9][0-9].json target_protocol: file urlpath: reference:// description: CREG025 NEMO outputs from different xios server in kerchunk format driver: intake_xarray.xzarr.ZarrSource metadata: catalog_dir: /ccc/work/cont003/gen7420/odakatin/monitor-sedna/notebook/../lib/ > took 2639.714877843857 seconds 0 merging gridT-2D ['samldr1_1'] param nav_lat will be included in data param mask2d will be included in data param nav_lon will be included in data ychunk= 10 calldatas_y_rechunk sum_num (13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12) start rechunking with (130, 122, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 120, 48) end of y_rechunk CPU times: user 11min 56s, sys: 12min 27s, total: 24min 24s Wall time: 44min 21s
<xarray.Dataset> Dimensions: (t: 31, y: 6540, x: 6560) Coordinates: nav_lat (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray> nav_lon (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray> * t (t) object 2012-03-01 12:00:00 ... 2012-03-31 12:00:00 * y (y) int64 1 2 3 4 5 6 7 8 ... 6534 6535 6536 6537 6538 6539 6540 * x (x) int64 1 2 3 4 5 6 7 8 ... 6554 6555 6556 6557 6558 6559 6560 mask2d (y, x) bool dask.array<chunksize=(130, 6560), meta=np.ndarray> Data variables: samldr1_1 (t, y, x) float32 dask.array<chunksize=(1, 130, 6560), meta=np.ndarray> Attributes: (12/26) CASE: DELTA CONFIG: SEDNA Conventions: CF-1.6 DOMAIN_dimensions_ids: [2, 3] DOMAIN_halo_size_end: [0, 0] DOMAIN_halo_size_start: [0, 0] ... ... nj: 13 output_frequency: 1d start_date: 20090101 timeStamp: 2022-Jan-19 12:40:29 GMT title: ocean T grid variables uuid: 0c903256-6713-4218-8157-b00d89bc0866
%%time
monitor.auto(df,data,savefig,daskreport,outputpath,file_exp='SEDNA'
)
#calc= True #save= False #plot= True Value='MLD_2D' Zone='ALL' Plot='maps' cmap='Blues' clabel='m' clim= (0, 80) outputpath='../results/SEDNA_DELTA_MONITOR/' nc_outputpath='../nc_results/SEDNA_DELTA_MONITOR/' filename='SEDNA_maps_ALL_MLD_2D' data=monitor.optimize_dataset(data) #3 Start computing data= data.samldr1_1 monitor.optimize_dataset(data) add optimise here once otimise can recognise
<xarray.DataArray 'samldr1_1' (t: 31, y: 6540, x: 6560)> dask.array<rechunk-merge, shape=(31, 6540, 6560), dtype=float32, chunksize=(1, 130, 6560), chunktype=numpy.ndarray> Coordinates: nav_lat (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray> nav_lon (y, x) float32 dask.array<chunksize=(130, 6560), meta=np.ndarray> * t (t) object 2012-03-01 12:00:00 ... 2012-03-31 12:00:00 * y (y) int64 1 2 3 4 5 6 7 8 ... 6534 6535 6536 6537 6538 6539 6540 * x (x) int64 1 2 3 4 5 6 7 8 ... 6554 6555 6556 6557 6558 6559 6560 mask2d (y, x) bool dask.array<chunksize=(130, 6560), meta=np.ndarray> Attributes: cell_methods: time: mean (interval: 40 s) interval_operation: 40 s interval_write: 1 d long_name: Mixed Layer Depth (dsigma = 0.01 wrt 1m) online_operation: average standard_name: ocean_mixed_layer_thickness_defined_by_sigma_theta units: m
#5 Plotting filename= plots.maps(data,path=outputpath,filename=filename,save=savefig,cmap=cmap,clim=clim,clabel=clabel) ../results/SEDNA_DELTA_MONITOR/SEDNA_maps_ALL_MLD_2D_20120301-20120331.html starts plotting
/ccc/cont003/home/ra5563/ra5563/monitor/lib/python3.10/site-packages/geoviews/operation/projection.py:99: ShapelyDeprecationWarning: __len__ for multi-part geometries is deprecated and will be removed in Shapely 2.0. Check the length of the `geoms` property instead to get the number of parts of a multi-part geometry. if proj_geom.geom_type == 'GeometryCollection' and len(proj_geom) == 0:
../results/SEDNA_DELTA_MONITOR/SEDNA_maps_ALL_MLD_2D_20120301-20120331.html created
CPU times: user 10min 36s, sys: 1min 47s, total: 12min 24s Wall time: 15min 20s