PEST IES run post-mortem: select and run one realization

[gmerritt123]

Hoping this is the right place to ask. Does a utility exist that takes a post-run pest-ies directory (e.g. a directory containing a .par.csv output from a particular IES iteration and all the requisite pest ies input files, .pst, .tpls etc.) and a user-selected realization number and run the model with those parameters? Essentially I want to run a one-off realization after running pest IES and do more analysis on the output. Thanks!

[mnfienen]

hey @gmerritt123 - I would say a robust recipe is to:

1. make a copy of your run directory
2. read in the Pst file using pst=pyemu.Pst(<filename>)
3. read in your realizations using pandas
4. update the pstparameter_data parval1 column using the parameters from the chosen realization
5. set noptmax=0 like pst.control_data['noptmax']=0
6. save out to a new file pst.write('newreality.pst', version=2)
7. run the new pst file at the command line

I know @jtwhite79 has also been working on some advanced functionality to automate some of the loading of results that might help as well.

Hope that helps!

Cheers

[jtwhite79]

with the latest pyemu, you can do @mnfienen steps 2,3, and 4 like this:
2) pst=pyemu.Pst(<path_to_pst_in_master_dir>,result_dir=<master_dir>)
3) rvals = pst.ies.paren.loc[<some_pandas_magic>,pst.par_names] (pst.ies.paren is a multi-indexed df of parameter ensembles from all iterations)
4) pst.parameter_data["parval1"] = rvals

[gmerritt123]

Thanks for the quick response guys, I more or less implemented @mnfienen 's steps, just all outside of pyemu-land ( because didn't know where to look until @jtwhite79 replied just now :-D )

import pandas as pd
import numpy as np
import subprocess
import os

def run_one_ensemble_member(wdir,pest_nm,iter_num,real_num,pest_exe='pestpp-ies',run=False):
    '''function that updates the initial par values with a particular ensemble member (real_num) from a particular iteration (iter_num).
    args:
            wdir = working directory (where all the pest files are)
            pest_nm = name of pest run (i.e. prefix in front of pest files)
            iter_num = iteration number to draw from
            real_num = realization number to use
            pest_exe (required if run == True) = name of pest executable in directory --> default pestpp-ies
            run (boolean) --> 
                If True, will update pst file to noptmax = 0, and call pest, which will run the model once. 
                If False will only update the pardata.csv.
    '''
    or_dir = os.getcwd()
    os.chdir(wdir)
    ens_file = pest_nm+'.'+str(iter_num)+'.par.csv'
    par_file = pest_nm+'.par_data.csv'
    pst_file = pest_nm+'.pst'
    
    #get parameters
    pdf = pd.read_csv(ens_file)
    pdf = pdf[pdf['real_name']==str(real_num)].iloc[0][1:].reset_index()
    pdf.columns = ['parnme','updVal']
    
    #update par data
    p = pd.read_csv(par_file).merge(pdf,how='inner',on='parnme')
    p['parval1'] = p['updVal'].copy()
    p = p.drop(columns='updVal')
    p.to_csv(par_file,index=False)
    
    if run == False:
        print('initial parameter values updated in '+par_file)
        print(pdf)
    elif run == True:
        with open(pst_file,'r') as f:
            rl = f.readlines()
            for i,l in enumerate(rl):
                if 'noptmax' in l:
                    rl[i] = l[:-3] +'0\n' 
                    break
        with open(pst_file,'w') as f:
            f.writelines(rl)
        print('pst update to 0 iterations')
        print('running model')
        subprocess.run(pest_exe+ ' '+pst_file)
    os.chdir(or_dir)
    
#usage
run_one_ensemble_member(wdir = r'C:\Projects\Orangeville_T3\OT3_Repo\Raven_PEST\manager'
                        ,pest_nm = 'Orangeville'
                        ,iter_num = 3
                        ,real_num = 2
                        ,run=True)