simplaceUtil

Gunther Krauss

July 27, 2023

1 Exploring solutions

1.1 Fetch information from solution

library(simplaceUtil)

We can read a solution xml file via getSolutionFromFile and get its content as an xml_document object. From the xml object we can get the components, user variables and links between components as dataframes.

solfile <- system.file("solution","Complete.sol.xml",
                       package="simplaceUtil")

sol <- getSolutionFromFile(solfile)
components <- getComponents(sol)
variables <- getUserVariables(sol)
links <- getLinks(sol, components)

As a shorthand we can use getElementsFromSolutionFile, which returns the xml object as well as the dataframes in a list.

el <- getElementsFromSolutionFile(solfile)
sol <- el$solution
components <- el$components
variables <- el$variables
links <- el$links

The components dataframe contains not only sim components, but also interfaces, resources and outputs.

components[c(1:3,30:31,42:44),c("id","type","subtype","nr")]
id type subtype nr
1 CURRENT var var 1
2 variables var var 2
3 compare interface CSV 3
30 crop resource resource 30
31 ndemand resource resource 31
42 init_transform resource transform 42
43 DefaultManagement simcomponent mgm 43
44 EvapTranDemand simcomponent normal 44

The links dataframe contains the links between resources, simcomponents, outputs, but also dependencies via key variables or via variables used in rules. Relationships between resources and transformers are included too.

links[c(1:2,23,300,313,322,325),]
from name to rel
1 CURRENT DOY DefaultManagement value
2 phenology emergencedoy DefaultManagement value
23 DefaultManagement DoHarvest LintulPhenology value
300 weatherfile - weather storage
313 soil - soil_transform data
322 variables vSoiltype soil key
325 DefaultManagement DoSow crop rule

User variables show the values as defined in the solution. Notice that user variables can be changed when running a solution in project mode.

variables[c(1:5,18:20),]
id value unit datatype kind
1 vCLIMATEZONE 00002 NA CHAR var
2 vSoiltype sandy loam NA CHAR var
3 vTempLimit 35 NA DOUBLE var
4 vTempSlopeAboveLimit 7 NA DOUBLE var
5 vBaseLUE 3.0 NA DOUBLE var
18 vCO2StartYear 2000 NA INT dyn
19 startdate 01.01.1991 NA DATE dyn
20 enddate 31.12.2004 NA DATE dyn

Filenames for interfaces may use placeholder variables like _WORKDIR_ or user defined variables (e.g. vFilename)

head(components[,c("id","ref")])
id ref
CURRENT SYSTEM
variables User
compare ${WORKDIR}/gk/data/lintul/compare/compare_Complete.csv
weatherfile ${vFileName}
phenologyfile ${WORKDIR}/gk/data/lintul/LintulPhenologyObservations.csv
cropfile ${WORKDIR}/gk/data/lintul/CropPropertiesLintulWaterstress.xml

With replaceVariablesWithValues we can replace the placeholder with the values from the user value dataframe. We can supply additional values for system variables like _WORKDIR_.

components$ref <- replaceVariablesWithValues(
  components$ref,
  variables, 
  additional=c("_WORKDIR_"="~"))
head(components[,c("id","ref")])
id ref
CURRENT SYSTEM
variables User
compare ~/gk/data/lintul/compare/compare_Complete.csv
weatherfile ~/gk/data/lintul/00002Weather.txt
phenologyfile ~/gk/data/lintul/LintulPhenologyObservations.csv
cropfile ~/gk/data/lintul/CropPropertiesLintulWaterstress.xml

1.3 Get Metadata about solution

getSolutionInfoAsDataframe gives information about the solutions content (simcomponents) as well as the solution itself (filename, modification date and possibly simulation experiment information derived from folder structure).

wd <- system.file(package="simplaceUtil")
solfiles <- list.files(wd,pattern=".sol.xml",recursive=TRUE,full.names = TRUE)

df <- do.call(rbind,
        lapply(solfiles,getSolutionInfoAsDataframe,workdir=wd))
df$file <- basename(df$file)
df[df$type=="simcomponent",c("id","file")] 
id file
43 DefaultManagement Complete.sol.xml
44 EvapTranDemand Complete.sol.xml
45 Co2InfluenceOnLUE Complete.sol.xml
46 Co2InfluenceOnTranspiration Complete.sol.xml
47 LintulPhenology Complete.sol.xml
48 VernalisationAndPhotoresponse Complete.sol.xml
49 SlimRoots Complete.sol.xml
50 SlimWater Complete.sol.xml
51 SlimNitrogen Complete.sol.xml
52 LintulWaterStress Complete.sol.xml
53 LintulPartitioning Complete.sol.xml
54 LintulBiomass Complete.sol.xml
55 NitrogenDemand Complete.sol.xml
56 SoilCN Complete.sol.xml
57 SnowCoverCalculator Complete.sol.xml
58 STMPsimCalculator Complete.sol.xml
78 DefaultManagement Yield.sol.xml
79 LintulPheno Yield.sol.xml
80 VernalisationAndPhotoresponse Yield.sol.xml
81 LintulPartitioning Yield.sol.xml
82 LintulBiomass Yield.sol.xml

2 Visualising solution structure

A graph showing components and links can be created from a solution file:

graph <- solutionToGraph(solfile)
DiagrammeR::render_graph(graph)

We can restrict the graph to specific component types and the neighborhood of selected components:

subgraph <- getNeighborhood(graph, c("SlimWater","SlimRoots"),
                            distance=1, type="simcomponent")
DiagrammeR::render_graph(subgraph)

3 Modifying solutions

3.1 Sensitivity analysis and calibration

A solution can be modified on the fly to meet requirements for sensitivity analysis or calibration by

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)
newsol <- sol |>
  removeNonMemoryOutputs() |>
  addMemoryOutput("Yields",frequence = "YEARLY") |>
  addOutputVariable("Yields","Year","CURRENT.YEAR","INT") |>
  addOutputVariable("Yields", "ClimateZone","vClimateZone","CHAR") |>
  addOutputVariable("Yields", "Yield","LintulBiomass.sWSO","DOUBLE","MAX") |>
  addUserVariable("vLUE",3.0,"DOUBLE") |>
  addUserVariable("vSLA",0.02,"DOUBLE") |>
  addUserVariable("vRGRL",0.009,"DOUBLE") |>
  replaceVariable("crop.LUE","vLUE") |>
  replaceVariable("crop.SLA","vSLA") |>
  replaceVariable("crop.RGRL","vRGRL")

Let’s output the beginning of the solution as text:

soltext <- getTextFromSolution(newsol)
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE solution PUBLIC "-//SIMPLACE/DTD SOL 1.0//EN" "http://simplace.net/dtd/SimSolution_4_1.dtd">
<solution version="4.1" check="LAZY">
  <description title="Yield" author="XXX_XXX" qualitylevel="I">WIKI_START
=== Purpose ===
Example solution for simplaceUtil package.
WIKI_END 
 * removed non-memory outputs for interfaces yieldout, sumyieldout 
 * added memory output  Yields 
 * added output variable Year  to output  Yields  
 * added output variable ClimateZone  to output  Yields  
 * added output variable Yield  to output  Yields  
 * added user variable vLUE with value 3 
 * added user variable vSLA with value 0.02 
 * added user variable vRGRL with value 0.009 
 * replaced crop.LUE with vLUE 
 * replaced crop.SLA with vSLA 
 * replaced crop.RGRL with vRGRL</description>
  <variables>
    <var id="vTempLimit" datatype="DOUBLE">35</var>
    <var id="vTempSlopeAboveLimit" datatype="DOUBLE">7</var>
    <var id="vCropName" datatype="CHAR">winter wheat</var>
    <var id="vBaseLUE" datatype="DOUBLE">3.0</var>
    <var id="vBaseTempBeforeAnt" datatype="DOUBLE">3.0</var>
    <var id="vBaseTempAfterAnt" datatype="DOUBLE">3.0</var>
    <var id="startdate" datatype="DATE">01.01.1991</var>
    <var id="enddate" datatype="DATE">31.12.2004</var>
    <var id="vClimateZone" datatype="CHAR">C02</var>
    <var id="vLUE" datatype="DOUBLE">3</var>
    <var id="vSLA" datatype="DOUBLE">0.02</var>
    <var id="vRGRL" datatype="DOUBLE">0.009</var>
  </variables>
  <interfaces defaul

Notice that the modifications are logged in the description section of the new solution.

The solution can be saved as xml file:

solution_modified <- "simulation/solution/YieldModified.sol.xml"  
writeSolutionToFile(newsol, solution_modified) 

3.2 Changing order of SimComponents

As the order of SimComponents can have a considerable impact, one can generate random orders of components and run the reordered solutions to analyse the impact.

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)

cmp <- getComponents(sol)
cmp[cmp$type=="simcomponent",c("id","subtype","nr")]
id subtype nr
14 DefaultManagement mgm 14
15 LintulPheno normal 15
16 VernalisationAndPhotoresponse normal 16
17 LintulPartitioning normal 17
18 LintulBiomass normal 18

We want to swap the components 2 to 5

toswap <- 2:5
swapped <- sample(toswap, length(toswap))
swapped
## [1] 5 3 4 2
newsol <- swapComponents(sol,swapped)
cmp <- getComponents(newsol)
cmp[cmp$type=="simcomponent",c("id","subtype","nr")]
id subtype nr
14 DefaultManagement mgm 14
15 LintulBiomass normal 15
16 VernalisationAndPhotoresponse normal 16
17 LintulPartitioning normal 17
18 LintulPheno normal 18

The order of the SimComponent is now different.

3.3 Adding timing SimComponent

There is a special SimComponent that records the processing time of SimComponents.

To add timing to a solution, one has to

This can be done for every solution automatically via addTimingSimComponent.

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)

newsol <- addTimingSimComponent(sol, componentlist=c("LintulBiomass","LintulPheno"))
cmp <- getComponents(newsol)
cmp[cmp$type=="simcomponent" | grepl("timing",cmp$id),c("id","type","subtype","nr")]
id type subtype nr
8 automatic_timing_interface interface MEMORY 8
15 DefaultManagement simcomponent mgm 15
16 LintulPheno simcomponent normal 16
17 VernalisationAndPhotoresponse simcomponent normal 17
18 LintulPartitioning simcomponent normal 18
19 LintulBiomass simcomponent normal 19
20 AutomaticTiming simcomponent normal 20
23 automatic_timing_output output output 23

The new solution has now an additional SimComponent AutomaticTiming as well as an additional output together with it’s interface.

4 Standard transformation and visualisation of outputs

4.1 Transform output data

With parseDate the standard Date column of Simplace output files is converted to Date.

outfile <- system.file("output","water.csv",
                       package="simplaceUtil")
water <- read.csv(outfile, header=TRUE, sep=";")
water <- parseDate(water)

By default the simplace outputs are in wide format, e.g. each layer is in an own column. We can transform the dataframe from wide format to long format, by putting values from all layers into one column and distinguish them by an extra column layer.

water[1:4,c(2,9:11,49)]
CURRENT.DATE MobileWater_1 MobileWater_2 MobileWater_3 RetainedWater_1
1991-07-18 0 0.5214681 1.197492 3.402230
1991-07-19 0 0.0000000 0.000000 2.045055
1991-07-20 0 0.0000000 0.000000 1.584985
1991-07-21 0 0.0000000 0.000000 2.906192
water_long <- transformLayeredData(water)
water_long[c(1:3,41:43,81:83),c(2,9:11)]
CURRENT.DATE layer MobileWater RetainedWater
1991-07-18 1 0.0000000 3.402230
1991-07-18 2 0.5214681 5.400000
1991-07-18 3 1.1974920 5.400000
1991-07-19 1 0.0000000 2.045055
1991-07-19 2 0.0000000 4.271450
1991-07-19 3 0.0000000 5.305210
1991-07-20 1 0.0000000 1.584985
1991-07-20 2 0.0000000 4.396188
1991-07-20 3 0.0000000 5.041876

4.2 Visualise output variables

Multiple scalar values can be plotted in a common diagram.

plotScalarOutput(water,"CURRENT.DATE",c("Evaporation","Transpiration"))

Default plot for layered values.

plotLayeredOutput(water,"RetainedWater")

5 Simple GUI as Shiny App

The package includes a Shiny App that runs locally within a webbrowser.

runSimplaceGuiApp()

We can interactively explore solutions by filtering and displaying solution graphs.

We can also use a local Simplace installation to run the simulation defined by the solution and plot some of the output variables.

6 Assistance for creating new Solutions (simulation configuration)

6.1 Creating XML stubs for resources

We can create the interface (location and format) and resource (structure) description for a solution as XML stubs from given input data.

Data can be given in CSV or XML format. The structure of the data has to be compatible with Simplace.

stubs <- createCsvResourceStubs(
  filename = system.file("input","weather.csv", package="simplaceUtil"),
  id = "weather",
  sep =",",
  keyvals = c("CURRENT.DATE" = "Date")
)
cat(stubs$interface)
## <interface id="weather_file" type="CSV">
##   <poolsize>100</poolsize>
##   <divider>,</divider
##   <filename>/tmp/Rtmp0ywPU1/Rinst1733189a2167a/simplaceUtil/input/weather.csv</filename>
## </interface>
cat(stubs$resource)
## <resource id="weather" interface="weather_file" frequence="DAILY">
##   <res id="DOY" datatype="INT", unit=""/>
##   <res id="Date" key="CURRENT.DATE" datatype="CHAR", unit=""/>
##   <res id="Rain" datatype="DOUBLE", unit=""/>
##   <res id="Radiation" datatype="DOUBLE", unit=""/>
##   <res id="Tmean" datatype="DOUBLE", unit=""/>
##   <res id="RelHum" datatype="DOUBLE", unit=""/>
## </resource>

The function tries to determine the datatype from the data. We have to check and possibly correct the inferred datatype (e.g. replacing ‘CHAR’ by ‘DATE’ or ‘INT’ by ‘DOUBLE’ or ‘BOOLEAN’).

7 Assistance for developing new SimComponents

7.1 Creating code stubs for SimVariables

When creating a new SimComponent, we can put all SimVariables (constants, inputs, states, rates, outputs) in a CSV table.

varfile <- system.file("variables","translocation.csv",package="simplaceUtil")
vartable <- read.csv(varfile, header=TRUE, sep=";")
vartable[c(4,8:9),]
contenttype id description datatype unit min max default
4 constant FruitingEfficiency Number of grains per storage organs mass Double g^-1 0 NA .1
8 state WEA Weight of ears Double g m^-2 0 NA 0
9 rate RWEA Change rate of ears Double g m^-2 d^-1 NA NA 0

With the function createCodeStubsForSimVariables we can generate so called boilerplate code for Java as well as XML stubs for solutions.

stubs <- createCodeStubsForSimVariables(varfile, "BMTransloc", sep=";")
names(stubs)
## [1] "JavaFields"            "JavaVariables"         "JavaInit"             
## [4] "ParametersXML"         "ResourceParametersXML" "ResourceInputsXML"    
## [7] "ComponentInputsXML"    "OutputsXML"

When we specify an output folder, the stubs are written to files.

stubs <- createCodeStubsForSimVariables(varfile, "BMTransloc", sep=";",
                                        outfolder="translocation_stubs")

7.2 Example code stubs

Defining fields in Java (stubs$JavaFields):

private FWSimVariable<Double> cFruitingEfficiency;
private FWSimVariable<Double> rRWEA;
private FWSimVariable<Double> sWEA;

Creating the SimVariables (stubs$JavaVariables):

addVariable(FWSimVariable.createSimVariable("cFruitingEfficiency", "Number of grains per storage organs mass", DATA_TYPE.DOUBLE, CONTENT_TYPE.constant, "g^-1",  0d, null, .1d, this));
addVariable(FWSimVariable.createSimVariable("rRWEA", "Change rate of ears", DATA_TYPE.DOUBLE, CONTENT_TYPE.rate, "g m^-2 d^-1", null, null, 0d, this));
addVariable(FWSimVariable.createSimVariable("sWEA", "Weight of ears", DATA_TYPE.DOUBLE, CONTENT_TYPE.state, "g m^-2",  0d, null, 0d, this));

Default initialisation of states and rates (sic!) (stubs$JavaInit):

rRWEA.setDefaultValue();
sWEA.setDefaultValue();

XML stub for linking inputs (stubs$ComponentInputsXML):

<input id="cTranslocationFraction" source="bmtransloc_parameters.TranslocationFraction" />
<input id="cFruitingEfficiency" source="bmtransloc_parameters.FruitingEfficiency" />
<input id="iEarPartitioningReductionFactor" source="bmtransloc_inputs.EarPartitioningReductionFactor" />

XML stubs for parameter file (stubs$ParametersXML) and coresponding resource section in solution (stubs$ResourceParametersXML):

<parameter id="EarsPartitioningTableDVS" datatype="DOUBLEARRAY" unit="" description="DVS for fraction of above-ground dry matter to ears">
      <value>0</value>
      <value>1</value>
      <value>2</value>
  </parameter>
<parameter id="EarsPartitioningTableFraction" datatype="DOUBLEARRAY" unit="" description="Fraction of above-ground dry matter to ears as function of DVS">
      <value>.0</value>
      <value>.1</value>
      <value>.0</value>
  </parameter>
<parameter id="TranslocationFraction" datatype="DOUBLE" unit="" description="Fraction of stems and leaves biomass  that is reserved for translocation at anthesis">0.2</parameter>
<parameter id="FruitingEfficiency" datatype="DOUBLE" unit="g^-1" description="Number of grains per storage organs mass">.1</parameter>
<res id="EarsPartitioningTableDVS" datatype="DOUBLEARRAY" unit="" description="DVS for fraction of above-ground dry matter to ears" />
<res id="EarsPartitioningTableFraction" datatype="DOUBLEARRAY" unit="" description="Fraction of above-ground dry matter to ears as function of DVS" />
<res id="TranslocationFraction" datatype="DOUBLE" unit="" description="Fraction of stems and leaves biomass  that is reserved for translocation at anthesis" />
<res id="FruitingEfficiency" datatype="DOUBLE" unit="g^-1" description="Number of grains per storage organs mass" />