Skip to content

Commit e75b687

Browse files
BenChungBenChung
committed
Add CI & docstrings
1 parent 3acab72 commit e75b687

File tree

5 files changed

+177
-51
lines changed

5 files changed

+177
-51
lines changed

‎.github/workflows/CI.yml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
name: CI
2+
# Run on master, tags, or any pull request
3+
on:
4+
push:
5+
branches: [master]
6+
tags: ["*"]
7+
pull_request:
8+
concurrency:
9+
# Skip intermediate builds: always.
10+
# Cancel intermediate builds: only if it is a pull request build.
11+
group: ${{ github.workflow }}-${{ github.ref }}
12+
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
13+
jobs:
14+
15+
test:
16+
name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }}
17+
runs-on: ubuntu-latest
18+
strategy:
19+
fail-fast: false
20+
matrix:
21+
version:
22+
- "1" # Latest Release
23+
os:
24+
- ubuntu-latest
25+
arch:
26+
- x64
27+
include:
28+
# Add specific version used to run the reference tests.
29+
# Must be kept in sync with version check in `test/runtests.jl`,
30+
# and with the branch protection rules on the repository which
31+
# require this specific job to pass on all PRs
32+
# (see Settings > Branches > Branch protection rules).
33+
- os: ubuntu-latest
34+
version: 1.10.4
35+
arch: x64
36+
steps:
37+
- uses: actions/checkout@v4
38+
- uses: julia-actions/setup-julia@v2
39+
with:
40+
version: ${{ matrix.version }}
41+
arch: ${{ matrix.arch }}
42+
- uses: actions/cache@v4
43+
env:
44+
cache-name: cache-artifacts
45+
with:
46+
path: ~/.julia/artifacts
47+
key: ${{ runner.os }}-${{ matrix.arch }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
48+
restore-keys: |
49+
${{ runner.os }}-${{ matrix.arch }}-test-${{ env.cache-name }}-
50+
${{ runner.os }}-${{ matrix.arch }}-test-
51+
${{ runner.os }}-${{ matrix.arch }}-
52+
${{ runner.os }}-
53+
- uses: julia-actions/julia-buildpkg@latest
54+
- run: |
55+
git config --global user.name Tester
56+
git config --global user.email te@st.er
57+
- uses: julia-actions/julia-runtest@latest
58+
- uses: julia-actions/julia-processcoverage@v1
59+
- uses: codecov/codecov-action@v4
60+
with:
61+
files: lcov.info
62+
token: ${{ secrets.CODECOV_TOKEN }}
63+
fail_ci_if_error: false
64+
docs:
65+
name: Documentation
66+
runs-on: ubuntu-latest
67+
steps:
68+
- uses: actions/checkout@v4
69+
- uses: julia-actions/setup-julia@v2
70+
with:
71+
version: '1'
72+
- run: |
73+
git config --global user.name name
74+
git config --global user.email email
75+
git config --global github.user username
76+
- run: |
77+
julia --project=docs -e '
78+
using Pkg
79+
Pkg.develop(PackageSpec(path=pwd()))
80+
Pkg.instantiate()'
81+
- run: |
82+
julia --project=docs -e '
83+
using Documenter: doctest
84+
using ModelTesting
85+
doctest(ModelTesting)'
86+
- run: julia --project=docs docs/make.jl
87+
env:
88+
JULIA_PKG_SERVER: ""
89+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
90+
DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}

‎docs/src/index.md‎

Lines changed: 3 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -6,41 +6,10 @@ CurrentModule = ModelTesting
66

77
Documentation for [ModelTesting](https://github.com/BenChung/ModelTesting.jl).
88

9-
ModelTesting is intended to facilitate the easy testing and post-facto analysis of ModelingToolkit models.
10-
Test ensembles are constructed using two core abstractions:
9+
ModelTesting is intended to facilitate the easy testing and post-facto analysis of ModelingToolkit models. It currently provides two key bits of functionality:
1110

12-
* *Rollouts*, or an execution of a model. A rollout consists of a model and the data required to execute the model forward (which could consist of parameter values, real observations/control inputs, or synthetic components that should be plugged into the system). Rollouts can also modify the model being executed (such as introducing new equations to track conservation properties).
13-
* *Timeseries*, which describe the state of a system as it executes forward (PDEs are TBD). A timeseries consists of time-indexed data whose values inhabit named states. Rollouts can be executed to create a timeseries, but you can also get a timeseries from test data or an analytic solution.
14-
Timeseries can be combined in a namespace-aware way with the <+ operator; combining timeseries requires that both are defined over an overlapping interval and can be evaluated at the same points (by default the union of the points). Timeseries are not nessecarily tabular!
15-
16-
The library then works by executing evaluations over rollouts. Timeseries are defined over a time span that's encoded into the timeseries and is checked for mutual consistentcy before evaluation.
17-
18-
Questions:
19-
* How to represent parameters? Want to be able to have multiple parameter maps and be able to overlay them. Should be version controlled separately from the models.
20-
* The results data in the below examples is implicitly timebased. What do we do about timeseries that don't have a consistent time base (for example how do we compare results between two different solvers or between an adaptive solver and sampled real data)?
21-
22-
23-
```julia
24-
reference = Timeseries.FromDataFrame([...])
25-
previous = Timseries.FromFile([...])
26-
27-
baseline = Rollout(mymodel) + ParametersFile("baseparams.jl") + Time(0, 10) + Solver(Tsit5())
28-
paramset1 = baseline + ParametersFile("params1.jl")
29-
rk4 = baseline + Solver(RK4())
30-
tracking = baseline + Tracking(reference)
31-
conservation = baseline + Metrics.Conservation()
32-
33-
@named baseline_sol = evaluate(baseline)
34-
@named params1_sol = evaluate(paramset1)
35-
36-
Results("output.nc") do
37-
baseline_sol
38-
<+ params1_sol
39-
<+ reference
40-
<+ Evaluations.within(baseline_sol, reference_sol, 0.5; measure=norm)
41-
<+ Evaluations.within(baseline_sol, rk4, 0.5; measure=norm)
42-
end
43-
```
11+
* Serialization of MTK solutions into a common format through `discretize_solution`.
12+
* Comparison of solutions to each other and to previously-saved data through `compare`.
4413

4514

4615
```@index

‎src/test/compare.jl‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
"""
2+
`DefaultComparison` is a convenience default comparison method for solutions and reference data.
3+
4+
By default it's instantiated with l∞, l1, l2, RMS, and final l1 comparsons. More can be added by
5+
passing a dict of name=>comparison function pairs into `field_cmp`, with the signature (delta, dt_delta, t) -> [vector difference].
6+
The arguments of these comparison functions are `delta` for the raw difference beteween the value, `dt_delta` for the
7+
product of dt and delta (if you want to approximate the integral of the error), and the timestamps at which the data was produced.
8+
The arguments consist of a vector per compared field.
9+
"""
110
struct DefaultComparison
211
field_cmp::Dict{Symbol, Function}
312
function DefaultComparison(field_cmp::Dict{Symbol, Function}=Dict{Symbol, Function}(); use_defaults=true)
@@ -12,6 +21,21 @@ struct DefaultComparison
1221
return new(field_cmp)
1322
end
1423
end
24+
25+
"""
26+
(d::DefaultComparison)(c, names, b, t, n, r)
27+
28+
Default comparison implementation. Returns a dataframe that describes the result of running each comparison operation on the given data.
29+
30+
#Arguments
31+
32+
* `c`: The symbolic container from which the values we're comparing came
33+
* `names`: The names to output for the fields we're comparing (not always the same as the names of the fields being compared)
34+
* `b`: The symbolic variables that are being compared
35+
* `t`: The timestamps at which the comparison is taking place
36+
* `n`: The new values being compared
37+
* `r`: The reference values to compare against
38+
"""
1539
function (d::DefaultComparison)(c, names, b, t, n, r)
1640
delta = map((o, re) -> o .- re, n, r)
1741
dt_delta = map(d -> 0.5 * (d[1:end-1] .+ d[2:end]) .* (t[2:end] .- t[1:end-1]), delta)
@@ -21,6 +45,17 @@ function (d::DefaultComparison)(c, names, b, t, n, r)
2145
end
2246

2347

48+
"""
49+
compare(
50+
new_sol::SciMLBase.AbstractTimeseriesSolution,
51+
reference::DataFrame,
52+
over::Vector{<:Union{Pair{<:Any, String}, Pair{<:Any, Pair{String, String}}}}
53+
cmp=DefaultComparison(); warn_observed=true)
54+
55+
Lower level version of `compare` that lets you specify the mapping from values in `new_sol` (specified as either variable => name or variable => (input_name, output_name) pairs) to
56+
the column names in `reference`. The tuple-result version lets you specify what the column names are when passed to the comparison method. Since the mapping is explicit
57+
this version does not take `to_name`, but it still does take `warn_observed`. See the implicit-comparison version of `compare` for more information.
58+
"""
2459
function compare(
2560
new_sol::SciMLBase.AbstractTimeseriesSolution,
2661
reference::DataFrame,
@@ -43,6 +78,28 @@ function compare(
4378
end
4479
end
4580

81+
"""
82+
compare(
83+
new_sol::SciMLBase.AbstractTimeseriesSolution,
84+
reference::DataFrame,
85+
cmp=DefaultComparison(); to_name=string, warn_observed=true)
86+
87+
`compare` compares the results of `new_sol` to a result in `reference`, which may come from either experiment or a previous model execution.
88+
The format of `reference` is that produced by [`discretize_solution`](@ref):
89+
90+
* A column named "timestamp" that indicates the time the measurement was taken (referenced to the same timebase as the solution provided)
91+
* Columns with names matching the names in the completed system `new_sol` (fully qualified)
92+
93+
If `new_sol` is dense then the discretization nodes don't need to line up with the `reference` and the interpolant will be used instead.
94+
If it is not dense then the user must ensure that the saved states in are at the same times as the "timestamp"s are in the reference data.
95+
96+
`compare` will use the comparison method specified by `cmp` to compare the two solutions; by default it uses the [`DefaultComparison`](@ref)
97+
(which offers l1, l2, and rms comparisons per observed value), but you can pass your own. Look at `DefaultComparison` for more details on how.
98+
99+
The two optional named parameters are `to_name` (which is used to convert the symbolic variables in `new_sol` into valid column names for
100+
the dataframe) and `warn_observed` which controls whether `compare` complains about comparing observed values to non-observed values. We suggest
101+
leaving `warn_observed` on for regression testing (it'll give you a note when MTK changes the simplifcation of the system).
102+
"""
46103
function compare(
47104
new_sol::SciMLBase.AbstractTimeseriesSolution,
48105
reference::DataFrame,

‎src/test/discretize.jl‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,17 @@ function make_cols(names, rows)
1616
return cols
1717
end
1818

19+
"""
20+
discretize_solution(solution::SciMLBase.AbstractTimeseriesSolution[, time_ref::SciMLBase.AbstractTimeseriesSolution]; measured=nothing, all_observed=false)
21+
22+
`discretize_solution` takes a solution and an optional time reference and converts it into a dataframe. This dataframe will contain either:
23+
* The variables (unknowns or observeds) named in `measured`, if provided.
24+
* The variables marked as `measured` if `all_observed` is `false` and `measured` is `nothing`.
25+
* All variables in the system if `all_observed` is `true` and `measured` is `nothing`.
26+
The dataframe will contain a column called `timestamp` for each of the discretization times and a column for each observed value.
27+
28+
If no time reference is provided then the timebase used to discretize `solution` will be used instead.
29+
"""
1930
function discretize_solution(solution::SciMLBase.AbstractTimeseriesSolution, time_ref::SciMLBase.AbstractTimeseriesSolution; measured=nothing, all_observed=false)
2031
container = SymbolicIndexingInterface.symbolic_container(time_ref)
2132
ref_t_vars = independent_variable_symbols(container)

‎src/test/instant.jl‎

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,29 +3,28 @@ function test_instantaneous(
33
sys::ModelingToolkit.AbstractSystem,
44
ic,
55
checks::Num;
6-
t = nothing) # todo: after porting to v9 use InitializationSystem to solve the checks system
7-
if !SymbolicIndexingInterface.is_observed(checks)
8-
@assert false "The given check values are not observed values of the given system"
9-
end
10-
sv = ModelingToolkit.varmap_to_vars(ic, variable_symbols(sys); defaults=default_values(sys))
11-
pv = ModelingToolkit.varmap_to_vars(ic, parameter_symbols(sys); defaults=default_values(sys))
12-
13-
obsfun = SymbolicIndexingInterface.observed(sys, checks)
14-
if SymbolicIndexingInterface.istimedependent(sys)
15-
@assert !isnothing(t) "The kwarg t must be given (and be a non-nothing value) if the system is time dependent"
16-
return obsfun(sv, pv, t)
17-
elseif !SymbolicIndexingInterface.constant_structure(sys)
18-
@assert false "The system's structure must be constant to use test_instantaneous; to be fixed" # TODO
19-
else
20-
return obsfun(sv, pv)
21-
end
6+
t = nothing)
7+
return test_instantaneous(sys, ic, checks; t = t)
228
end
239

10+
"""
11+
test_instantaneous(sys::ModelingToolkit.AbstractSystem,
12+
ic,
13+
checks::Union{Num, Array};
14+
t = nothing)
15+
16+
`test_instantaneous` is a helper wrapper around constructing and executing an `ODEProblem` at a specific condition. It's intended for use in basic
17+
sanity checking of MTK models, for example to ensure that the derivative at a given condition has the correct sign. It should be passed the system,
18+
the initial condition dictionary to be given to the ODEProblem constructor (including parameters), and a list of observable values from the system to
19+
extract from the ODEProblem. If `checks` is a single `Num` then the result will be that observed value; otherwise it will return an array in the same
20+
order as the provided checks.
21+
"""
2422
function test_instantaneous(
2523
sys::ModelingToolkit.AbstractSystem,
2624
ic,
2725
checks::Array;
28-
t = nothing) # todo: after porting to v9 use InitializationSystem to solve the checks system
26+
t = nothing)
27+
t = isnothing(t) ? 0.0 : t
2928
prob = ODEProblem(sys, ic, (0.0, 0.0))
3029
getter = getu(prob, checks)
3130
return getter(prob)

0 commit comments

Comments
 (0)