Temporary terms computed in previous blocks were not used in the bytecode
output of a given block. This was inefficient (because this means that
expressions already computed and store in the temporary terms vector would be
recomputed), and incidentally it would break the external functions
output (because it would trigger a lookup in the “TEF terms”, which would thus
fail).
Closes: #115
If the model is purely backward, determine whether all original equations have
a single contemporaneous endogenous on the LHS. If this is the case, then first
try a normalization by enforcing that each original equation is matched with
the endogenous on the LHS.
This helps with the simulation of purely backward models, where equations are
renormalized with mfs=3, since it produces a simpler system to be recursively
evaluated/solved.
It would previously return a boolean. The exception is more convenient for
producing a different error message in the case of the specialized algorithm
introduced in the next commit.
The C99 copysign() function was used in the generated C output, but that
function does not correctly handle zero. Replace it by a custom sign()
function.
Improve performance on very large models (⩾ 5000 equations).
Note that std::unordered_set cannot be used for the temporary_terms_t type,
because ordering is needed there (for writing the output files).
The implementation no longer relies on ramsey_policy.m at the MATLAB/Octave
level.
By the way, drop ModFileStructure::ramsey_model_present, which is now
redundant.
Options “order”, “partial_information” and “k_order_solver” are not accepted by
the command (this code is probably a remnant of the deprecated “ramsey_policy”
command).
– M_.ramsey_orig_endo_nbr is the number of endogenous variables in the model
present just before adding the Lagrange multipliers and computing the Ramsey
FOC; it is by construction equal to the number of equations that will be added
by the process of computing the FOCs
– M_.ramsey_orig_eq_nbr is the number of equations in the model present just
before adding the Lagrange multipliers and computing the Ramsey FOC; it is by
construction equal to the number of Lagrange multipliers that will be added by
the process of computing the FOCs
Note that both may be greater than the number of endogenous/equations written
by the user, because some auxiliary variables may have already been added.
Also note that the number of policy instruments in M_.ramsey_orig_endo_nbr −
M_.ramsey_orig_eq_nbr
As a consequence, drop M_.ramsey_eq_nbr (which was actually equal to
M_.ramsey_orig_endo_nbr) and M_.orig_eq_nbr (which was actually equal to
M_.ramsey_orig_eq_nbr, but would also be set in the non-Ramsey case). The new
names are clearer.
The computing of the Ramsey steady state relies on the fact that Lagrange
multipliers appear linearly in the system to be solved. Instead of directly
solving for the Lagrange multipliers along with the other variables,
dyn_ramsey_static.m reduces the size of the problem by always computing the
value of the multipliers that minimizes the residuals, given the other
variables (using a minimum norm solution, easy to compute because of the
linearity property). That function thus needs the derivatives of the optimality
FOCs with respect to the multipliers. The problem is that, when multipliers
appear in an auxiliary variable related to a lead/lag, then those derivatives
need to be retrieved by a chain rule derivation, which cannot be easily done
with the regular static file.
This commit implements the creation of a new file,
ramsey_multipliers_static_g1.{m,mex}, that provides exactly the needed
derivatives w.r.t. Lagrange multipliers through chain rule derivation.
Ref. dynare#633, dynare#1119, dynare#1133
This is effectively a revert of commits 1b4f68f934,
32fb90d5f3 and f6f4ea70fb.
This transformation had been introduced in order to fix the computation of the
Ramsey steady state in the case where Lagrange multipliers appeared with a lead
or lag ⩾ 2 (and where thus part of the definition of an auxiliary variable).
But this transformation had introduced bugs in the handling of external
functions which were difficult to tackle.
Moreover, it seems preferable to keep the strict correspondence between the
dynamic and static model, in order to make reasoning about the preprocessor
internals easier (in particular, for this reason this transformation was not
implemented in ModFile::transformPass() but in ModFile::computingPass(), which
was a bit confusing).
A better solution for the Ramsey steady state issue will is implemented in the
descendent of the present commit.
Ref. dynare#633, dynare#1119, dynare#1133
If a given external function was called two times with different arguments,
then the second call would always be computed in the same temporary terms
chunk (derivation order or block) as the first call, even if this was not
necessary (technically, the second call would be promoted in the temporary
terms computation in the same category as the first call).
This could possibly lead to an inefficiency.
If an endogenous with a lead ⩾ 2 or an exogenous with a lead ⩾ 1 appeared in
the argument(s) of a call to an external function, the auxiliary variable
transformation was incorrect (the variable was replaced inside the function
call, while it is the whole function call that has to be replaced).
This could lead to incorrect results in stochastic contexts, when the external
function is nonlinear.
The first argument to ExprNode::computeTemporaryTerms() is supposed to be a
pair (endo derivation order, param derivation order). The two elements were
interverted in the call. This would not affect the result, because parameter
derivatives are not computed there and what matters is the lexical ordering
which remains the same. But fixing the order is better, for consistency with
the method description.
It was erroneously using MATLAB costs, leading to possible
inefficiencies (though those cost tables are probably not very accurate and
should be revised).
The argument had the same name as the data member “datatree”, so this could
lead to confusion (though there was no bug, since the argument was masking
the data member).
Previously, the MinGW location was appended multiple times to the PATH
variable, which in some cases would make the variable too long and thus
dysfunctional.
The variable is now initialized once when the worker threads are created.
By the way, move the macOS+Octave environment variable initializations to the
same place, for consistency.
Variable indices would be incorrect in the evaluated Jacobian if recursive
variables were present. This would lead to incorrect results and/or crashes in
bytecode MEX. This bug has been exposed by commit
f45a99fc68, which actually enabled mfs>0 for
static models.
Commit 23b0c12d8e introduced caching in chain
rule derivation (used by block decomposition), which increased speed for mfs >
0, but actually decreased it for mfs=0.
This patch introduces the pre-computation of derivatives which are known to be
zero using symbolic a priori (similarly to what is done in the non-chain rule
context). The algorithms are now identical between the two contexts (both
symbolic a priori + caching), the difference being that in the chain rule
context, the symbolic a priori and the cache are not stored within the ExprNode
class, since they depend on the list of recursive variables.
This patch brings a significant performant improvement for all values of the
“mfs” option (the improvement is greater for small values of “mfs”).
Input and output ranges should not overlap when calling std::set_union(),
otherwise the behaviour is undefined.
It seems that in this precise case the computation would still be
correct (though inefficient), because of the properties of std::set or because
of the specific implementation in libstdc++. But it’s better to be on the safe
side.
In a dynamic context, the only potentially non-null derivatives of
STEADY_STATE(…) are the parameters. We know that the derivatives w.r.t. other
variables are zero, so store that information in non_null_derivatives.
More precisely, incorrect equation normalization could occur in the presence of
cos, sin, tan, cosh and x^n (where n is an even integer).
Also add some comments explaining why some other rules are (hopefully) correct.
Note that DynamicModel::determineBlockDerivativesType(), it’s legitimate to
replace max_{lead,lag} by max_endo_{lead,lag}, because for exogenous
lag=lead=0, and we no longer compute derivatives w.r.t. to endogenous that do
not belong to the block (so-called “other” endogenous).
As a consequence, and as a temporary measure, always output the
non-block-decomposed legacy representation.
Also drop the block kalman filter output, and drop now useless variables in
M_.block_structure.
— No longer call std::exit() from threads when compilation fails, that function
is marked as not thread-safe under GNU/Linux; and it leads to deadlocks under
Windows. Rather store the list of failed objects, and exit with a message and
an error code from the main thread when that list is not empty at the end of
preprocessing.
– Fix the condition used for waiting until all compilation threads finish;
checking that the queue is empty is not enough, since a compilation may be
ongoing. So also track objects whose compilation is ongoing.
It would previously forget to write the indices for the highest derivation
order (e.g. if order=3, then it would only write sparce indices up to 2nd
order).
Ref. dynare#1859
Such a decomposition has to be simulated with periods as the outer loop and
blocks as the inner loop.
It is enabled by default for purely backward/forward/static models, as long as
the “block” option is not given. In that case, “mfs” is also set to 3 by
default (until that value becomes the global default).
M_.time_recursive_block_decomposition is set to “true” when that decomposition
has been performed, “false” otherwise for the traditional decomposition (the
latter has to be simulated with blocks as the outer loop and periods as the
inner loop).
It is no longer used in the new implementation of solve_algo={12,14}.
However, the M_.lhs field that was created by setup_solvers.m is used from
other places, so reimplement it at the preprocessor level.
The files are created under <basename>/+debug/dynamic_resid.m and
<basename>/+debug/static_resid.m.
Their purpose is to evaluate separately the LHS and RHS of each equation.
Within a block, if mfs>0, the recursive variables would not appear in recursive
order. This could lead to wrong results in case of dependency relationships
between recursive variables.
The new representation is only supported for MATLAB/Octave, C and Julia output
for the time being. Bytecode and JSON are unsupported.
This commit adds new fields in M_.
This is a preliminary step for dynare#1859.
– print the information only for the final matching (and not the intermediary
ones that may have failed);
– print the equation name next to its number.
In an “evaluate” block, the LHS of a renormalized equation (such as “log(x)=…”)
could be associated to a temporary term that would then be incorrectly
computed: that temporary term would be evaluated *before* (and not after) the
evaluation of the associated variable (“x” in the example).
Should have no impact though, since diff nodes are already substituted out at
that point. But it’s better to implement it properly, in case we change the
substitution rules later.
By the way, make the computeSubExprContainingVariable method protected.
The previous system would spawn as many threads as there are object files to be
compiled (which could lead to hundreds of threads for large block-decomposed
models). This could pose a memory usage problem (even when not just waiting,
threads require memory for their own stack).
– DataTree::packageDir() now takes a std::string_view, returns a
std::filesystem::path, and no longer creates that directory
– DataTree::writeToFileIfModified() now takes a std::filesystem::path as
argument
– Do not call DataTree::writeToFileIfModified() for generating MATLAB/Octave
files, since it does not work (the directory inside which the file is written
has been deleted by the preprocessor just before)
– Consistently use DataTree::packageDir() everywhere (for compatibility with
planner_objective)
By the way:
– improve the semantics by having a consistent treatment of empty substring
components (previously, only the last one was ignored)
– move it to DataTree to make it more accessible from elsewhere (even though
ideally it should be in a “utilities” namespace).
Johannes Pfeifer
Sébastien Villemot
Stéphane Adjemian (Guts)
MichelJuillard