- `dynasave`: if a variable being saved was named `n` or `s`, the `eval` statements would break the code
- `dynasave`: use the `-struct` option to `save` to avoid `eval` statements
- `dynasave` and `dynatype`: do everything in 1 loop instead of 2
- `dynasave` and `dynatype`: use `strcmp` instead of `strfind`
- preprocessor update contains:
- Partial reversion of global indentation of macro processor header files introduced in e2d5a83592634f0604d8c86409748cd2ec5906d2
- Symbol List check pass: allow caller to specify the valid types of variables in a Symbol List
- Allow `dynasave` and `dynatype` to support exogenous variables in their var_list
issue #1691
Note that I still need to do a code clean up (provide some licenses for functions from other people) and to double check order=3. There is also much room for speed and memory improvement, but the code works fine for now. I will also provide more information to the merge request soon about the detailed changes for future reference.
It applies the approximated policy function to a set of particles, using
Dynare++ routines.
There is support for parallelization, using Dynare++ multithreading
model (itself based on C++11 threads; we don’t use OpenMP because it is
incompatible with MKL). For the time being, default to a single thread. This
should be later refined through empirical testing.
In case the epilogue formula is non-linear, the non additive non-linear term is distributed proportionally to the size of the individual shock contribution.
It is triggered by new option with_epilogue, applicable to commands:
1) shock_decomposition, realtime_shock_decomposition,
where preprocessor should trigger
options_.shock_decomp.with_epilogue=true;
2) initial_condition_decomposition
where preprocessor should trigger
options_.initial_condition_decomp.with_epilogue=true;
Under Octave, having namespaces called “get” and “set” overshadows the builtin
functions with the same names, which are needed for graphics manipulation.
Therefore we go back to the initial function naming scheme, but moving all
those functions under an “accessors” subdirectory.
Among other things, this is a revert of
e4134ab59b and
c5e86fcb59.
Ref. !1655, !1686
Accordingly update the MATLAB routines, the testsuite, and the manual.
In particular, “squeeze_shock_decomp” has been renamed to
“squeeze_shock_decomposition” for consistency with other commands.
Ref. #1687, !1655
# Preliminary comments
I finished the identification toolbox at orders two and three using the pruned state space system, but before I merge request this, I decided to first merge the new functionality to compute parameter derivatives of perturbation solution matrices at higher orders. So after this is approved, I merge the identification toolbox.
I guess @rattoma, @sebastien, and @michel are best choices to review this.
I outline the main idea first and then provide some more detailed changes I made to the functions.
***
# Main idea
This merge request is concerned with the *analytical*computation of the parameter derivatives of first, second and third order perturbation solution matrices, i.e. using _closed-form_ expressions to efficiently compute the derivative of $g_x$ , $g_u$, $g_{xx}$, $g_{xu}$, $g_{uu}$, $g_{\sigma\sigma}$, $g_{xxx}$, $g_{xxu}$, $g_{xuu}$, $g_{uuu}$, $g_{x\sigma\sigma}$, $g_{u\sigma\sigma}$ *with respect to model parameters* $\theta$. Note that $\theta$ contains model parameters, stderr and corr parameters of shocks. stderr and corr parameters of measurement errors are not yet supported, (they can easily be included as exogenous shocks). The availability of such derivatives is beneficial in terms of more reliable analysis of model sensitivity and parameter identifiability as well as more efficient estimation methods, in particular for models solved up to third order, as it is well-known that numerical derivatives are a tricky business, especially for large models.
References for my approach are:
* Iskrev (2008, 2010) and Schmitt-Grohé and Uribe (2012, Appendix) who were the first to compute the parameter derivatives analytically at first order, however, using inefficient (sparse) Kronecker products.
* Mutschler (2015) who provides the expressions for a second-order, but again using inefficient (sparse) Kronecker products.
* Ratto and Iskrev (2012) who show how the first-order system can be solved accurately, fast and efficiently using existing numerical algorithms for generalized Sylvester equations by taking the parameter derivative with respect to each parameter separately.
* Julliard and Kamenik (2004) who provide the perturbation solution equation system in tensor notation at any order k.
* Levintal (2017) who introduces permutation matrices to express the perturbation solution equation system in matrix notation up to fifth order.
Note that @rattoma already implemented the parameter derivatives of $g_x$ and $g_u$ analytically (and numerically), and I rely heavily on his work in `get_first_order_solution_params_derivs.m` (previously `getH.m`). My additions are mainly to this function and thus it is renamed to `get_perturbation_params_derivs.m`.
The basic idea of this merge request is to take the second- and third-order perturbation solution systems in Julliard and Kamenik (2004), unfold these into an equivalent matrix representation using permutation matrices as in Levintal (2017). Then extending Ratto and Iskrev (2012) one takes the derivative with respect to each parameter separately and gets a computational problem that is linear, albeit large, as it involves either solving generalized Sylvester equations or taking inverses of highly sparse matrices. I will now briefly summarize the perturbation solution system at third order and the system that results when taking the derivative with respect to parameters.
## Perturbation Solution
The following systems arise at first, second, and third order:
$(ghx): f_{x} z_{x} = f_{y_{-}^*} + f_{y_0} g_{x} + f_{y_{+}^{**}} g^{**}_{x} g^{*}_{x}= A g_{x} + f_{y_{-}^*}=0$
$(ghu): f_{z} z_{u} = f_{y_0} g_{u} + f_{y_{+}^{**}} g^{**}_{x} g^{*}_{u} + f_{u}= A g_u + f_u = 0$
$(ghxx) : A g_{xx} + B g_{xx} \left(g^{*}_{x} \otimes g^{*}_{x}\right) + f_{zz} \left( z_{x} \otimes z_{x} \right) = 0$
$(ghxu) : A g_{xu} + B g_{xx} \left(g^{*}_{x} \otimes g^{*}_{u}\right) + f_{zz} \left( z_{x} \otimes z_{u} \right) = 0$
$(ghuu) : A g_{uu} + B g_{xx} \left(g^{*}_{u} \otimes g^{*}_{u}\right) + f_{zz} \left( z_{u} \otimes z_{u} \right) = 0$
$(ghs2) : (A+B) g_{\sigma\sigma} + \left( f_{y^{**}_{+}y^{**}_{+}} \left(g^{**}_{u} \otimes g^{**}_{u}\right) + f_{y^{**}_{+}} g^{**}_{uu}\right)vec(\Sigma) = 0$
$(ghxxx) : A g_{xxx} + B g_{xxx} \left(g^{*}_{x} \otimes g^{*}_{x} \otimes g^{*}_{x}\right) + f_{y_{+}}g^{**}_{xx} \left(g^{*}_x \otimes g^{*}_{xx}\right)P_{x\_xx} + f_{zz} \left( z_{x} \otimes z_{xx} \right)P_{x\_xx} + f_{zzz} \left( z_{x} \otimes z_{x} \otimes z_{x} \right) = 0$
$(ghxxu) : A g_{xxu} + B g_{xxx} \left(g^{*}_{x} \otimes g^{*}_{x} \otimes g^{*}_{u}\right) + f_{zzz} \left( z_{x} \otimes z_{x} \otimes z_{u} \right) + f_{zz} \left( \left( z_{x} \otimes z_{xu} \right)P_{x\_xu} + \left(z_{xx} \otimes z_{u}\right) \right) + f_{y_{+}}g^{**}_{xx} \left( \left(g^{*}_{x} \otimes g^{*}_{xu}\right)P_{x\_xu} + \left(g^{*}_{xx} \otimes g^{*}_{u}\right) \right) = 0$
$(ghxuu) : A g_{xuu} + B g_{xxx} \left(g^{*}_{x} \otimes g^{*}_{u} \otimes g^{*}_{u}\right) + f_{zzz} \left( z_{x} \otimes z_{u} \otimes z_{u} \right)+ f_{zz} \left( \left( z_{xu} \otimes z_{u} \right)P_{xu\_u} + \left(z_{x} \otimes z_{uu}\right) \right) + f_{y_{+}}g^{**}_{xx} \left( \left(g^{*}_{xu} \otimes g^{*}_{u}\right)P_{xu\_u} + \left(g^{*}_{x} \otimes g^{*}_{uu}\right) \right) = 0$
$(ghuuu) : A g_{uuu} + B g_{xxx} \left(g^{*}_{u} \otimes g^{*}_{u} \otimes g^{*}_{u}\right) + f_{zzz} \left( z_{u} \otimes z_{u} \otimes z_{u} \right)+ f_{zz} \left( z_{u} \otimes z_{uu} \right)P_{u\_uu} + f_{y_{+}}g^{**}_{xx} \left(g^{*}_{u} \otimes g^{*}_{uu}\right)P_{u\_uu} = 0$
$(ghx\sigma\sigma) : A g_{x\sigma\sigma} + B g_{x\sigma\sigma} g^{*}_x + f_{y_{+}} g^{**}_{xx}\left(g^{*}_{x} \otimes g^{*}_{\sigma\sigma}\right) + f_{zz} \left(z_{x} \otimes z_{\sigma\sigma}\right) + F_{xu_{+}u_{+}}\left(I_{n_x} \otimes vec(\Sigma)\right) = 0$
$F_{xu_{+}u_{+}} = f_{y_{+}^{\ast\ast}} g_{xuu}^{\ast\ast} (g_x^{\ast} \otimes I_{n_u^2}) + f_{zz} \left( \left( z_{xu_{+}} \otimes z_{u_{+}} \right)P_{xu\_u} + \left(z_{x} \otimes z_{u_{+}u_{+}}\right) \right) + f_{zzz}\left(z_{x} \otimes z_{u_{+}} \otimes z_{u_{+}}\right)$
$(ghu\sigma\sigma) : A g_{u\sigma\sigma} + B g_{x\sigma\sigma} g^{*}_{u} + f_{y_{+}} g^{**}_{xx}\left(g^{*}_{u} \otimes g^{*}_{\sigma\sigma}\right) + f_{zz} \left(z_{u} \otimes z_{\sigma\sigma}\right) + F_{uu_{+}u_{+}}\left(I_{n_u} \otimes vec(\Sigma_u)\right) = 0$
$F_{uu_{+}u_{+}} = f_{y_{+}^{\ast\ast}} g_{xuu}^{\ast\ast} (g_u^{\ast} \otimes I_{n_u^2}) + f_{zz} \left( \left( z_{uu_{+}} \otimes z_{u_{+}} \right)P_{uu\_u} + \left(z_{u} \otimes z_{u_{+}u_{+}}\right) \right) + f_{zzz}\left(z_{u} \otimes z_{u_{+}} \otimes z_{u_{+}}\right)$
A and B are the common perturbation matrices:
$A = f_{y_0} + \begin{pmatrix} \underbrace{0}_{n\times n_{static}} &\vdots& \underbrace{f_{y^{**}_{+}} \cdot g^{**}_{x}}_{n \times n_{spred}} &\vdots& \underbrace{0}_{n\times n_{frwd}} \end{pmatrix}$and $B = \begin{pmatrix} \underbrace{0}_{n \times n_{static}}&\vdots & \underbrace{0}_{n \times n_{pred}} & \vdots & \underbrace{f_{y^{**}_{+}}}_{n \times n_{sfwrd}} \end{pmatrix}$
and $z=(y_{-}^{\ast}; y; y_{+}^{\ast\ast}; u)$ denotes the dynamic model variables as in `M_.lead_lag_incidence`, $y^\ast$ denote state variables, $y^{\ast\ast}$ denote forward looking variables, $y_+$ denote the variables with a lead, $y_{-}$ denote variables with a lag, $y_0$ denote variables at period t, $f$ the model equations, and $f_z$ the first-order dynamic model derivatives, $f_{zz}$ the second-order dynamic derivatives, and $f_{zzz}$ the third-order dynamic model derivatives. Then:
$z_{x} = \begin{pmatrix}I\\g_{x}\\g^{**}_{x} g^{*}_{x}\\0\end{pmatrix}$, $z_{u} =\begin{pmatrix}0\\g_{u}\\g^{**}_{x} \cdot g^{*}_{u}\\I\end{pmatrix}$, $z_{u_{+}} =\begin{pmatrix}0\\0\\g^{**}_{u}\\0\end{pmatrix}$
$z_{xx} = \begin{pmatrix} 0\\g_{xx}\\g^{**}_{x} \left( g^{*}_x \otimes g^{*}_{x} \right) + g^{**}_{x} g^{*}_{x}\\0\end{pmatrix}$, $z_{xu} =\begin{pmatrix}0\\g_{xu}\\g^{**}_{xx} \left( g^{*}_x \otimes g^{*}_{u} \right) + g^{**}_{x} g^{*}_{xu}\\0\end{pmatrix}$, $z_{uu} =\begin{pmatrix}0\\g_{uu}\\g^{**}_{xx} \left( g^{*}_u \otimes g^{*}_{u} \right) + g^{**}_{x} g^{*}_{uu}\\0\end{pmatrix}$,
$z_{xu_{+}} =\begin{pmatrix}0\\0\\g^{**}_{xu} \left( g^{*}_x \otimes I \right)\\0\end{pmatrix}$, $z_{uu_{+}} =\begin{pmatrix}0\\0\\g^{**}_{xu} \left( g^{*}_{u} \otimes I \right)\\0\end{pmatrix}$, $z_{u_{+}u_{+}} =\begin{pmatrix}0\\0\\g^{\ast\ast}_{uu}\\0\end{pmatrix}$, $z_{\sigma\sigma} = \begin{pmatrix}0\\ g_{\sigma\sigma}\\ g^{\ast\ast}_{x}g^{\ast}_{\sigma\sigma} + g^{\ast\ast}_{\sigma\sigma}\\0 \end{pmatrix}$
$P$ are permutation matrices that can be computed using Matlab's `ipermute` function.
## Parameter derivatives of perturbation solutions
First, we need the parameter derivatives of first, second, third, and fourth derivatives of the dynamic model (i.e. g1,g2,g3,g4 in dynamic files), I make use of the implicit function theorem: Let $f_{z^k}$ denote the kth derivative (wrt all dynamic variables) of the dynamic model, then let $df_{z^k}$ denote the first-derivative (wrt all model parameters) of $f_{z^k}$ evaluated at the steady state. Note that $f_{z^k}$ is a function of both the model parameters $\theta$ and of the steady state of all dynamic variables $\bar{z}$, which also depend on the parameters. Hence, implicitly $f_{z^k}=f_{z^k}(\theta,\bar{z}(\theta))$ and $df_{z^k}$ consists of two parts:
1. direct derivative wrt to all model parameters given by the preprocessor in the `_params_derivs.m` files
2. contribution of derivative of steady state of dynamic variables (wrt all model parameters): $f_{z^{k+1}} \cdot d\bar{z}$
Note that we already have functionality to compute $d\bar{z}$ analytically.
Having this, the above perturbation systems are basically equations of the following types
$AX +BXC = RHS$ or $AX = RHS$
Now when taking the derivative (wrt to one single parameter $\theta_j$), we get
$A\mathrm{d}\{X\} + B\mathrm{d}\{X\}C = \mathrm{d}\{RHS\} - \mathrm{d}\{A\}X - \mathrm{d}\{B\}XC - BX\mathrm{d}\{C\}$
or
$A\mathrm{d}\{X\} = \mathrm{d}\{RHS\} - \mathrm{d}\{A\}X$
The first one is a Sylvester type equation, the second one can be solved by taking the inverse of $A$. The only diffculty and tedious work arrises in computing (the highly sparse) derivatives of $RHS$.
***
# New functions: `
## get_perturbation_params_derivs.m`and `get_perturbation_params_derivs_numerical_objective.m`
* The parameter derivatives up to third order are computed in the new function`get_perturbation_params_derivs.m` both analytically and numerically. For numerical derivatives `get_perturbation_params_derivs_numerical_objective.m` is the objective for `fjaco.m` or `hessian_sparse.m` or `hessian.m`.
* `get_perturbation_params_derivs.m` is basically an extended version of the previous `get_first_order_solution_params_derivs.m` function.
* * `get_perturbation_params_derivs_numerical_objective.m`builds upon `identification_numerical_objective.m`. It is used for numerical derivatives, whenever `analytic_derivation_mode=-1|-2`. It takes from `identification_numerical_objective.m` the parts that compute numerical parameter Jacobians of steady state, dynamic model equations, and perturbation solution matrices. Hence, these parts are removed in `identification_numerical_objective.m` and it only computes numerical parameter Jacobian of moments and spectrum which are needed for identification analysis in `get_identification_jacobians.m`, when `analytic_derivation_mode=-1` only.
* Detailed changes:
* Most important: notation of this function is now in accordance to the k_order_solver, i.e. we do not compute derivatives of Kalman transition matrices A and B, but rather the solution matrices ghx,ghu,ghxx,ghxu,ghuu,ghs2,ghxxx,ghxxu,ghxuu,ghuuu,ghxss,ghuss in accordance with notation used in `oo_.dr`. As a byproduct at first-order, focusing on ghx and ghu instead of Kalman transition matrices A and B makes the computations slightly faster for large models (e.g. for Quest the computations were faster by a couple of seconds, not much, but okay).
* Removed use of `kstate`, see also Dynare/dynare#1653 and Dynare/dynare!1656
* Output arguments are stored in a structure `DERIVS`, there is also a flag `d2flag` that computes parameter hessians needed only in `dsge_likelihood.m`.
* Removed `kronflag` as input. `options_.analytic_derivation_mode` is now used instead of `kronflag`.
* Removed `indvar`, an index that was used to selected specific variables in the derivatives. This is not needed, as we always compute the parameter derivatives for all variables first and then select a subset of variables. The selection now takes place in other functions, like `dsge_likelihood.m`.
* Introduced some checks: (i) deterministic exogenous variables are not supported, (ii) Kronecker method only compatible with first-order approximation so reset to sylvester method, (iii) for purely backward or forward models we need to be careful with the rows in `M_.lead_la g_incidence`, (iv) if `_params_derivs.m` files are missing an error is thrown.
* For numerical derivatives, if mod file does not contain an `estimated_params_block`, a temporary one with the most important parameter information is created.
## `unfold_g4.m`
* When evaluating g3 and g4 one needs to take into account that these do not contain symmetric elements, so one needs to use `unfold_g3.m` and the new function `unfold_g4.m`. This returns an unfolded version of the same matrix (i.e. with symmetric elements).
***
# New test models
`.gitignore` and `Makefile.am` are changed accordingly. Also now it is possible to run test suite on analytic_derivatives, i.e. run `make check m/analytic_derivatives`
## `analytic_derivatives/BrockMirman_PertParamsDerivs.mod`
* This is the Brock Mirman model, where we know the exact policy function $g$ for capital and consumption. As this does not imply a nonzero $g_{\sigma\sigma}$, $g_{x\sigma\sigma}$, $g_{u\sigma\sigma}$ I added some artificial equations to get nonzero solution matrices with respect to $\sigma$. The true perturbation solution matrices $g_x$ , $g_u$, $g_{xx}$, $g_{xu}$, $g_{uu}$, $g_{\sigma\sigma}$, $g_{xxx}$, $g_{xxu}$, $g_{xuu}$, $g_{uuu}$, $g_{x\sigma\sigma}$, $g_{u\sigma\sigma}$ are then computed analytically with Matlab's symbolic toolbox and saved in `nBrockMirmanSYM.mat`. There is a preprocessor flag that recreates these analytical computations if changes are needed (and to check whether I made some errors here ;-) )
* Then solution matrices up to third order and their parameter Jacobians are then compared to the ones computed by Dynare's `k_order_solver` and by `get_perturbation_params_derivs` for all `analytic_derivation_mode`'s. There will be an error if the maximum absolute deviation is too large, i.e. for numerical derivatives (`analytic_derivation_mode=-1|-2`) the tolerance is choosen lower (around 1e-5); for analytical methods we are stricter: around 1e-13 for first-order, 1e-12 for second order, and 1e-11 for third-order.
* As a side note, this mod file also checks Dynare's `k_order_solver` algorithm and throws an error if something is wrong.
* This test model shows that the new functionality works well. And analytical derivatives perform way better and accurate than numerical ones, even for this small model.
## `analytic_derivatives/burnside_3_order_PertParamsDerivs.mod`
* This builds upon `tests/k_order_perturbation/burnside_k_order.mod` and computes the true parameter derivatives analytically by hand.
* This test model also shows that the new functionality works well.
## `analytic_derivatives/LindeTrabandt2019.mod`
* Shows that the new functionality also works for medium-sized models, i.e. a SW type model solved at third order with 35 variables (11 states). 2 shocks and 20 parameters.
* This mod file can be used to tweak the speed of the computations in the future.
* Compares numerical versus analytical parameter derivatives (for first, second and third order). Note that this model clearly shows that numerical ones are quite different than analytical ones even at first order!
## `identification/LindeTrabandt2019_xfail.mod`
* This model is a check for issue Dynare/dynare#1595, see fjaco.m below, and will fail.
* Removed `analytic_derivatives/ls2003.mod` as this mod file is neither in the testsuite nor does it work.
***
# Detailed changes in other functions
## `get_first_order_solution_params_derivs.m`
* Deleted, or actually, renamed to `get_perturbation_params_derivs.m`, as this function now is able to compute the derivatives up to third order
## `identification_numerical_objective.m`
* `get_perturbation_params_derivs_numerical_objective.m`builds upon `identification_numerical_objective.m`. It takes from `identification_numerical_objective.m` the parts that compute numerical parameter Jacobians of steady state, dynamic model equations, and perturbation solution matrices. Hence, these parts are removed in `identification_numerical_objective.m` and it only computes numerical parameter Jacobian of moments and spectrum which are needed for identification analysis in `get_identification_jacobians.m`, when `analytic_derivation_mode=-1` only.
## `dsge_likelihood.m`
* As `get_first_order_solution_params_derivs.m`is renamed to `get_perturbation_params_derivs.m`, the call is adapted. That is,`get_perturbation_params_derivs` does not compute the derivatives of the Kalman transition `T`matrix anymore, but instead of the dynare solution matrix `ghx`. So we recreate `T` here as this amounts to adding some zeros and focusing on selected variables only.
* Added some checks to make sure the first-order approximation is selected.
* Removed `kron_flag` as input, as `get_perturbation_params_derivs` looks into `options_.analytic_derivation_mode` for `kron_flag`.
## `dynare_identification.m`
* make sure that setting `analytic_derivation_mode` is set both in `options_ident` and `options_`. Note that at the end of the function we restore the `options_` structure, so all changes are local. In a next merge request, I will remove the global variables to make all variables local.
## `get_identification_jacobians.m`
* As `get_first_order_solution_params_derivs.m`is renamed to `get_perturbation_params_derivs.m`, the call is adapted. That is,`get_perturbation_params_derivs` does not compute the derivatives of the Kalman transition `A` and `B` matrix anymore, but instead of the dynare solution matrix `ghx` and `ghu`. So we recreate these matrices here instead of in `get_perturbation_params_derivs.m`.
* Added `str2func` for better function handles in `fjaco.m`.
## `fjaco.m`
* make `tol`an option, which can be adjusted by changing `options_.dynatol.x`for identification and parameter derivatives purposes.
* include a check and an informative error message, if numerical derivatives (two-sided finite difference method) yield errors in `resol.m` for identification and parameter derivatives purposes. This closes issue Dynare/dynare#1595.
* Changed year of copyright to 2010-2017,2019
***
# Further suggestions and questions
* Ones this is merged, I will merge request an improvement of the identification toolbox, which will work up to third order using the pruned state space. This will also remove some issues and bugs, and also I will remove global variables in this request.
* The third-order derivatives can be further improved by taking sparsity into account and use mex versions for kronecker products etc. I leave this for further testing (and if anybody actually uses this ;-) )
— deprecation warning for “simul” (#1683)
— Ramsey: use information from transformed model for filling
M_.nonzero_hessian_eqs. (Closes: #1681)
Also, M_.hessian_eq_zero is not generated if order = 1
— New field M_.endo_trends (#1648)
In pathological cases, the Jacobian at the initial but true steady state value is ill-behaved. In this case we should return with the valid steady state instead of trying random starting values
previously, `nopathchange` and `nopreprocessoroutput` were set even if they were values instead of option names.
`nopathchange` would further remove all options that contained `'nopathchange'`
e.g. `dynare example1.mod savemacro=nopathchange` would erroneously set `nopathchange` to true and would delete the `savemacro` option altogether
In the fix, just check that the match starts in position 1 as, if the argument passed is longer than the matching pattern (e.g. nopathchangee), the preprocessor will stop processing with a usage error
It’s not clear that such utilities are useful (people should rather update
their MATLAB). It’s also a lot of work, and it’s probably better to focus on
documenting the incompatibilities.
Closes: #915
— use XLS instead of XLSX in testsuite datafiles
— use optimset instead of optimoptions
— use quadv/quadl/quadgk instead of integral
— fix race condition in load_m_file_data_legacy.m tests
— use fallback implementation for intersect(…, 'stable')
Previously, the solver would recompute the Jacobian at every iteration. But, if
an iteration fails, the current point is not moved (only the radius of the
trust region changes), hence it is not necessary to recompute the Jacobian in
that case. This commit implements this optimization.
* split up report output, creating new files for the preamble, the body of the report, and each individual page of the report.
* allow the user to set the page number
* allow for the removal of headers and footers from a given page
Since version 1.6, the optim Forge package has an implementation of fmincon.
Hence we can now use mode_compute=1 under Octave.
This commit also adds tests/optimizers/fs2000_1.mod to the testsuite. It will
be skipped under MATLAB if the optimization toolbox is not there, or under
Octave if optim ≥ 1.6 is not there.
The scripts are based the former “dynare-build” project. They have been
overhauled and simplified.
Building a Windows package (both installer and zip archive) is as easy as
running “make -C windows” (provided the right Debian packages are installed,
use the “windows/install-packages.sh” script for that purpose).
The layout of MEX files for Octave in the package has been
changed (mex/octave/win32/ and mex/octave/win64/ instead of mex/octave32/ and
mex/octave/), for consistency with MATLAB MEX.
It constructs the stacked residuals and jacobian of the perfect foresight
problem.
It is an almost perfect replacement for the perfect_foresight_problem.m
routine, while being much more efficient.
Note however that the DLL never return complex numbers (it instead puts NaNs at
the place where there would have been complex). This may create problems for
some MOD files; the algorithms will need to be adapted to use a more
line-search method.
In those releases, the intersect function was behaving differently when given
arguments with different orientations.
Put the two arguments in the same orientation, to avoid the problem (recalling
that options_.varobs is column-oriented).
# Improvements
* heavily commented (also auxiliary functions) and changed notation to make all the functions (hopefully) more readable and understandable, and hence, easier to debug
* added identification criteria of Komunjer and Ng (2011, Econometrica) and Qu and Tkachenko (2012, Quantitative Economics)
* tests can be turned of, i.e. nostrength disables identification strenght, noreducedform disables reduced form criteria, nomoments disables moment criteria, nospectrum disables spectrum criteria, nominimal disables minimal system criteria
* all kronflags (analytic_derivation_mode) actually work in all functions
* added functionality when there is correlation in Sigma_e and when one wants to consider corr parameters of exogenous shocks. Previously, (1) corr parameters were not allowed when calling identification and (2) when Sigma_e was not diagonal then the toolbox relied on numerical derviatives only (kronflag=-1). Now it is possible to handle both identification of corr parameters as well as correct analytical derivatives when Sigma_e is not diagonal with all possible kronflag values (-1|-2|0|1)
* all plots and results are stored in the same folder named identification (previously there was another one with a capital I (Identification))
# Needed changes to preprocessor
* add as field to options_ident:
- tex (same as in options_)
- nostrength (to turn off identification strength)
- noreducedform (to turn off reduced form criteria)
- nomoments (to turn off Iskrev's moment criteria)
- nominimal (to turn off Komunjer and Ng's minimal system criteria)
- nospectrum (to turn off Qu and Tkachenko's spectrum criteria)
* add to options_ident:
- normalize_jacobians (whether to normalize Jacobians or not)
- grid_nbr (integer used to discretize the interval [-pi;pi]
- tol_rank (tolerance level to compute ranks)
- tol_deriv (tolerance level to select nonzero columns in derivatives)
- tol_sv (tolerance level to select nonzero singular values)
- ChecksViaSubsets (for debugging purposes, uses different function to find problematic parameter sets)
- max_dim_subsets_groups (for debugging purposes, used for ChecksViaSubsets)
# Further Suggestions
* Rename getH.m into getParamsDerivReducedForm.m to make the purpose of this function evident
* Rename getJJ.m into getIdentificationJacobians.m to make the purpose of this function evident
* Rename thet2tau.m into IdentificationNumericalObjectiveFunction.m to make the purpose of this function evident
* dYss, d2Yss, dg1 should also include derivatives wrt to stderr and corr parameters (even though these are just 0), as in other functions (getJJ, dynare_estimation) we always add these manually
* I am pretty sure the current handling in getH.m of dYss and d2Yss is not correct in the case of nonstationary variables (if g2static is nonempty), I added a warning message, as I am not sure whether this is ever used
* It would be straigthforward to also include stderr and corr parameters of measurement errors (these is not possible right now). Should I do this?
* Computations of d2A and d2Om need to be checked, as the differences between computing these with analytically (kronflag=0|1) or numerically kronflag=-1|-2 is really large for the example model of AnSchorfheide.
* I am not sure how to best normalize Qu and Tkachenko's G matrix. It looks (and in the Gaussian case actually is) very similar to the Ahess matrix. So I used the same normalization rule as for the Ahess matrix. See comments in identification_checks.m. Anyone has a better idea? Please also check the models in test/identification/cgg for differences.
* parts that are unclear to me are marked by a [@wmutschl] tag
* the run time of tests/identification/as2007.mod increases from 0h01m27s to 0h03m46s (as Qu and Tkachenko's G matrix takes a little while to compute). One could decrease prior_mc=250 to prior_mc=150.
# New functions
* commutation: Returns Magnus and Neudecker's commutation matrix that solves k*vec(X)=vec(X')
* DerivABCD: Derivative of X(p)=A(p)*B(p)*C(p)*D(p) w.r.t to p as in Magnus and Neudecker (1999), p. 175
* DeriveMinimalState: Derives minimal state space system by checking observability and controllability of all possible combinations of variables
* duplication: Duplication Matrix (and its Moore Penrose Inverse) as defined by Magnus and Neudecker (2002), p.49, Dp*vec(X) = X
* identification_checks_via_subsets: finds problematic parameters in a bruteforce fashion: It computes the rank of the Jacobians for all possible parameter combinations, if the rank condition is not fullfilled, these parameter sets are flagged as non-identifiable. For debugging purposes only, as the current identification_checks.m (based on nullspace and multicorrelation coefficients) is much faster
# Detailed changes in getH.m
* functionality improvements
- heavily commented (also auxiliary functions) and changed notation of several variables to make this function (hopefully) more readable and understandable, and hence, easier to debug
- added functionality when Sigma_e is not diagonal and/or when one wants to consider corr parameters of exogenous shocks independent of the value of kronflag
- fixed function for all values of kronflag, i.e. kronflag=-2|-1|0|1. Previosuly, only kronflag=-2|0 were working, all other kronflags ran into errors (-1 was actually never called , but was dealt with in getJJ.m). I assume kronflag=-1|1 was used only for debugging issues, but still was not working. I fixed this now, the function now works out-of-the-box for all kronflag values.
- I also outlined and documented what each kronflag does and point to the corresponding equations in Ratto and Iskrev (2012) or Iskrev (2010,Appendix A)
- the function additionally outputs the Jacobians of B and Sig, which are needed for Qu and Tkachenko (2012) and Komunjer and Ng (2011)'s criteria
- Moved computation of Jacobian of tau=[ys;vec(A);vech(B * M_.Sigma_e * B')] into getJJ.m to have all Jacobians which are needed for identification in one place. That is, getH.m computes first and second parameter derivatives of (1) reduced-form solution, (2) steady state and (3) Jacobian of dynamic model, whereas getJJ computes and sets up all Jacobians which are used for identification purposes. Therefore, getH might be useful more generally for other purposes than identification. For instance, when doing a GMM estimation, we could use this function to compute analytically the gradient of the moments and provide this to the optimizer used in a GMM context.
* output arguments
- renamed `H` (Jacobian wrt parameters of tau=[ys;vec(A);vech(B * M_.Sigma_e * B')] into dTAU, (as H is very confusing, e.g. in other functions it is a Hessian, or Hss and H2ss is also just the steady state. Morevoer, tau is used in Iskrev(2010) for the steady state and reduced-form solution)
- renamed `Hss` (Jacobian of steady state wrt model parameters only) into `dYss` (as H is very confusing here, see above)
- renamed `H2ss` (Hessian wrt model parameters only of ys) into d2Yss (as H is very confusing, see above)
- renamed `gp` into `dg1`, where g1 corresponds to the same variable as in dynamic model files. Note that in params_deriv files gp lacks the contribution of Jacobian wrt steady state and dg1 includes this using the implicit function theorem as outlined in Ratto and Iskrev (2012). Hence, dg1 denotes Jacobian wrt to parameters. It is useful and important to distinguish gp and dg1.
- added `dB` (Jacobian wrt parameters of solution matrix B) needed for Qu and Tkachenko (2012) as well as Komunjer and Ng (2011)
- added `dSig` (Jacobian wrt parameters of M_.Sigma_e) needed for Qu and Tkachenko (2012) as well as Komunjer and Ng (2011)
* input arguments
- renamed `indx` (index of model parameters to be checked) into `indpmodel`, the p makes it more clear that this is a parameter index
- renamed `indexo` (index of stderr parameters) into `indpstderr`, the p makes it more clear that this is a parameter index
- renamed `iv` (index of variables to consider) into `indvar`
- Renamed `M_` to `M`, `estim_params_` to `estim_params`, `options_` to `options` , `oo_` to `oo` to visualize that these are local and not global variables
- included `indpcorr` a matrix of indices for corr parameters to be checked
* misc
- distinguished clearly between variables in DR or in declaration order without overwriting this in between
- added which functions call getH.m
- updated copyright to 2010-2019
# Detailed changes in getJJ.m
* functionality improvements
- heavily commented and changed notation of several variables to make this function (hopefully) more readable and understandable, and hence, easier to debug
- added functionality when Sigma_e is not diagonal and/or when one wants to consider corr parameters of exogenous shocks independent of the value of kronflag
- tidied the function up, such that it sets up all Jacobians which are needed for identification, i.e. Iskrev's J matrix, Qu and Tkachenko (2012)'s G matrix, Komunjer and Ng (2011)'s D matrix, reduced-form solution (dTAU), linear rational expectation (i.e. Jacobian of steady state and dynamic model equations dLRE).
- dTAU is now constructed in getJJ instead of in getH (see comment above in getH.m)
- works for all kronflags, i.e. for numerical derivatives (-1 and -2) as well as for analytical derivatives based on kronecker products (1) or Sylvester Equations (0)
- added functionality for stderr and corr parameters independent of the value of kronflag (previously this was only possible with numerical derivatives, now it works for all kronflags)
- finds minimal state vector needed for Komunjer and Ng (2011)'s criteria (function `DeriveMinimalState.m`)
- moved computations from kronflag=-1 (which were used in case of corr in shock block) into getH.m, so that getJJ now only sets up the Jacobians for LRE, Iskrev's J, Qu and Tkachenko's G and Komunjer and Ng's D, whereas getH computes the Jacobians (wrt parameters) of A, B, Sigma_e, Om, Yss and g1. This should simplify debugging as everything is now in one place and not in two
* output arguments
- renamed `JJ` into `J`
- renamed `H` into `dTAU` (as H is very confusing, e.g. in other functions it is a Hessian, or Hss and H2ss is also just the steady state. Morevoer, tau is used in Iskrev(2010) for the steady state and reduced-form solution)
- renamed `gp` into `dLRE`, as this corresponds to Jacobian of LRE=[Yss;vec(g1)] where g1 is the Jacobian of the dynamic model equations.
- renamed `gam` into `MOMENTS`
- added `G` for Qu and Tkachenko's Jacobian matrix G
- added `D` for Komunjer and Ng's Jacobian matrix D
- reordered output arguments
* input arguments
- added `options_ident` as input argument; hence, `kronflag`, `nlags` and `useautocorr` are removed from input arguments as these are available in options_ident
- Renamed `M_` to `M`, `estim_params_` to `estim_params`, `options_` to `options` , `oo_` to `oo` to visualize that these are local and not global variables
- renamed `indx` (index of model parameters to be checked) into `indpmodel`, the p makes it more clear that this is a parameter index
- renamed `indexo` (index of stderr parameters) into `indpstderr`, the p makes it more clear that this is a parameter index
- added `indpcorr` (index of corr parameters)
- renamed `mf` (index of VAROBS variables) into `indvobs`
* misc
- updated copyright to 2010-2019
- provided some comments on several ways to compute the spectral density matrix
- added which functions call getJJ.m
# Detailed changes in thet2tau.m
* functionality improvements
- heavily commented and changed notation of several variables to make this function (hopefully) more readable and understandable, and hence, easier to debug
- Added output option to compute spectral density matrix
- Reorded and added some output option.
- Instead of Om, `outputflag=0` computes B and Sigma_e, which are needed for Qu and Tkachenko as well as Komunjer and Ng. The Jacobian of Om is then computed in getJJ or getH from Jacobian of B and Sigma_e. Due to some testing with An and Schorfheide model this seems to be more accurate when I compare these with the analytical derivatives. The old behavior (computing Om directly) can be restored by setting `outputflag=-2`.
- In total this function can now be used to compute numerically Jacobians of Yss, A, B, Sigma_e, Om, g1, autocovariogram and spectral density
- Clearly distinguished (and commented) on the different outputs of this function.
- Works for all types of parameters, ie. model, stderr and corr.
- This function can now also be used when there is no estimated_params block. Previously, there was an error when there was no estimated_params block when calling `set_all_parameters` as this requires some information in `estim_params`. I fixed this by providing a temporary local estim_parms structure with the necessary information on model, stderr and corr parameters. In this way, this can be easily extended to also include stderr and corr parameters of measurement errors.
* output arguments
- renamed `tau` into `out`, as this function computes *very* different things (and not only tau) depending on an input flag
* input arguments
- renamed `flagmoments` into `outputflag` as this function does not only compute moments but many other things (see above)
- renamed `indx` (index of model parameters to be checked) into `indpmodel`, the p makes it more clear that this is a parameter index
- renamed `indexo` (index of stderr parameters) into `indpstderr`, the p makes it more clear that this is a parameter index
- added `indpcorr` (index of corr parameters)
- merged `mf` (index of observable variables) and `iv` (index of variables to consider) into a single index `indvar` as there is no need to distinguish between these two indices (they were never used in combination)
- added `grid_nbr` (number of grid points to compute spectral density)
- reordered input arguments
* misc
- added which functions call thet2tau
- updated copyright to 2010-2019
# Detailed changes in identification_analysis.m
* functionality improvements
- heavily commented and changed notation of several variables to make this function (hopefully) more readable and understandable, and hence, easier to debug
- renamed `dg1` to `dLRE`, renamed `vecg1` to `lre`, renamed `H` to `dTAU` (see comments above)
- added option `numzerotolderiv` with default `1.e-8` used for non-zero derivatives
- added option `numzerotolrank` with default `1.e-10` used for rank computations
- added theoretical identification analysis based on Komunjer and Ng (2011)'s method, i.e. steady state and observational equivalent spectral densities within a minimal system
- added theoretical identification analysis based on Qu and Tkachenko (2012)'s method, i.e. steady state and spectral density
- restructured the code slightly to combined chunks of code that belong together on the one hand, and on the other hand to differentiate between the different criteria
- added call to new function `identification_checks_via_subsets.m` (see above for the definition of the functionality) to perform identification checks differently as find it more intuitive and (most likely) more precise.
* input arguments
- removed `bounds` and `dataset_` as input argument, because these are not needed
- moved `name_tex` and `tittxt` into `options_ident` as these two inputs are only used in `ident_bruteforce.m` and already set in `dynare_identification.m`
* output arguments
- added `ide_spectrum` structure for Qu and Tkachenko's criteria based on the spectral density
- added `ide_minimal` structure for Komunjer and Ng's criteria based on the minimal state space system
- reordered output arguments
* misc
- added which functions call identification_analysis
- updated copyright to 2010-2019
# Detailed changes in dynare_identification.m
* functionality improvements
- heavily commented and changed notation of several variables to make this function (hopefully) more readable and understandable, and hence, easier to debug
- included more options (and default values) which can be set by the user, i.e. nostrength, nomoments, nominimal, nospectrum, tex, tol_rank, tol_deriv, tol_sv, grid_nbr, ChecksViaSubsets, max_dim_subsets_group
- instead of turning warnings globally off, I specified the relevant warnings for matlab and octave, respectively, off
- improved the warning messages slightly
- restructured chunks of code with respect to different criteria
* output arguments
- renamed arguments: TAU to STO_TAU, GAM to STO_MOMENTS, LRE to STO_LRE, gp to STO_si_dLRE, H to STO_si_dTAU, JJ to STO_si_J
- added arguments: STO_G and STO_D for the two new criteria
* misc
- added which functions call dynare_identification
- updated copyright to 2010-2019
# Detailed changes in identification_checks.m
* functionality improvements
- added checks for Komunjer and Ng's D matrix. Note that the Jacobian D=[D_par D_rest], where D_par depends on the parameters and D_rest does not. So this is taken into account.
- added checks for Qu and Tkachenko's G matrix. Note that the Jacobian G is a Gram matrix with dimension nparam x nparam, similar to Ahess. So this is taken into account. I am, however, not sure whether this is correct regarding the multicorrelation and pairwise correlation coefficients. Please double check.
- the rank is now actually computed at the prespecified tolerance level (and not Matlab's default level), so this is in accordance to the further analysis of problematic parameter sets
* output arguments
- added the rank to output arguments which is later also displayed
- replaced the J or JJ part in the variable names with X as this function is used for all sorts of Jacobians, not only Iskrev's J
* input arguments
- renamed hess_flag to output_flag (and clearly outlined what each value does)
- added tol_rank and tol_sv as input arguments, such that the tolerance levels can be changed by the user and not preimplemented in this function
- added param_nbr which is needed for Komunjer and Ng's D matrix
* misc
- updated copyright to 2010-2019
# Detailed changes in ident_bruteforce.m
* functionality improvements
- the output directory was set with a capital I, i.e. Identification, whereas in all other functions we rely on lower case i, i.e. identification. I changed this to lower-cases, so everything is now saved in the same folder.
- changed displayed strings to be more precise with the corresponding papers and notation
* input arguments
- renamed `n` to `max_dim_cova_group` to name options the same across functions
- renamed `pnames_TeX` to `name_tex` to name options the same across functions
- added `tol_deriv` as tolerance level which can be changed by the user
* misc
- Added some comments
- updated copyright to 2010-2019
# Detailed changes in disp_identification.m
* functionality improvements
- this function displays the same output for different Jacobians, hence I put the common code into a for loop. This should simplify changing the output that is printed to the console. Previously the code was simply repeated for the different criteria and only the strings changed.
- some settings relevant for the computation are now printed as a summary to the console
- the tolerance level, rank and required rank are always displayed on the command line to see how many problematic sets there are and which tolerance level was used
- the function is also able to display problematic parameters computed by the new function `identification_checks_via_subsets.m` which is only used for debugging.
* input arguments
- added `idespectrum` structure for analysis based on Qu and Tkachenko
- added `ideminimal` structure for analysis based on Komunjer and Ng
- added `options_ident` to have all necessary settings in a structure
* misc
- Added some comments
- Removed uncommented code that was not used as this was redundant and probably an artifact of the original programming?!
- updated copyright to 2010-2019
# Detailed changes in dsge_likelihood.m
* misc
- adjusted call of getH due to changes of input and output arguments
- updated copyright to 2010-2019
# Detailed changes in cosn.m
* misc
- commented functionality, input and output arguments of this function
- updated copyright to 2010-2019
The gamrnd fallback under matlab/missing/stats/ does not work under Octave
because the +gamrnd/ folder is not accessible (it has the same name as the
function, which does not work under Octave).
Instead of fixing this, rather make the statistics toolbox a requirement, since
anyways it is very easy to obtain under Octave.
Accordingly:
- do not try to run the unit tests of matlab/missing/stats/ under Octave
- merge the matlab/missing/stats-matlab/ into matlab/missing/stats/, since this
directory is now MATLAB-only.
Also:
- move matlab/distributions/+gamrnd/ under matlab/missing/stats/ for
consistency
- in the manual: remove obsolete link to Octave downloads on the Dynare
website; update URL of Octave Forge
Closes#1638
The 'stable' option of intersect(), which keeps the element order of the first
argument, is not available on Octave. Provide a fallback implementation, and
adapt the code.
Sébastien Villemot
Houtan Bastani
Johannes Pfeifer
Marco Ratto
Willi Mutschler
Stéphane Adjemian (Charybdis)
Dóra Kocsis
Michel Juillard
Stéphane Adjemian (Charybdis)
Stéphane Adjemian (Charybdis)
Stéphane Adjemian (Charybdis)
Stéphane Adjemian
Michel Juillard
Sumudu Kankanamge
Marco Ratto