\item Dynare is called from the host language platform with the syntax \texttt{dynare <<filename>>.mod}
\item This call can be followed by certain options:
\begin{itemize}
\item Some of these options impact host language platform functionality, \textit{e.g.}\texttt{nograph} prevents graphs from being displayed in Matlab
\item Some cause differences in the output created by default, \textit{e.g.}\texttt{notmpterms} prevents temporary terms from being written to the static/dynamic files
\item While others impact the functionality of the macroprocessor or the preprocessor, \textit{e.g.}\texttt{nostrict} shuts off certain checks that the preprocessor does by defalut
\item Parsing is the action of transforming an input text (a \texttt{mod} file in our case) into a data structure suitable for computation
\item The parser consists of three components:
\begin{itemize}
\item the \alert{lexical analyzer}, which recognizes the ``words'' of the \texttt{mod} file (analog to the \textit{vocabulary} of a language)
\item the \alert{syntax analyzer}, which recognizes the ``sentences'' of the \texttt{mod} file (analog to the \textit{grammar} of a language)
\item the \alert{parsing driver}, which coordinates the whole process and constructs the data structure using the results of the lexical and syntax analyses
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Lexical analysis}
\begin{itemize}
\item The lexical analyzer recognizes the ``words'' (or \alert{lexemes}) of the language
\item Defined in \texttt{DynareFlex.ll}, it is transformed into C++ source code by the program \texttt{flex}
\item This file details the list of known lexemes (described by regular expressions) and the associated \alert{token} for each of them
\item For punctuation (semicolon, parentheses, \ldots), operators (+, -, \ldots) or fixed keywords (\textit{e.g.}\texttt{model}, \texttt{varexo}, \ldots), the token is simply an integer uniquely identifying the lexeme
\item For variable names or numbers, the token also contains the associated string for further processing
%\item \textit{Note:} the list of tokens can be found at the beginning of \texttt{DynareBison.yy}
\item When invoked, the lexical analyzer reads the next characters of the input, tries to recognize a lexeme, and either produces an error or returns the associated token
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{Lexical analysis}
\framesubtitle{An example}
\begin{itemize}
\item Suppose the \texttt{mod} file contains the following:
\begin{verbatim}
model;
x = log(3.5);
end;
\end{verbatim}
\item Before lexical analysis, it is only a sequence of characters
\item The lexical analysis produces the following stream of tokens:
\item The \texttt{mod} file grammar is described in \texttt{DynareBison.yy}, which is transformed into C++ source code by the program \texttt{bison}
\item The grammar tells a story which looks like:
\begin{itemize}
\item A \texttt{mod} file is a list of statements
\item A statement can be a \texttt{var} statement, a \texttt{varexo} statement, a \texttt{model} block, an \texttt{initval} block, \ldots
\item A \texttt{var} statement begins with the token \texttt{VAR}, then a list of \texttt{NAME}s, then a semicolon
\item A \texttt{model} block begins with the token \texttt{MODEL}, then a semicolon, then a list of equations separated by semicolons, then an \texttt{END} token
\item An equation can be either an expression, or an expression followed by an \texttt{EQUAL} token and another expression
\item An expression can be a \texttt{NAME}, or a \texttt{FLOAT\_NUMBER}, or an expression followed by a \texttt{PLUS} and another expression, \ldots
Using the list of tokens produced by lexical analysis, the syntax analyzer determines which ``sentences'' are valid in the language, according to a \alert{grammar} composed of \alert{rules}.
\begin{block}{A grammar for lists of additive and multiplicative expressions}
\begin{footnotesize}
\begin{verbatim}
%start expression_list;
expression_list := expression SEMICOLON
| expression_list expression SEMICOLON;
expression := expression PLUS expression
| expression TIMES expression
| LEFT_PAREN expression RIGHT_PAREN
| INT_NUMBER;
\end{verbatim}
\end{footnotesize}
\end{block}
\begin{itemize}
\item\texttt{(1+3)*2; 4+5;} will pass the syntax analysis without error
\item\texttt{1++2;} will fail the syntax analysis, even though it has passed the lexical analysis
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Semantic actions}
\begin{itemize}
\item So far we have only described how to accept valid \texttt{mod} files and to reject others
\item Or, if there is a parsing error (unknown keyword, undeclared symbol, syntax error), it displays the line and column numbers where the error occurred and exits
\item A \alert{symbol} is simply the name of a variable (endogenous, exogenous, local, auxiliary, etc), parameter, external function, \ldots basically everything that is not recognized as a Dynare keyword
\item\alert{SymbolTable} is a simple class used to maintain the list of the symbols used in the \texttt{mod} file
\item Local variables outside model: no declaration needed (\textit{e.g.} lhs symbols in equations from \texttt{steady\_state\_model} block, expression outside of model block, \ldots)
\item External functions
\item Trend variables
\item Log Trend variables
\item Unused Endogenous variables (created when \texttt{nostrict} option is passed)
\item using the \texttt{var}, \texttt{varexo}, \texttt{varexo\_det}, \texttt{parameter}, \texttt{external\_function}, \texttt{trend\_var}, and \texttt{log\_trend\_var} declarations
\item\texttt{UnaryOpNode} for unary operators (\textit{e.g.} unary minus, $\log$, $\sin$): contains an enumerator representing the operator, and a pointer to its child
\item\texttt{BinaryOpNode} for binary operators (\textit{e.g.}$+$, $*$, pow): contains an enumerator representing the operator, and pointers to its two children
\item\texttt{TrinaryOpNode} for trinary operators (\textit{e.g.}$normcdf$, $normpdf$): contains an enumerator representing the operator and pointers to its three children
\item The abstract class \texttt{ExprNode} has an abstract sub-class called \texttt{AbstractExternalFunctionNode}
\item This abstract sub-class has 3 sub-classes, corresponding to the 3 types of external function nodes:
\begin{itemize}
\item\texttt{ExternalFunctionNode} for external functions. Contains the identifier of the external function and a vector of its arguments
\item\texttt{FirstDerivExternalFunctionNode} for the first derivative of an external function. In addition to the information contained in \texttt{ExternalFunctionNode}, contains the index w.r.t. which this node is the derivative.
\item\texttt{SecondDerivExternalFunctionNode} for the second derivative of an external function. In addition to the information contained in \texttt{FirstDerivExternalFunctionNode}, contains the index w.r.t. which this node is the second derivative.
\item Class \texttt{ModelTree} is a sub-class container of \texttt{DataTree}, specialized for storing a set of model equations.
\item In the code, we use \texttt{ModelTree}-derived classes: \texttt{DynamicModel} (the model with lags) and \texttt{StaticModel} (the model without lags)
\item several instances of \texttt{DynamicModel}, one each for storing the equations of the model block for the original model, modified model, original Ramsey model, the Ramsey FOCs, etc.
\item one instance of \texttt{StaticModel} for storing the equations of model block without lags
\item It maintains 9 tables which keep track of the already-constructed nodes: one table by type of node (constants, variables, unary ops, binary ops, trinary ops, external functions, first deriv of external functions, second deriv of external functions, local variables)
\item Suppose you want to create the node $e_1+e_2$ (where $e_1$ and $e_2$ are sub-expressions):
\begin{itemize}
\item the algorithm searches the binary ops table for the tuple equal to (address of $e_1$, address of $e_2$, op code of +) (it is the \alert{search key})
\item The class \texttt{ModFile} stores a list of pointers of type \texttt{Statement*}, corresponding to the statements of the \texttt{mod} file, in their order of declaration
\item Heavy use of polymorphism in the check pass, computing pass, and when writing outputs: abstract class \texttt{Statement} provides a virtual method for these 3 actions
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Evaluation context}
\begin{itemize}
\item The \texttt{ModFile} class contains an \alert{evaluation context}
\item It is a map associating a numerical value to some symbols
\item Used in testing that trends are compatible with a balanced growth path, for finding non-zero cross partials of equations with respect to trend variables and endogenous varibales
\item checks for coherence in statements (\textit{e.g.} options passed to statements do not conflict with each other, required options have been passed)
\item checks for coherence among statements (\textit{e.g.} if \texttt{osr} statement is present, ensure \texttt{osr\_params} and \texttt{optim\_weights} statements are present)
\item checks for coherence between statements and attributes of \texttt{mod} file (\textit{e.g.}\texttt{use\_dll} is not used with \texttt{block} or \texttt{bytecode})
\end{itemize}
\end{itemize}
\end{frame}
\section{Transform pass}
\begin{frame}
\frametitle{Transform pass (1/2)}
\begin{itemize}
\item The transform pass is implemented through the method \texttt{ModFile::transformPass(bool nostrict)}
\item It makes necessary transformations (notably to the dynamic model, symbol table, and statements list) preparing the \texttt{ModFile} object for the computing pass. Examples of transformations include:
\begin{itemize}
\item creation of auxiliary variables and equations for leads, lags, expectation operator, differentiated forward variables, etc.
\item detrending of model equations if nonstationary variables are present
\item decreasing leads/lags of predetermined variables by one period
\item addition of FOCs of Langrangian for Ramsey problem
\item addition of \texttt{dsge\_prior\_weight} initialization before all other statements if estimating a DSGE-VAR where the weight of the DSGE prior of the VAR is calibrated
\item same number of endogenous varibables as equations (not done in certain situations, \textit{e.g.} Ramsey, discretionary policy, etc.)
\item correspondence among variables and statements, \textit{e.g.} Ramsey policy, identification, perfect foresight solver, and simul are incompatible with deterministic exogenous variables
\item correspondence among statements, \textit{e.g.} for DSGE-VAR without \texttt{bayesian\_irf} option, the number of shocks must be greater than or equal to the number of observed variables
\item In the context of class \texttt{ModelTree}, a \alert{variable} is a pair (symbol, lag)
\item The symbol must correspond to a variable of type endogenous, exogenous, deterministic exogenous variable, or parameter
\item The \texttt{SymbolTable} class keeps track of valid symbols while the \texttt{variable\_node\_map} keeps track of model variables (symbol, lag pairs stored in \texttt{VariableNode} objects)
\item After the computing pass, the \texttt{DynamicModel} class writes the leag/lag incidence matrix:
\item three rows: the first row indicates $t-1$, the second row $t$, and the third row $t+1$
\item one column for every endogenous symbol in order of declaration; NB: includes endogenous auxiliary variables created during the transform pass
\item elements of the matrix are either 0 (if the variable does not appear in the model) or correspond to the variable's column in the Jacobian of the dynamic model
\item For Ramsey policy: the same as above, but with one further order of derivation than declared by the user with \texttt{order} option (the derivation order is determined in the check pass, see \texttt{RamseyPolicyStatement::checkPass()})
\item Derivation of the model implemented in \texttt{ModelTree::computeJacobian()}, \texttt{ModelTree::computeHessian()}, \texttt{ModelTree::computeThirdDerivatives()}, and \texttt{ModelTree::computeParamsDerivatives()}
\item Simply call \texttt{ExprNode::getDerivative(deriv\_id)} on each equation node
\item for a constant or variable node, derivative is straightforward ($0$ or $1$)
\item for a unary, binary, trinary op nodes and external function nodes, recursively calls method \texttt{computeDerivative()} on children to construct derivative
\item without any computation, you know its derivative w.r.t. $z$ is zero
\item each node stores in an attribute (\texttt{non\_null\_derivatives}) the set of variables which appear in the expression it represents ($\{x,y\}$ in the example)
\item this set is computed in \texttt{prepareForDerivation()}
\item when \texttt{getDerivative(deriv\_id)} is called, immediately returns zero if \texttt{deriv\_id} is not in that set
\item Expression storage in the preprocessor implements maximal sharing but this is not optimal for the Matlab output files, because creating a temporary variable also has a cost (in terms of CPU and of memory)
\item Computation of temporary terms implements a trade-off between:
\begin{itemize}
\item cost of duplicating sub-expressions
\item cost of creating new variables
\end{itemize}
\item Algorithm uses a recursive cost calculation, which marks some nodes as being ``temporary''
\item\textit{Problem}: redundant with optimizations done by the C/C++ compiler (when Dynare is in DLL mode) $\Rightarrow$ compilation very slow on big models
\item For most statements, the method \texttt{computingPass()} is a no-op\ldots
\item\ldots except for \texttt{planner\_objective} statement, which serves to declare planner objective when doing optimal policy under commitment
\item Class \texttt{PlannerObjectiveStatement} contains an instance of \texttt{ModelTree}, which stores the objective function (\texttt{i.e.} only one equation in the tree)
\item call to Matlab functions corresponding to the statements of the \texttt{mod} file (written by calling \texttt{Statement::writeOutput()} on all statements through polymorphism)
\frametitle{Proposed changes with addition of Julia support (2/2)}
\begin{enumerate}
\suite
\item Create libraries out of the preprocessor
\begin{itemize}
\item Pros: Dynare interaction similar across HLPs, preprocessor used as is
\item Cons: difficult for outsiders to contribute, big project, not much benefit in speed when compared to\ldots
\end{itemize}
\item Write \texttt{mod} file from HLP then call preprocessor; option to output JSON file representing \texttt{ModFile} object at every step of the preprocessor
\begin{itemize}
\item Pros: Dynare interaction similar across HLPs, preprocessor used as is, minimal amount of work, easy incremental step, allows users to support any given HPL given the JSON output
\item Cons: unnecessary processing when certain changes made in host language, keeps defaults of current preprocessor, speed (?)
\item Allows interactivity for all HLPs; requires only
\begin{itemize}
\item A definition of a mod file class in the HLP
\item A library function that converts an HLP mod file object to a \texttt{mod} file
\end{itemize}
\item Allows users to use Dynare with any HPL. Standard JSON output can be read in any HPL; user can use it construct desired HPL objects and work with model in their language of preference
\item Easy first step
\item No divergence of codebase: don't need to repeat code (checks, transformations, etc.) across platforms
\item Creates \texttt{mod} files that can be used on other host language platforms
\item Adds one more HLP library to distribute
\item Need to design/implement classes that will store processed dynare \texttt{mod} file in various HLPs