ethdebug: Publish the state variables in the program-level context

Changelog.md

@@ -7,6 +7,7 @@ Compiler Features:
 * Commandline Interface: Selecting an ethdebug output no longer implicitly enables the `ethdebug` and `ast-id` components of `--debug-info`; without `ethdebug` in the selection the `--ethdebug-program` and `--ethdebug-program-runtime` outputs carry no semantic debug info.
 * ethdebug: Stamp the `ethdebug/format/program` and `ethdebug/format/info/resources` outputs with their schema and the version of the ethdebug/format specification they follow (experimental).
 * ethdebug: Populate the type and pointer tables of `ethdebug/format/info/resources` with the type documents of every type a contract's state variables, functions and modifiers reference and with a pointer template for every struct, array and mapping type of its state variables (experimental).
+* ethdebug: Publish the state variables in storage and transient storage and, in the deployed program, the immutable variables in the program-level context of the `ethdebug/format/program` outputs, with their declaration, type reference and pointer (experimental).
 * Standard JSON Interface: Remove the experimental `yulCFGJson` output.
 * Standard JSON Interface: Selecting an ethdebug output no longer implicitly enables the `ethdebug` and `ast-id` components of `settings.debug.debugInfo`; without `ethdebug` in the selection the `evm.bytecode.ethdebug` and `evm.deployedBytecode.ethdebug` outputs carry no semantic debug info.
 * Yul Optimizer: Improve performance of `DataFlowAnalyzer` using flat hash containers.

docs/compiler-guide/ethdebug.rst

@@ -24,7 +24,7 @@ Which Types Get a Document
 
 The resources of a contract contain a document for
 
-- the types of its state variables in storage and transient storage,
+- the types of its state variables in storage and transient storage and of its immutable variables,
 - the types of the parameters and return variables of every function and modifier compiled into the contract: the ones it defines, the ones it inherits, the free functions of the source units it references and the internal functions of the libraries in those source units,
 - and, transitively, every type these types are composed of.
 

docs/ethdebug.rst

@@ -163,3 +163,11 @@ The Programs
 A program (schema ``ethdebug/format/program``) describes one bytecode: the creation bytecode or the deployed bytecode of a contract.
 It names the ``contract`` and the source range of its definition, states the ``environment`` the bytecode runs in, ``create`` or ``call``, and lists its ``instructions``.
 Every instruction carries its byte ``offset`` in the bytecode, the ``operation`` with the ``mnemonic`` of the opcode and the ``arguments`` of a push, and, where the compiler knows it, a ``context`` with the source range of the code the instruction was generated from.
+
+The program of a contract with state variables in storage or transient storage also carries a program-level ``context`` listing them as ``variables``.
+Every such variable names its ``identifier`` and the source range of its ``declaration``, refers to its ``type`` by identifier into the type table and carries its ``pointer``: a single region for a value type, a reference to the template of its type with ``slot`` bound to the variable's slot for a struct, an array, ``bytes`` or ``string``, and the region of its base slot for a mapping, whose entries the mapping's template locates once a key is bound.
+
+The deployed program lists the immutable variables of the contract as well, after the variables in storage and transient storage.
+The deployed code holds a copy of the value of an immutable wherever the code reads it, so its pointer is a region of the ``code`` covering the value in one of those copies.
+The value takes the trailing bytes of the 32-byte copy, or the leading ones for a ``bytesN`` value, the way it is aligned on the stack.
+An immutable that the code never reads has no copy and is not listed, and neither is any immutable in the creation program, which holds the values in memory until it returns the deployed code.

libevmasm/Ethdebug.cpp

@@ -171,7 +171,13 @@ schema::materials::Compilation materialCompilation(std::vector<Source> const& _s
 
 } // anonymous namespace
 
-Json ethdebug::program(std::string_view _name, unsigned _sourceID, Assembly const& _assembly, LinkerObject const& _linkerObject)
+Json ethdebug::program(
+	std::string_view _name,
+	unsigned _sourceID,
+	Assembly const& _assembly,
+	LinkerObject const& _linkerObject,
+	std::optional<schema::program::Context> _programContext
+)
 {
 	schema::Program const result{
 		.ethdebug = schema::data::Stamp{"ethdebug/format/program", std::string{SpecificationVersion}},
@@ -187,7 +193,7 @@ Json ethdebug::program(std::string_view _name, unsigned _sourceID, Assembly cons
 			}
 		},
 		.environment = _assembly.isCreation() ? schema::Program::Environment::CREATE : schema::Program::Environment::CALL,
-		.context = std::nullopt,
+		.context = std::move(_programContext),
 		.instructions = programInstructions(_assembly, _linkerObject, _sourceID)
 	};
 	return Json(result);

libevmasm/Ethdebug.h

@@ -25,6 +25,7 @@
 #include <libevmasm/LinkerObject.h>
 
 #include <map>
+#include <optional>
 
 namespace solidity::evmasm::ethdebug
 {
@@ -45,7 +46,13 @@ struct Source
 constexpr std::string_view SpecificationVersion = "0.1.0-draft.2";
 
 // returns ethdebug/format/program, stamped with its schema and the specification version.
-Json program(std::string_view _name, unsigned _sourceID, Assembly const& _assembly, LinkerObject const& _linkerObject);
+Json program(
+	std::string_view _name,
+	unsigned _sourceID,
+	Assembly const& _assembly,
+	LinkerObject const& _linkerObject,
+	std::optional<schema::program::Context> _programContext = std::nullopt
+);
 
 // returns ethdebug/format/info/resources, stamped with its schema and the specification version.
 Json resources(

liblangutil/EthdebugSchema.cpp

@@ -595,7 +595,9 @@ void schema::program::to_json(Json& _json, Context::Variable const& _contextVari
 {
 	auto const numProperties =
 		_contextVariable.identifier.has_value() +
-		_contextVariable.declaration.has_value();
+		_contextVariable.declaration.has_value() +
+		_contextVariable.type.has_value() +
+		_contextVariable.pointer.has_value();
 	solRequire(numProperties >= 1, EthdebugException, "Context variable has no properties.");
 	if (_contextVariable.identifier)
 	{
@@ -604,6 +606,13 @@ void schema::program::to_json(Json& _json, Context::Variable const& _contextVari
 	}
 	if (_contextVariable.declaration)
 		_json["declaration"] = *_contextVariable.declaration;
+	if (_contextVariable.type)
+		_json["type"] = *_contextVariable.type;
+	if (_contextVariable.pointer)
+	{
+		solAssert(!hasInternalExpression(*_contextVariable.pointer), "Context variable pointer contains a compiler-internal expression.");
+		_json["pointer"] = *_contextVariable.pointer;
+	}
 }
 
 void schema::program::to_json(Json& _json, Context const& _context)
@@ -1348,3 +1357,202 @@ std::map<std::string, schema::Pointer::Template> schema::info::pointersFromJson(
 	}
 	return pointers;
 }
+
+namespace
+{
+
+/// Collects the free names of pointers, resolving template references. The
+/// free names of a table template are computed once, with the template's
+/// own parameters bound, and cached.
+class FreeNameCollector
+{
+public:
+	explicit FreeNameCollector(std::map<std::string, schema::Pointer::Template> const& _templates):
+		m_templates(_templates)
+	{}
+
+	std::set<std::string> collect(schema::Pointer const& _pointer)
+	{
+		std::set<std::string> free;
+		std::set<std::string> bound;
+		LocalTemplates localTemplates;
+		visit(_pointer, bound, localTemplates, free);
+		return free;
+	}
+
+private:
+	using LocalTemplates = std::map<std::string, schema::Pointer::Template const*>;
+
+	/// Binds names for the lifetime of the object, restoring the previous state afterwards.
+	class ScopedNames
+	{
+	public:
+		ScopedNames(std::set<std::string>& _bound, std::vector<std::string> const& _names):
+			m_bound(_bound)
+		{
+			for (std::string const& name: _names)
+				if (m_bound.insert(name).second)
+					m_added.emplace_back(name);
+		}
+		~ScopedNames()
+		{
+			for (std::string const& name: m_added)
+				m_bound.erase(name);
+		}
+
+	private:
+		std::set<std::string>& m_bound;
+		std::vector<std::string> m_added;
+	};
+
+	void visit(schema::Pointer::Expression const& _expression, std::set<std::string> const& _bound, std::set<std::string>& _free)
+	{
+		using Pointer = schema::Pointer;
+		auto const operands = [&](Pointer::Operands const& _operands) {
+			for (Pointer::Expression const& operand: _operands)
+				visit(operand, _bound, _free);
+		};
+		std::visit(util::GenericVisitor{
+			// A Yul local is a dependency on the current Yul code by
+			// definition; nothing binds it. An unbound variable is one too.
+			[&](Pointer::YulLocal const& _yulLocal) { _free.insert(_yulLocal.name); },
+			[&](Pointer::Variable const& _variable)
+			{
+				if (!_bound.count(_variable.identifier))
+					_free.insert(_variable.identifier);
+			},
+			[&](Pointer::Arithmetic const& _arithmetic) { operands(_arithmetic.operands); },
+			[&](Pointer::Keccak256 const& _keccak256) { operands(_keccak256.operands); },
+			[&](Pointer::Concat const& _concat) { operands(_concat.operands); },
+			[&](Pointer::Resize const& _resize) { visit(*_resize.operand, _bound, _free); },
+			// Literals, constants, lookups and reads: the latter two name
+			// regions, which live in a separate namespace.
+			[](auto const&) {}
+		}, _expression.value);
+	}
+
+	void visit(
+		std::optional<schema::Pointer::Expression> const& _expression,
+		std::set<std::string> const& _bound,
+		std::set<std::string>& _free
+	)
+	{
+		if (_expression)
+			visit(*_expression, _bound, _free);
+	}
+
+	void visit(
+		schema::Pointer const& _pointer,
+		std::set<std::string>& _bound,
+		LocalTemplates& _localTemplates,
+		std::set<std::string>& _free
+	)
+	{
+		using Pointer = schema::Pointer;
+		auto const sub = [&](std::shared_ptr<Pointer const> const& _sub) {
+			if (_sub)
+				visit(*_sub, _bound, _localTemplates, _free);
+		};
+		std::visit(util::GenericVisitor{
+			[&](Pointer::Region const& _region)
+			{
+				visit(_region.slot, _bound, _free);
+				visit(_region.offset, _bound, _free);
+				visit(_region.length, _bound, _free);
+			},
+			[&](Pointer::Group const& _group)
+			{
+				for (Pointer const& member: _group.members)
+					visit(member, _bound, _localTemplates, _free);
+			},
+			[&](Pointer::List const& _list)
+			{
+				visit(_list.count, _bound, _free);
+				ScopedNames index{_bound, {_list.each}};
+				sub(_list.is);
+			},
+			[&](Pointer::Conditional const& _conditional)
+			{
+				visit(_conditional.condition, _bound, _free);
+				sub(_conditional.then);
+				sub(_conditional.otherwise);
+			},
+			[&](Pointer::Scope const& _scope)
+			{
+				// Definitions are ordered: each may reference the earlier ones.
+				std::vector<std::string> definedNames;
+				for (auto const& [name, value]: _scope.definitions)
+				{
+					visit(value, _bound, _free);
+					definedNames.emplace_back(name);
+				}
+				ScopedNames defined{_bound, definedNames};
+				sub(_scope.in);
+			},
+			[&](Pointer::TemplateReference const& _reference)
+			{
+				for (std::string const& name: freeNamesOfTemplate(_reference.name, _localTemplates))
+					if (!_bound.count(name))
+						_free.insert(name);
+			},
+			[&](Pointer::Templates const& _templates)
+			{
+				// Locally defined templates are visible to references inside the target.
+				std::vector<std::string> added;
+				for (auto const& [name, definition]: _templates.templates)
+					if (_localTemplates.emplace(name, &definition).second)
+						added.emplace_back(name);
+				sub(_templates.in);
+				for (std::string const& name: added)
+					_localTemplates.erase(name);
+			}
+		}, _pointer.value);
+	}
+
+	/// The free names of a template, with its own parameters bound. Local
+	/// templates shadow the table's; results for the table's templates are
+	/// cached, results for local ones are recomputed per reference.
+	std::set<std::string> freeNamesOfTemplate(std::string const& _name, LocalTemplates& _localTemplates)
+	{
+		if (m_resolving.count(_name))
+			return {};
+
+		schema::Pointer::Template const* definition = nullptr;
+		bool fromTable = false;
+		if (auto const local = _localTemplates.find(_name); local != _localTemplates.end())
+			definition = local->second;
+		else if (auto const it = m_templates.find(_name); it != m_templates.end())
+		{
+			definition = &it->second;
+			fromTable = true;
+			if (auto const cached = m_tableTemplateFreeNames.find(_name); cached != m_tableTemplateFreeNames.end())
+				return cached->second;
+		}
+		if (!definition)
+			return {};
+
+		m_resolving.insert(_name);
+		std::set<std::string> free;
+		std::set<std::string> bound;
+		{
+			ScopedNames parameters{bound, definition->expect};
+			visit(*definition->body, bound, _localTemplates, free);
+		}
+		m_resolving.erase(_name);
+
+		if (fromTable)
+			m_tableTemplateFreeNames.emplace(_name, free);
+		return free;
+	}
+
+	std::map<std::string, schema::Pointer::Template> const& m_templates;
+	std::set<std::string> m_resolving;
+	std::map<std::string, std::set<std::string>> m_tableTemplateFreeNames;
+};
+
+}
+
+std::set<std::string> schema::freeNames(Pointer const& _pointer, std::map<std::string, Pointer::Template> const& _templates)
+{
+	return FreeNameCollector{_templates}.collect(_pointer);
+}

liblangutil/EthdebugSchema.h

@@ -25,6 +25,7 @@
 #include <map>
 #include <memory>
 #include <optional>
+#include <set>
 #include <string>
 #include <string_view>
 #include <utility>
@@ -592,8 +593,11 @@ struct Context
 	{
 		std::optional<std::string> identifier;
 		std::optional<materials::SourceRange> declaration;
-		// TODO: type
-		// TODO: pointer according to ethdebug/format/spec/pointer
+		/// The variable's type, usually a reference into the type resources.
+		std::optional<Type::Specifier> type;
+		/// Where the value lives. Omitted for a variable whose pointer is a
+		/// template expecting parameters, which stays in the pointer resources.
+		std::optional<Pointer> pointer;
 	};
 
 	std::optional<materials::SourceRange> code;
@@ -726,6 +730,14 @@ Pointer::Template templateFromJson(Json const& _json, std::string_view _path, Po
 /// contains a compiler-internal expression such as YulLocal.
 bool hasInternalExpression(Pointer const& _pointer);
 
+/// The names @a _pointer reads and nothing inside it binds: Yul locals and
+/// variables that no scope definition, list index or template parameter
+/// binds. Template references are followed through the templates blocks
+/// enclosing them and through @a _templates, with the template's own
+/// parameters bound; a reference to an unknown template contributes
+/// nothing, and neither does a template that references itself.
+std::set<std::string> freeNames(Pointer const& _pointer, std::map<std::string, Pointer::Template> const& _templates);
+
 namespace info
 {
 /// The type table of a resources object, keyed by type ID.

libsolidity/interface/CompilerStack.cpp

@@ -83,6 +83,7 @@
 
 #include <boost/algorithm/string/replace.hpp>
 
+#include <range/v3/algorithm/min_element.hpp>
 #include <range/v3/view/concat.hpp>
 #include <range/v3/view/map.hpp>
 
@@ -1212,8 +1213,28 @@ Json CompilerStack::ethdebug(Contract const& _contract, bool _runtime) const
 	if (!assembly)
 		return {};
 
-	solAssert(sourceIndices().contains(_contract.contract->sourceUnitName()));
-	return evmasm::ethdebug::program(_contract.contract->name(), sourceIndices()[_contract.contract->sourceUnitName()], *assembly, object);
+	std::map<std::string, unsigned> const sourceIndexMap = sourceIndices();
+	solAssert(sourceIndexMap.contains(_contract.contract->sourceUnitName()));
+
+	// The code generator names an immutable after the AST ID of its declaration;
+	// the creation code holds no copies.
+	std::map<int64_t, size_t> immutableOffsets;
+	if (_runtime)
+		for (VariableDeclaration const* immutable: ContractType(*_contract.contract).immutableVariables())
+			for (auto const& [name, codeOffsets]: object.immutableReferences | ranges::views::values)
+				if (name == std::to_string(immutable->id()) && !codeOffsets.empty())
+					immutableOffsets[immutable->id()] = *ranges::min_element(codeOffsets);
+
+	return evmasm::ethdebug::program(
+		_contract.contract->name(),
+		sourceIndexMap.at(_contract.contract->sourceUnitName()),
+		*assembly,
+		object,
+		ethdebug::programContext(
+			ethdebug::stateVariableScope(*_contract.contract, sourceIndexMap, immutableOffsets),
+			ethdebug::resources(*_contract.contract, sourceIndexMap)
+		)
+	);
 }
 
 bytes CompilerStack::cborMetadata(std::string const& _contractName, bool _forIR) const