mypy.server.update

module documentation
(source)

Update build by processing changes using fine-grained dependencies.

Use fine-grained dependencies to update targets in other modules that may be affected by externally-visible changes in the changed modules.

This forms the core of the fine-grained incremental daemon mode. This module is not used at all by the 'classic' (non-daemon) incremental mode.

Here is some motivation for this mode:

  • By keeping program state in memory between incremental runs, we only have to process changed modules, not their dependencies. The classic incremental mode has to deserialize the symbol tables of all dependencies of changed modules, which can be slow for large programs.
  • Fine-grained dependencies allow processing only the relevant parts of modules indirectly affected by a change. Say, if only one function in a large module is affected by a change in another module, only this function is processed. The classic incremental mode always processes an entire file as a unit, which is typically much slower.
  • It's possible to independently process individual modules within an import cycle (SCC). Small incremental changes can be fast independent of the size of the related SCC. In classic incremental mode, any change within a SCC requires the entire SCC to be processed, which can slow things down considerably.

Some terms:

  • A target is a function/method definition or the top level of a module. We refer to targets using their fully qualified name (e.g. 'mod.Cls.method'). Targets are the smallest units of processing during fine-grained incremental checking.
  • A trigger represents the properties of a part of a program, and it gets triggered/fired when these properties change. For example, '<mod.func>' refers to a module-level function. It gets triggered if the signature of the function changes, or if the function is removed, for example.

Some program state is maintained across multiple build increments in memory:

  • The full ASTs of all modules are stored in memory all the time (this includes the type map).
  • A fine-grained dependency map is maintained, which maps triggers to affected program locations (these can be targets, triggers, or classes). The latter determine what other parts of a program need to be processed again due to a fired trigger.

Here's a summary of how a fine-grained incremental program update happens:

  • Determine which modules have changes in their source code since the previous update.
  • Process changed modules one at a time. Perform a separate full update for each changed module, but only report the errors after all modules have been processed, since the intermediate states can generate bogus errors due to only seeing a partial set of changes.
  • Each changed module is processed in full. We parse the module, and run semantic analysis to create a new AST and symbol table for the module. Reuse the existing ASTs and symbol tables of modules that have no changes in their source code. At the end of this stage, we have two ASTs and symbol tables for the changed module (the old and the new versions). The latter AST has not yet been type checked.
  • Take a snapshot of the old symbol table. This is used later to determine which properties of the module have changed and which triggers to fire.
  • Merge the old AST with the new AST, preserving the identities of externally visible AST nodes for which we can find a corresponding node in the new AST. (Look at mypy.server.astmerge for the details.) This way all external references to AST nodes in the changed module will continue to point to the right nodes (assuming they still have a valid target).
  • Type check the new module.
  • Take another snapshot of the symbol table of the changed module. Look at the differences between the old and new snapshots to determine which parts of the changed modules have changed. The result is a set of fired triggers.
  • Using the dependency map and the fired triggers, decide which other targets have become stale and need to be reprocessed.
  • Create new fine-grained dependencies for the changed module. We don't garbage collect old dependencies, since extra dependencies are relatively harmless (they take some memory and can theoretically slow things down a bit by causing redundant work). This is implemented in mypy.server.deps.
  • Strip the stale AST nodes that we found above. This returns them to a state resembling the end of semantic analysis pass 1. We'll run semantic analysis again on the existing AST nodes, and since semantic analysis is not idempotent, we need to revert some changes made during semantic analysis. This is implemented in mypy.server.aststrip.
  • Run semantic analyzer passes 2 and 3 on the stale AST nodes, and type check them. We also need to do the symbol table snapshot comparison dance to find any changes, and we need to merge ASTs to preserve AST node identities.
  • If some triggers haven been fired, continue processing and repeat the previous steps until no triggers are fired.

This is module is tested using end-to-end fine-grained incremental mode test cases (test-data/unit/fine-grained*.test).

Class ​Fine​Grained​Build​Manager No class docstring; 0/12 instance variable, 6/6 methods documented
Function calculate​_active​_triggers Determine activated triggers by comparing old and new symbol tables.
Function dedupe​_modules Undocumented
Function delete​_module Undocumented
Function ensure​_deps​_loaded Ensure that the dependencies on a module are loaded.
Function ensure​_trees​_loaded Ensure that the modules in initial and their deps have loaded trees.
Function find​_relative​_leaf​_module Find a module in a list that directly imports no other module in the list.
Function find​_symbol​_tables​_recursive Find all nested symbol tables.
Function find​_targets​_recursive Find names of all targets that need to reprocessed, given some triggers.
Function find​_unloaded​_deps Find all the deps of the nodes in initial that haven't had their tree loaded.
Function fix​_fg​_dependencies Populate the dependencies with stuff that build may have missed
Function get​_module​_to​_path​_map Undocumented
Function get​_sources Undocumented
Function is​_verbose Undocumented
Function lookup​_target Look up a target by fully-qualified name.
Function propagate​_changes​_using​_dependencies Transitively rechecks targets based on triggers and the dependency map.
Function refresh​_suppressed​_submodules Look for submodules that are now suppressed in target package.
Function replace​_modules​_with​_new​_variants Replace modules with newly builds versions.
Function reprocess​_nodes Reprocess a set of nodes within a single module.
Function target​_from​_node Return the target name corresponding to a deferred node.
Function update​_deps Undocumented
Function update​_module​_isolated Build a new version of one changed module only.
Constant INIT​_SUFFIXES Undocumented
Constant MAX​_ITER Undocumented
Constant SENSITIVE​_INTERNAL​_MODULES Undocumented
Variable ​Blocked​Update Undocumented
Variable ​Normal​Update Undocumented
Variable ​Update​Result Undocumented
def calculate_active_triggers(manager, old_snapshots, new_modules): (source)

Determine activated triggers by comparing old and new symbol tables.

For example, if only the signature of function m.f is different in the new symbol table, return {'<m.f>'}.

Parameters
manager:BuildManagerUndocumented
old​_snapshots:Dict[str, Dict[str, SnapshotItem]]Undocumented
new​_modules:Dict[str, Optional[MypyFile]]Undocumented
Returns
Set[str]Undocumented
def dedupe_modules(modules): (source)

Undocumented

Parameters
modules:List[Tuple[str, str]]Undocumented
Returns
List[Tuple[str, str]]Undocumented
def delete_module(module_id, path, graph, manager): (source)

Undocumented

Parameters
module​_id:strUndocumented
path:strUndocumented
graph:GraphUndocumented
manager:BuildManagerUndocumented
def ensure_deps_loaded(module, deps, graph): (source)

Ensure that the dependencies on a module are loaded.

Dependencies are loaded into the 'deps' dictionary.

This also requires loading dependencies from any parent modules, since dependencies will get stored with parent modules when a module doesn't exist.

Parameters
module:strUndocumented
deps:Dict[str, Set[str]]Undocumented
graph:Dict[str, State]Undocumented
def ensure_trees_loaded(manager, graph, initial): (source)
Ensure that the modules in initial and their deps have loaded trees.
Parameters
manager:BuildManagerUndocumented
graph:Dict[str, State]Undocumented
initial:Sequence[str]Undocumented
def find_relative_leaf_module(modules, graph): (source)

Find a module in a list that directly imports no other module in the list.

If no such module exists, return the lexicographically first module from the list. Always return one of the items in the modules list.

NOTE: If both 'abc' and 'typing' have changed, an effect of the above rule is that
we prefer 'abc', even if both are in the same SCC. This works around a false positive in 'typing', at least in tests.
Args:
modules: List of (module, path) tuples (non-empty) graph: Program import graph that contains all modules in the module list
Parameters
modules:List[Tuple[str, str]]Undocumented
graph:GraphUndocumented
Returns
Tuple[str, str]Undocumented
def find_symbol_tables_recursive(prefix, symbols): (source)

Find all nested symbol tables.

Args:
prefix: Full name prefix (used for return value keys and to filter result so that
cross references to other modules aren't included)

symbols: Root symbol table

Returns a dictionary from full name to corresponding symbol table.

Parameters
prefix:strUndocumented
symbols:SymbolTableUndocumented
Returns
Dict[str, SymbolTable]Undocumented
def find_targets_recursive(manager, graph, triggers, deps, up_to_date_modules): (source)

Find names of all targets that need to reprocessed, given some triggers.

Returns: A tuple containing a:
  • Dictionary from module id to a set of stale targets.
  • A set of module ids for unparsed modules with stale targets.
Parameters
manager:BuildManagerUndocumented
graph:GraphUndocumented
triggers:Set[str]Undocumented
deps:Dict[str, Set[str]]Undocumented
up​_to​_date​_modules:Set[str]Undocumented
Returns
Tuple[Dict[str, Set[FineGrainedDeferredNode]], Set[str], Set[TypeInfo]]Undocumented
def find_unloaded_deps(manager, graph, initial): (source)

Find all the deps of the nodes in initial that haven't had their tree loaded.

The key invariant here is that if a module is loaded, so are all of their dependencies. This means that when we encounter a loaded module, we don't need to explore its dependencies. (This invariant is slightly violated when dependencies are added, which can be handled by calling find_unloaded_deps directly on the new dependencies.)

Parameters
manager:BuildManagerUndocumented
graph:Dict[str, State]Undocumented
initial:Sequence[str]Undocumented
Returns
List[str]Undocumented
def fix_fg_dependencies(manager, deps): (source)
Populate the dependencies with stuff that build may have missed
Parameters
manager:BuildManagerUndocumented
deps:Dict[str, Set[str]]Undocumented
def get_module_to_path_map(graph): (source)

Undocumented

Parameters
graph:GraphUndocumented
Returns
Dict[str, str]Undocumented
def get_sources(fscache, modules, changed_modules): (source)

Undocumented

Parameters
fscache:FileSystemCacheUndocumented
modules:Dict[str, str]Undocumented
changed​_modules:List[Tuple[str, str]]Undocumented
Returns
List[BuildSource]Undocumented
def is_verbose(manager): (source)

Undocumented

Parameters
manager:BuildManagerUndocumented
Returns
boolUndocumented
def lookup_target(manager, target): (source)

Look up a target by fully-qualified name.

The first item in the return tuple is a list of deferred nodes that needs to be reprocessed. If the target represents a TypeInfo corresponding to a protocol, return it as a second item in the return tuple, otherwise None.

Parameters
manager:BuildManagerUndocumented
target:strUndocumented
Returns
Tuple[List[FineGrainedDeferredNode], Optional[TypeInfo]]Undocumented
def propagate_changes_using_dependencies(manager, graph, deps, triggered, up_to_date_modules, targets_with_errors, processed_targets): (source)

Transitively rechecks targets based on triggers and the dependency map.

Returns a list (module id, path) tuples representing modules that contain a target that needs to be reprocessed but that has not been parsed yet.

Processed targets should be appended to processed_targets (used in tests only, to test the order of processing targets).

Parameters
manager:BuildManagerUndocumented
graph:Dict[str, State]Undocumented
deps:Dict[str, Set[str]]Undocumented
triggered:Set[str]Undocumented
up​_to​_date​_modules:Set[str]Undocumented
targets​_with​_errors:Set[str]Undocumented
processed​_targets:List[str]Undocumented
Returns
List[Tuple[str, str]]Undocumented
def refresh_suppressed_submodules(module, path, deps, graph, fscache, refresh_file): (source)

Look for submodules that are now suppressed in target package.

If a submodule a.b gets added, we need to mark it as suppressed in modules that contain "from a import b". Previously we assumed that 'a.b' is not a module but a regular name.

This is only relevant when following imports normally.

Args:
module: target package in which to look for submodules path: path of the module refresh_file: function that reads the AST of a module (returns error messages)

Return a list of errors from refresh_file() if it was called. If the return value is None, we didn't call refresh_file().

Parameters
module:strUndocumented
path:Optional[str]Undocumented
deps:Dict[str, Set[str]]Undocumented
graph:GraphUndocumented
fscache:FileSystemCacheUndocumented
refresh​_file:Callable[[str, str], List[str]]Undocumented
Returns
Optional[List[str]]Undocumented
def replace_modules_with_new_variants(manager, graph, old_modules, new_modules): (source)

Replace modules with newly builds versions.

Retain the identities of externally visible AST nodes in the old ASTs so that references to the affected modules from other modules will still be valid (unless something was deleted or replaced with an incompatible definition, in which case there will be dangling references that will be handled by propagate_changes_using_dependencies).

Parameters
manager:BuildManagerUndocumented
graph:Dict[str, State]Undocumented
old​_modules:Dict[str, Optional[MypyFile]]Undocumented
new​_modules:Dict[str, Optional[MypyFile]]Undocumented
def reprocess_nodes(manager, graph, module_id, nodeset, deps, processed_targets): (source)

Reprocess a set of nodes within a single module.

Return fired triggers.

Parameters
manager:BuildManagerUndocumented
graph:Dict[str, State]Undocumented
module​_id:strUndocumented
nodeset:Set[FineGrainedDeferredNode]Undocumented
deps:Dict[str, Set[str]]Undocumented
processed​_targets:List[str]Undocumented
Returns
Set[str]Undocumented
def target_from_node(module, node): (source)

Return the target name corresponding to a deferred node.

Args:
module: Must be module id of the module that defines 'node'

Returns the target name, or None if the node is not a valid target in the given module (for example, if it's actually defined in another module).

Parameters
module:strUndocumented
node:Union[FuncDef, MypyFile, OverloadedFuncDef]Undocumented
Returns
Optional[str]Undocumented
def update_deps(module_id, nodes, graph, deps, options): (source)

Undocumented

Parameters
module​_id:strUndocumented
nodes:List[FineGrainedDeferredNode]Undocumented
graph:Dict[str, State]Undocumented
deps:Dict[str, Set[str]]Undocumented
options:OptionsUndocumented
def update_module_isolated(module, path, manager, previous_modules, graph, force_removed): (source)

Build a new version of one changed module only.

Don't propagate changes to elsewhere in the program. Raise CompileError on encountering a blocking error.

Args:

module: Changed module (modified, created or deleted) path: Path of the changed module manager: Build manager graph: Build graph force_removed: If True, consider the module removed from the build even it the

file exists

Returns a named tuple describing the result (see above for details).

Parameters
module:strUndocumented
path:strUndocumented
manager:BuildManagerUndocumented
previous​_modules:Dict[str, str]Undocumented
graph:GraphUndocumented
force​_removed:boolUndocumented
Returns
UpdateResultUndocumented
INIT_SUFFIXES: tuple[str, ...] = (source)

Undocumented

Value
('/__init__.py', '/__init__.pyi')
MAX_ITER: int = (source)

Undocumented

Value
1000
SENSITIVE_INTERNAL_MODULES = (source)

Undocumented

Value
tuple(core_modules)+('mypy_extensions', 'typing_extensions')
BlockedUpdate = (source)

Undocumented

NormalUpdate = (source)

Undocumented

UpdateResult = (source)

Undocumented