feat(validation): implement incremental validation (#17)#283
Open
ntoulasm wants to merge 8 commits intohyperjump-io:mainfrom
Open
feat(validation): implement incremental validation (#17)#283ntoulasm wants to merge 8 commits intohyperjump-io:mainfrom
ntoulasm wants to merge 8 commits intohyperjump-io:mainfrom
Conversation
…o#17) Previous implementation was built on the assumption that we only cared to track dependencies that resolved to actual files in the workspace. This proved wrong because we need to know the intent of a schema A to reference a schema B so the dependencies are complete in case B becomes available.
Author
|
Hi @jdesrosiers, this PR is ready for review. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #17
This PR implements incremental validation. The server now tracks the dependencies of the workspace and revalidates only the set of schemas affected by a change.
Key Changes
Dependenciesservice, which is responsible for tracking the dependencies of the workspace and computing which schemas are affected by each change.validate-workspace.jsto pass all changes to theDependenciesservice and revalidate only the returned schemas instead of all the schemas in the workspace.build-server.jsto instantiate theDependenciesservice and pass it to theValidateWorkspacefeature.validate-workspace.test.tsand implemented some integration tests to verify the incremental validation functionality.Implementation Details
The
Dependenciesservice stores aDependencyRecordfor each schema that exists in the workspace. TheDependencyRecordstores aSetof all its dependencies and aSetof all of its definitions (uriand the provided$ids). TheDependenciesservice also stores adependentsMapto easily find the dependents of a file when it is edited. Although aDependencyRecordindicates that a schema exists in the workspace, this is not true for thedependentsMapbecause an existing schema may reference a schema that does not exist in the workspace (yet).When the workspace is changed, the
ValidateWorkspaceFeaturecallsDependencies.syncand passes the changes. Then theDependenciesservice gets updated and the affected schemas are returned to be validated. If the changes are empty, it is considered a full dependencies discovery, so theDependenciesservice clears its internal state and starts from scratch returning all the schemas in the workspace. Otherwise it gets updated with the changes and returns only the affected schemas.To find the affected schemas for a change, first we need to retrieve the
DependencyRecordfor the changed schema, which is easy because the dependency records are stored in aMapwith theiruris as keys. From theDependencyRecordwe can get all its handles from thedefinitionsSet. Using these handles as indexes to thedependentsMapwe can get all its direct dependents. TheDependenciesservice uses this logic recursively, while checking if a node is already visited to avoid infinite loops. Also, theuriof the changed file itself is added to theSetof the affected files. When a change occurs, the affected files actually need to be computed twice, once with the old workspace state and once with the current one. This is necessary because of 2 edge cases:uriand its$ids no longer exist. Or when it is changed and its provided$ids changed. If someone depended on this schema and the affected schemas were computed with the new workspace image, we would miss that.uriand its$ids become available. Or when it's changed and its provided$ids changed. If a schema depended on the previously missing schema and the affected schemas were computed based on the previous image we would miss that.The
Dependenciesservice is also incrementally updated:DependencyRecordis removed from therecordsand itsuriis removed from thedependentsvalues.DependencyRecordfor this schema is created and itsuriis added to thedependentsvalues for each of its dependencies.