docs: Improve SEO and site structure with robots directives and sitemap config - #94
Merged
Merged
Conversation
- Exclude auth/dashboard/login from sitemap and robots.txt, and add noindex meta to those client-rendered account pages - Delete leftover Docusaurus scaffold page (markdown-page.md) - Add static SSR content to the DAG Visualizer page so crawlers see real text instead of just a client-side "Loading…" shell - Document why trailingSlash can't be forced without breaking the site's relative doc links; recommend a host-level redirect instead
github-actions
Bot
changed the title
Contributor
|
🚀 Web preview: https://preview-94.treq-9zy.pages.dev |
Move the static SEO intro text into a <noscript> block so it's still present in the server-rendered HTML for crawlers, but doesn't render a second <h1> alongside the client component's own heading once JS hydrates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves the site's SEO and search engine crawling behavior by adding proper robot directives to prevent indexing of private pages, configuring the sitemap to exclude non-public routes, and enhancing the DAG Visualizer page with better documentation and breadcrumb navigation.
Key Changes
noindex, nofollowmeta tags to private pages:/dashboard,/login, and/auth/callback,/auth/reset-passwordrobots.txtto explicitly disallow crawling of/dashboard,/login, and/auth/routes/toolsdocusaurus.config.tsexplaining whytrailingSlashis intentionally left unset to preserve relative doc link resolutionmarkdown-page.mdexample fileImplementation Details
Headcomponent from@docusaurus/Headto inject robot directives/auth/**) to comprehensively exclude authentication routeshttps://claude.ai/code/session_01D5XCzhhzFkLie9b9HA8DiL