Skip to content

Add Reflect JSON importer#509

Open
rossbrg wants to merge 14 commits intoobsidianmd:masterfrom
rossbrg:codex/reflect-import-pr
Open

Add Reflect JSON importer#509
rossbrg wants to merge 14 commits intoobsidianmd:masterfrom
rossbrg:codex/reflect-import-pr

Conversation

@rossbrg
Copy link
Copy Markdown

@rossbrg rossbrg commented Feb 27, 2026

Fixes #248.

This adds a native Reflect (.json) importer so people can move from Reflect to Obsidian without relying on HTML export.

The issue describes formatting loss when importing Reflect via HTML. This importer reads Reflect's JSON export directly and converts the ProseMirror document model to Markdown.

What is included

  • New ReflectImporter wired into the importer list (Reflect (.json)).
  • Conversion support for common Reflect content:
    • headings, paragraphs, blockquotes, horizontal rules, code blocks, embeds
    • bullet lists, task lists, and legacy list nodes
    • backlinks (including aliases), tags, and inline formatting marks
    • images as embeds (with optional local download)
  • Import options in the UI:
    • download attachments (opt-in)
    • add YAML tags
    • add YAML created/updated timestamps
    • add YAML title
  • Reflect notes are written with preserved note timestamps (ctime/mtime).

Hardening work from real export edge cases

  • Note filename collision handling is deterministic, while backlink targets stay stable.
  • Backlink labels with ], |, or \\ fall back to Markdown link form when wiki-link syntax would be unsafe.
  • Nested list handling preserves non-paragraph blocks inside list items (for example headings, code blocks, and blockquotes).
  • Attachment download handling avoids path collisions and deduplicates repeated image URLs across notes.
  • Reflect image proxy URLs (reflect.academy/_next/image) are unwrapped to download the original asset URL.
  • Import toggle values are initialized and snapshotted at import start to avoid state drift mid-run.

Files and fixtures

  • New importer and converter:
    • src/formats/reflect-json.ts
    • src/formats/reflect/convert.ts
    • src/formats/reflect/models.ts
  • Registration and metadata updates:
    • src/main.ts
    • manifest.json
    • README.md
  • Regression fixtures:
    • tests/reflect/sample-reflect-export.json
    • tests/reflect/hardening-edge-cases.json

Guideline alignment

  • Implemented in TypeScript with no new runtime dependencies.
  • Importer UI is added through the existing modal/settings pattern.
  • Follows the existing importer architecture (FormatImporter + registration in main.ts).
  • Kept import processing sequential to avoid concurrency-related memory risk.

Testing

  • npm run build

Related

- Use case-insensitive claimed note path keys to avoid case-only filename collisions during import.

- Refactor legacy list conversion so non-paragraph children stay properly nested under list items.

- Count attachments only when files are actually downloaded (not cache-hit path reuse).
Copy link
Copy Markdown
Contributor

@tgrosinger tgrosinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on adding this new format! Looks like a great start so far.

I left a few comments in line. In addition, please consider padding human-readable error messages to reportFailed(). Passing error objects through directly may result in verbose and hard to understand errors, or [object Object] being logged. The full error can be logged to the console.

.gitignore Outdated
# Exclude files specific to personal development process
.hotreload
.devtarget

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not add these entries here.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI if you need to exclude files locally without modifying .gitignore, you can modify your .git/info/exclude file.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, thats on me 🤦

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed these

src/main.ts Outdated

if (selectedId && importers.hasOwnProperty(selectedId)) {
let importer = this.importer = new selectedImporter.importer(this.app, this);
importer.init();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change necessary? This should probably be reverted.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted this

titleFrontmatter: boolean;

init() {
// Initialize defaults in init() because FormatImporter calls init() from its constructor.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This contradicts another change in this PR.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be consistent again now that the extra importer.init() call in main.ts is gone


const MAX_FILENAME_LENGTH = 200;

function truncateTitle(title: string): string {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use truncateText

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to truncateText


// Build frontmatter
let content = result.markdown;
const frontMatter: Record<string, any> = {};
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const frontMatter: Record<string, any> = {};
const frontMatter: FrontmatterCache = {};

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to FrontMatterCache

idToSubject: Map<string, string>;
tags: Set<string>;
images: ImageInfo[];
depth: number;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appears to be unused.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unused depth field from ConvertContext

if (note.daily_at) {
return moment(note.daily_at).format(userDNPFormat);
}
return truncateTitle(note.subject);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Provide a fallback title ("Untitled") if the returned value would be empty.

Suggested change
return truncateTitle(note.subject);
return truncateTitle(note.subject).trim() || 'Untitled';

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the Untitled fallback here

};
}

private async downloadImage(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are images generally downloaded from? Is it a centralized location which would require rate-limiting or backoffs?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good question. Reflect assets are stored with a link that looks something like this

https://reflect-assets.app/v1/users/tfgFpw.../999abc79-e45...?key=2227f9…

I doubt they have strict rate limits, but it's probably not a bad idea to put it in there as a safety measure.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I actually didn't have enough images to even know if there were limits. My tests on my own export only had around 190 images and they all downloaded fine. I'll add some sort of rate limiting

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added retries with backoff for attachment downloads and respect Retry-After when the server sends it. Downloads are still sequential.

note.subject,
convertOptions,
);
const outputPath = idToOutputPath.get(note.id);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider putting the note.id in the frontmatter like is done in other importer format. That would allow incremental imports in the future.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added id to the frontmatter


async import(ctx: ImportContext) {
// Snapshot option values for this run so they can't drift mid-import.
const shouldDownloadAttachments = this.downloadAttachments === true;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The downloadAttachments toggle is snapshotted at import start (here) with a comment about preventing mid-import drift, but tagsFrontmatter, dateFrontmatter, and titleFrontmatter are read directly from this throughout the loop (lines 265, 282, 285, 289).

Either all four settings should be snapshotted for consistency, or none of them need to be — the import modal presumably isn't interactive while the import is running.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, all four settings are snapshotted at import start now

@avi-cenna
Copy link
Copy Markdown

@rossbrg Thanks for your work on this; I'm currently in the process of importing Reflect Notes into Obsidian (just using a temporary Python script for the time being).

rossbrg and others added 7 commits March 16, 2026 23:27
- Revert .gitignore to upstream state (remove personal dev entries)
- Remove duplicate importer.init() call from main.ts
- Move this.init() into FormatImporter base constructor
- Use truncateText utility with 'Untitled' fallback for note titles
- Wrap JSON parsing in try/catch with structural validation
- Use FrontMatterCache type, include note.id unconditionally
- Snapshot all four settings consistently at import start
- Add retry/backoff for image downloads (max 4 attempts, exponential delay)
- Pass human-readable error messages to reportFailed()
- Remove unused depth field from ConvertContext
- Fix ordered list numbering and nested list handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract sanitizeFileName into standalone src/sanitize-file-name.ts to
break the transitive obsidian dependency, enabling Node.js test imports.
Wire up node:test + esbuild bundling with a smoke test for convertDocument.
Change hardBreak output from bare '\n' to '<br>\n' in both convertNode
(block-level) and convertInline paths so forced line breaks render
correctly in Obsidian/CommonMark.
Add text.trim() guards and skippedContent flag to all three list
serializers so tag-only items produce no bare bullets. Archived
annotations are preserved since the guard checks after appending
the <!-- archived --> comment.
Add prefixFirstLine helper so headings as first children of bullet and
legacy list items appear on the same line as the prefix (e.g. "- ### Heading")
instead of on a separate indented line. Task list items are intentionally
unchanged since heading semantics don't work with checkbox prefixes.
@rossbrg
Copy link
Copy Markdown
Author

rossbrg commented Mar 19, 2026

@tgrosinger Just pushed a follow-up pass for the review comments, appreciate the help.

This removes the local .gitignore changes, drops the extra importer.init() call, switches title truncation to truncateText, adds an Untitled fallback, validates the Reflect JSON before import, and makes the reportFailed() messages readable while still logging the full error.

I also changed the frontmatter object to FrontMatterCache, added id to frontmatter, snapshotted all four import settings the same way, and added retry/backoff handling for attachment downloads.

While testing this against my own Reflect export, I ran into a few small converter issues and fixed them here too: hard breaks now render as
, tag-only list items are skipped when inline tags are stripped, legacy ordered lists keep their numbering, and headings at the start of a list item stay on the same bullet instead of creating a blank one.

@sigrunixia
Copy link
Copy Markdown

@tgrosinger When this is merged, please merge obsidianmd/obsidian-help#1039 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import from Reflect

4 participants