Redacted manifest

Overview

Status

Scanner and diff built, server designed

What done looks like

One tagged file, through all five parts.

1. 1

   Manifest mapping

   Your scanner writes each tagged file down as a number and a date. The file stays home.

   Scroll to section
2. 2

   Cuts and publishing

   To share the work itself, cut a page for someone. A separate act from the manifest.

   See spec
3. 3

   Visibility system

   One visibility list per file decides which manifest and which cut it lands in.

   No spec yet
4. 4

   Community Data Commons MCP

   An always-on server answers a member’s AI with only the ids and dates they may see.

   See spec
5. 5

   Activity log data

   Your record and the community’s overlap. The overlap proves you took part.

   See spec

The two records

Both come out of the same scan. Only the second one crosses the boundary.

Full manifest

One row per file in the folder: id, path, hash, lastActive, bytes and visibility. It also records how each number is made and any ids two paths share. It stays on the owner’s machine, or in the private zone of the community server.

{
  "scannedAt": "<YYYY-MM-DD>",
  "manifest": {
    "idScheme": "sha256(target-relative path), first 6 hex, prefixed 0x",
    "hashScheme": "sha256(file bytes), first 12 hex",
    "lastActiveFrom": "filesystem mtime, local calendar day",
    "idCollisions": [],
    "rows": [
      { "id": "0x<6 hex>", "path": "<folder/file.md>", "hash": "<12 hex>",
        "lastActive": "<YYYY-MM-DD>", "bytes": <n>, "visibility": ["member-b"] }
    ]
  }
}

Redacted manifest

Written only when a tag is named, because without a recipient there is nobody to redact for. Only rows whose visibility list carries that tag, and only two fields on each, newest first.

{
  "audience": "member-b",
  "scannedAt": "<YYYY-MM-DD>",
  "idScheme": "sha256(target-relative path), first 6 hex, prefixed 0x",
  "rows": [
    { "id": "0x<6 hex>", "lastActive": "<YYYY-MM-DD>" }
  ]
}

What never appears in it

No path, file name, extension, folder name, byte count or content hash.

The two numbers

Two hashes, because they answer two different questions.

id · which row is this

SHA-256 of the file’s path relative to the scanned folder, first 6 hex characters, prefixed 0x. Recomputed on every run, so nothing is written back into anyone’s files. A rename gives the file a new id.

hash · did the content change

SHA-256 of the file’s bytes, first 12 hex characters. A sync, a checkout or a backup can move a file’s modified time without changing a byte. A diff on dates alone reports those as edits.

Why the hash stays private

A content hash is the same for anyone who holds the same file. Publishing it would let them confirm a match and learn the content by elimination.

Collisions

Six hex characters can collide. When two paths share an id, the scanner lists them under idCollisions instead of letting one hide the other.

The scanner

Read-only. It writes its own two files and touches nothing in the folder it reads.

Require nothing

Every input is optional. A folder with no frontmatter and no tags still produces a full manifest; the redacted one is empty. The scanner reports what it looked for and did not find.

Walk

Every file under the folder, skipping .git, node_modules, .obsidian, .venv and __pycache__.

Read tags

The top-level visibility key in YAML frontmatter, as an inline list, a block list or a single value. A nested value is recorded as present, never guessed.

Date

The modified time as a local calendar day. A UTC timestamp can stamp tomorrow’s date on this evening’s work.

Run it

A folder, an output folder for this run, and the tag to redact for.

node scan.mjs <folder> --out runs/<date> --share member-b

The diff

Compares two scans of the same folder over time. Never two different folders: a path id only means something inside one folder.

Added and removed

An id in only the later scan, or only the earlier one.

Edited

Same id, different content hash.

Touched but identical

Same id and hash, different date. Reported apart from edits, because nothing changed.

Renamed or moved

A removed row and an added row with the same content hash. Claimed only when exactly one of each carries that hash. With more on either side it stays an add and a remove, and the diff reports the count, because pairing would be guessing.

On a redacted manifest

No hashes, so only shape: a row appeared, vanished or changed its date. The diff prints that limit both ways: a date change may not be an edit, and a same-day edit reads as unchanged.

Run it

Either kind of file, in any combination.

node diff.mjs runs/<earlier>/scan.json runs/<later>/scan.json

Test it

Five files and four changes prove every rule above. Use a test folder, never real work.

The fixture

Five Markdown files tagged member-b, each with different text. Scan. Then edit one, rename one without changing its bytes, delete one, add one, and move one file’s date to another day without changing its bytes. Scan again.

Pass

The diff reports exactly one each: added, removed, edited, renamed, touched but identical, unchanged.

Leak check

Search the redacted file for every file name, the test folder’s name, the .md extension and every value of the full manifest’s hash field. Pass when none is found.

Quiet check

Scan an unchanged folder twice and diff the two. Pass when every row reads unchanged.

Hand it to an agent

Four prompts for a coding agent such as Claude Code. Each stands alone. Run them in order in an empty folder, and fill in the angle brackets.

1 · Build the scanner

Writes scan.mjs.

Build a read-only command-line scanner in Node.js called scan.mjs. Use only the standard library.

Usage: node scan.mjs <folder> [--out <dir>] [--share <tag>]

1. Walk <folder> recursively. Skip .git, node_modules, .obsidian, .venv and __pycache__. Never write anything inside <folder>: --out defaults to ./scan-out, and an --out inside <folder> is refused with an error.
2. For every file compute:
   - id: SHA-256 of the path relative to <folder>, with forward slashes, first 6 hex characters, prefixed 0x.
   - hash: SHA-256 of the file’s bytes, first 12 hex characters.
   - lastActive: the modified time as a local calendar day, YYYY-MM-DD. Do not use toISOString(); it is UTC and can stamp tomorrow’s date on this evening’s work.
   - visibility: for files with YAML frontmatter, the values of the top-level visibility key, whether written as [a, b], as a block list or as one value. If it holds a map or a nested list, put the string "<nested>" in its place; do not guess. A file with no frontmatter or no visibility key gets null.
3. Require nothing. A folder with no frontmatter still produces a manifest. In the summary, count what you looked for and did not find: files with no frontmatter, files with no visibility key, nested values.
4. Write <out>/scan.json shaped as { "scannedAt": <local YYYY-MM-DD>, "manifest": { idScheme, hashScheme, lastActiveFrom, idCollisions, rows } }. idCollisions lists every id shared by two or more paths, with the paths. rows are { id, path, hash, lastActive, bytes, visibility }, sorted by path. The three scheme fields are one-line descriptions of how each value is made.
5. Only when --share <tag> is given, also write <out>/manifest.<tag>.redacted.json holding { audience, scannedAt, idScheme, rows }. rows holds only files whose visibility list contains <tag>, each exactly { id, lastActive }, sorted newest first and then by id. No path, file name, extension, byte count or content hash. Leave the content hash out on purpose: anyone holding the same file could match it.
6. Print a short summary: files read, rows written, collisions, and the paths of the files written.

No server, no network call, no sync. When it runs, show me the redacted file from a scan of a test folder.

Copy prompt

2 · Build the diff

Writes diff.mjs beside the scanner.

Build diff.mjs beside scan.mjs. Use only the standard library. Read-only.

Usage: node diff.mjs <earlier.json> <later.json> [--out <file>]

Each input is either a full scan (rows under manifest.rows) or a redacted manifest (rows at the top level). Report which kind each one is. For anything else, or for an input where two rows share an id, exit with a clear error.

Match rows by id and classify:
- added: id only in the later file
- removed: id only in the earlier file
- edited: same id, different hash, whatever the date does
- touched but identical: same id and hash, different lastActive
- unchanged: same id, hash and lastActive
- renamed or moved: a removed row and an added row sharing one hash. Claim it only when exactly one removed and exactly one added row carry that hash, then take both out of added and removed. Otherwise leave them as they are and report how many hashes were ambiguous.

Use hashes only when every row of both inputs has one. Otherwise compare on shape only: appeared, vanished, date moved, unchanged. Print that limit in the output, both ways: a date change may not be an edit, a same-day edit reads as unchanged, and a rename reads as one row vanishing and another appearing.

Print counts for each class and the rows in each. Write JSON only when --out is given.

Copy prompt

3 · Test both

Runs the fixture, the leak check and the quiet check.

Test scan.mjs and diff.mjs on a test folder. Do not touch any real files.

1. Make a folder with five Markdown files named note-1.md to note-5.md. Give each YAML frontmatter with visibility: [member-b] and different body text.
2. Run: node scan.mjs <test folder> --out runs/before --share member-b
3. Change the folder: edit the text of file 1; rename file 2 without changing its bytes; delete file 3; add file 6 tagged member-b; set file 4’s modified time to a different day without changing its bytes. Leave file 5 alone.
4. Run: node scan.mjs <test folder> --out runs/after --share member-b
5. Run: node diff.mjs runs/before/scan.json runs/after/scan.json
   Pass when it reports exactly one each of added, removed, edited, renamed, touched but identical, and unchanged.
6. Leak check: search the rows of runs/after/manifest.member-b.redacted.json for every file name with and without .md, the test folder’s name, the string .md and every value of the hash field in runs/after/scan.json. Pass when none is found.
7. Quiet check: scan the folder twice more, to runs/q1 and runs/q2, and diff those two scans. Pass when every row is unchanged.

Report each check as pass or fail with the counts you measured. If a check fails, fix the code and run all the checks again.

Copy prompt

4 · Connect it to a server

Optional. Takes the manifest to the Local Assembly. Read that page first.

Read these two pages in full before writing any code:
- https://houstonbloom.github.io/houstonbloom/spec-manifest.html
- https://houstonbloom.github.io/houstonbloom/spec-local-assembly.html

Then extend the MCP server from the Local Assembly page so it answers from the scanner’s output:
1. The owner’s machine sends only scan.json, to a private folder on the server that the MCP tools never return directly.
2. When a member asks what changed, the server derives that member’s redacted rows from the full manifest, using the member’s tag, and diffs them against the last redacted manifest it sent that member.
3. Use the content hashes on the server side to drop rows that were only touched, but return only id and lastActive to the member. Never return a path, a hash or a file name.
4. After answering, save the rows just sent as that member’s new baseline.
5. Check the community token and the member identity on every call, as the Local Assembly page describes.

Both pages list decisions that are still open, such as how often the owner syncs and whether answers carry an “as of” date. Do not decide them. Stop and ask me, and say which one you hit.

Copy prompt

How it works

Scanner

Reads only the files you tagged for the community

Full manifest

Stays private, outside the part of the server anyone can ask

Redacted manifest

An opaque number and a last-active date

Compare

The server checks it against the last broadcast and answers only the person who asked

Baseline

After an answer, the redacted manifest becomes the new baseline

Settled

Shapes, not content

The work stays on its owner’s machine

Two separate acts

Contributing a shape and granting access to the work

Still open

Full manifest

Content or metadata only

Commons contribution

How a pair’s manifest becomes one is not specified

Field names

This page uses id and lastActive; the Local Assembly page calls the same two fields file_number and last_updated

Last active

The scanner uses the file’s modified time; the design leans toward the frontmatter last-updated field, which survives a move

Rename

The path id changes when a file moves. A fingerprint written once into frontmatter would survive it. The switch waits for the first rename the diff cannot pair on real files

Design principles

Whether they travel in the redacted manifest or open at the next depth of trust

Transport

Getting the redacted manifest to another person is not built

Read and use

• SHA-256 in Node.js: crypto.createHash (nodejs.org)