Persist data
Keep entity storage and the index across restarts so objects can be read again.
The two data locations
Omnio data lives in two places.
- Entities: the object bytes, stored through unikvs.
- Index: keys, metadata, embeddings, and purge reservations, stored in the
IObjectIndexdatabase.
You need both to read an object. Back up, migrate, and restore them together.
Node.js files
import { SqliteNodeIndex } from "@omniojs/idx-sqlite.node";
import { NodeFs } from "@unikvs/fs.node";
import * as unikvs from "unikvs";
type EntityMapping = Record<string, unikvs.Value<Uint8Array>>;
const entity = new unikvs.Builder<EntityMapping>().persist(new NodeFs("./data/entities"));
const index = new SqliteNodeIndex({ location: "./data/index.sqlite3" });
NodeFs creates the root directory and stores one file per entityId. The location of SqliteNodeIndex is the SQLite database file.
Browser storage
import { PGliteIndex } from "@omniojs/idx-pglite";
import { Opfs } from "@unikvs/opfs";
import * as unikvs from "unikvs";
type EntityMapping = Record<string, unikvs.Value<Uint8Array>>;
const entity = new unikvs.Builder<EntityMapping>().persist(new Opfs("omnio-entities"));
const index = new PGliteIndex({ location: "idb://omnio-index" });
OPFS and IndexedDB are scoped to a browser origin. Reloading the page keeps the data, but private browsing modes may discard it.
PostgreSQL
import { PostgresqlIndex } from "@omniojs/idx-postgresql.node";
const index = new PostgresqlIndex({
connectionString: process.env.DATABASE_URL!,
});
PostgreSQL stores only the index. Entities still need a persistent store such as NodeFs.
In memory
For tests and prototypes, use @unikvs/memory and an in-memory index. The data is lost when the process exits.
import { SqliteWasmIndex } from "@omniojs/idx-sqlite.wasm";
import { Memory } from "@unikvs/memory";
import * as unikvs from "unikvs";
type EntityMapping = Record<string, unikvs.Value<Uint8Array>>;
const entity = new unikvs.Builder<EntityMapping>().persist(new Memory());
const index = new SqliteWasmIndex();
Back up
- Stop writes. Call
close()if you need to release open connections. - Copy both the index database and the entity root directory.
- Restore with the same configuration and call
open().
Avoid copying a SQLite file while it is being written. For PostgreSQL, take a database backup and copy the entity directory at the same point in time.
Recovery after restart
open() starts background work that:
- physically deletes entities past the grace period;
- drains purge reservations left by an interrupted operation or crash;
- reindexes objects whose embedding model or dimensions changed.
As a result, some incomplete work from a previous run is recovered on the next open().
Caveats
- Multiple processes writing to the same index and entity store are not coordinated by Omnio. You need database-level or application-level locking.
- An
entityIdis immutable. Do not modify the bytes for an existingentityIdoutside Omnio. - Deleting the index without deleting entities leaves orphaned bytes. Treat both as a set.