13.6.9

Fix typing of `Y.Map` iterators

.github/workflows/node.js.yml

@@ -16,16 +16,16 @@ jobs:
 
     strategy:
       matrix:
-        node-version: [16.x, 18.x]
+        node-version: [16.x, 20.x]
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - name: Use Node.js ${{ matrix.node-version }}
-      uses: actions/setup-node@v1
+      uses: actions/setup-node@v3
       with:
         node-version: ${{ matrix.node-version }}
     - run: npm ci
     - run: npm run lint
-    - run: npm run test-extensive
+    - run: npm run test
       env:
         CI: true

INTERNALS.md

@@ -149,8 +149,8 @@ concepts that can be used to create a custom network protocol:
 
 * `update`: The Yjs document can be encoded to an *update* object that can be
   parsed to reconstruct the document. Also every change on the document fires
-an incremental document updates that allows clients to sync with each other.
+an incremental document update that allows clients to sync with each other.
-The update object is an Uint8Array that efficiently encodes `Item` objects and
+The update object is a Uint8Array that efficiently encodes `Item` objects and
 the delete set.
 * `state vector`: A state vector defines the known state of each user (a set of
   tuples `(client, clock)`). This object is also efficiently encoded as a

LICENSE

@@ -1,7 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2014
+Copyright (c) 2023
-  - Kevin Jahns <kevin.jahns@rwth-aachen.de>.
+  - Kevin Jahns <kevin.jahns@protonmail.com>.
   - Chair of Computer Science 5 (Databases & Information Systems), RWTH Aachen University, Germany
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

README.md

@@ -60,12 +60,19 @@ on Yjs. [![Become a Sponsor](https://img.shields.io/static/v1?label=Become%20a%2
   sharing analyses, documentation, spreadsheets, and dashboards.
 * [Nosgestesclimat](https://nosgestesclimat.fr/groupe) The french carbon
   footprint calculator has a group P2P mode based on yjs
+* [oorja.io](https://oorja.io) Online meeting spaces extensible with
+  collaborative apps, end-to-end encrypted.
+* [LegendKeeper](https://legendkeeper.com) Collaborative campaign planner and
+  worldbuilding app for tabletop RPGs.
+* [IllumiDesk](https://illumidesk.com/) Build courses and content with A.I.
+* [btw](https://www.btw.so) Open-source Medium alternative
 
 ## Table of Contents
 
 * [Overview](#Overview)
   * [Bindings](#Bindings)
   * [Providers](#Providers)
+  * [Ports](#Ports)
 * [Getting Started](#Getting-Started)
 * [API](#API)
   * [Shared Types](#Shared-Types)
@@ -137,6 +144,14 @@ Also includes a peer-sync mechanism to catch up on missed updates.
 an append-only log of CRDT local updates (hypercore). Multifeed manages and sync
 hypercores and y-dat listens to changes and applies them to the Yjs document.
 </dd>
+  <dt><a href="https://github.com/liveblocks/liveblocks">@liveblocks/yjs</a></dt>
+  <dd>
+<a href="https://liveblocks.io/document/yjs">Liveblocks Yjs</a> provides a fully
+hosted WebSocket infrastructure and persisted data store for Yjs
+documents. No configuration or maintenance is required. It also features
+Yjs webhook events, REST API to read and update Yjs documents, and a
+browser DevTools extension.
+  </dd>
   <dt><a href="https://github.com/yousefED/matrix-crdt">Matrix-CRDT</a></dt>
   <dd>
 Use <a href="https://www.matrix.org">Matrix</a> as an off-the-shelf backend for
@@ -150,9 +165,30 @@ Encryption (E2EE).
   <dd>
 Adds persistent storage to a server with MongoDB. Can be used with the
 y-websocket provider.
+</dd>
+  <dt><a href="https://github.com/toeverything/AFFiNE/tree/master/packages/y-indexeddb">
+@toeverything/y-indexeddb</a></dt>
+  <dd>
+Like y-indexeddb, but with sub-documents support and fully TypeScript.
 </dd>
 </dl>
 
+# Ports
+
+There are several Yjs-compatible ports to other programming languages.
+
+* [y-octo](https://github.com/toeverything/y-octo) - Rust implementation by
+[AFFiNE](https://affine.pro)
+* [y-crdt](https://github.com/y-crdt/y-crdt) - Rust implementation with multiple
+language bindings to other languages
+  * [yrs](https://github.com/y-crdt/y-crdt/tree/main/yrs) - Rust interface
+  * [ypy](https://github.com/y-crdt/ypy) - Python binding
+  * [yrb](https://github.com/y-crdt/yrb) - Ruby binding
+  * [yswift](https://github.com/y-crdt/yswift) - Swift binding
+  * [yffi](https://github.com/y-crdt/y-crdt/tree/main/yffi) - C-FFI
+  * [ywasm](https://github.com/y-crdt/y-crdt/tree/main/ywasm) - WASM binding
+* [ycs](https://github.com/yjs/ycs) - .Net compatible C# implementation.
+
 ## Getting Started
 
 Install Yjs and a provider with your favorite package manager:
@@ -662,7 +698,8 @@ type. Doesn't log types that have not been defined (using
   <b><code>on('update', function(updateMessage:Uint8Array, origin:any, Y.Doc):void)</code></b>
   <dd>
 Listen to document updates. Document updates must be transmitted to all other
-peers. You can apply document updates in any order and multiple times.
+peers. You can apply document updates in any order and multiple times. Use `updateV2`
+to receive V2 events.
   </dd>
   <b><code>on('beforeTransaction', function(Y.Transaction, Y.Doc):void)</code></b>
   <dd>Emitted before each transaction.</dd>
@@ -754,7 +791,7 @@ const diff2 = Y.diffUpdate(currentState2, stateVector1)
 
 // sync clients
 currentState1 = Y.mergeUpdates([currentState1, diff2])
-currentState1 = Y.mergeUpdates([currentState1, diff1])
+currentState2 = Y.mergeUpdates([currentState2, diff1])
 ```
 
 #### Obfuscating Updates
@@ -787,8 +824,10 @@ Yjs implements two update formats. By default you are using the V1 update format
 You can opt-in into the V2 update format wich provides much better compression.
 It is not yet used by all providers. However, you can already use it if
 you are building your own provider. All below functions are available with the
-suffix "V2". E.g. `Y.applyUpdate` ⇒ `Y.applyUpdateV2`. We also support conversion
+suffix "V2". E.g. `Y.applyUpdate` ⇒ `Y.applyUpdateV2`. Also when listening to updates
-functions between both formats: `Y.convertUpdateFormatV1ToV2` & `Y.convertUpdateFormatV2ToV1`.
+you need to specifically need listen for V2 events e.g. `yDoc.on('updateV2', …)`.
+We also support conversion functions between both formats:
+`Y.convertUpdateFormatV1ToV2` & `Y.convertUpdateFormatV2ToV1`.
 
 #### Update API
 
@@ -1033,7 +1072,7 @@ doc.transact(() => {
   ytext.insert(0, 'abc')
 }, 41)
 undoManager.undo()
-ytext.toString() // => '' (not tracked because 41 is not an instance of
+ytext.toString() // => 'abc' (not tracked because 41 is not an instance of
                  //        `trackedTransactionorigins`)
 ytext.delete(0, 3) // revert change
 

package-lock.json

@@ -1,15 +1,15 @@
 {
   "name": "yjs",
-  "version": "13.6.6",
+  "version": "13.6.9",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "yjs",
-      "version": "13.6.6",
+      "version": "13.6.9",
       "license": "MIT",
       "dependencies": {
-        "lib0": "^0.2.74"
+        "lib0": "^0.2.86"
       },
       "devDependencies": {
         "@rollup/plugin-commonjs": "^24.0.1",
@@ -2481,9 +2481,9 @@
       }
     },
     "node_modules/lib0": {
-      "version": "0.2.74",
+      "version": "0.2.86",
-      "resolved": "https://registry.npmjs.org/lib0/-/lib0-0.2.74.tgz",
+      "resolved": "https://registry.npmjs.org/lib0/-/lib0-0.2.86.tgz",
-      "integrity": "sha512-roj9i46/JwG5ik5KNTkxP2IytlnrssAkD/OhlAVtE+GqectrdkfR+pttszVLrOzMDeXNs1MPt6yo66MUolWSiA==",
+      "integrity": "sha512-kxigQTM4Q7NwJkEgdqQvU21qiR37twcqqLmh+/SbiGbRLfPlLVbHyY9sWp7PwXh0Xus9ELDSjsUOwcrdt5yZ4w==",
       "dependencies": {
         "isomorphic.js": "^0.2.4"
       },
@@ -2492,7 +2492,7 @@
         "0serve": "bin/0serve.js"
       },
       "engines": {
-        "node": ">=14"
+        "node": ">=16"
       },
       "funding": {
         "type": "GitHub Sponsors ❤",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "yjs",
-  "version": "13.6.6",
+  "version": "13.6.9",
   "description": "Shared Editing Library",
   "main": "./dist/yjs.cjs",
   "module": "./dist/yjs.mjs",
@@ -75,7 +75,7 @@
   },
   "homepage": "https://docs.yjs.dev",
   "dependencies": {
-    "lib0": "^0.2.74"
+    "lib0": "^0.2.86"
   },
   "devDependencies": {
     "@rollup/plugin-commonjs": "^24.0.1",

src/index.js

@@ -18,8 +18,10 @@ export {
   Item,
   AbstractStruct,
   GC,
+  Skip,
   ContentBinary,
   ContentDeleted,
+  ContentDoc,
   ContentEmbed,
   ContentFormat,
   ContentJSON,
@@ -93,6 +95,9 @@ export {
   obfuscateUpdate,
   obfuscateUpdateV2,
   UpdateEncoderV1,
+  UpdateEncoderV2,
+  UpdateDecoderV1,
+  UpdateDecoderV2,
   equalDeleteSets,
   snapshotContainsUpdate
 } from './internals.js'

src/structs/ContentType.js

@@ -108,7 +108,7 @@ export class ContentType {
     while (item !== null) {
       if (!item.deleted) {
         item.delete(transaction)
-      } else {
+      } else if (item.id.clock < (transaction.beforeState.get(item.id.client) || 0)) {
         // This will be gc'd later and we want to merge it if possible
         // We try to merge all deleted items after each transaction,
         // but we have no knowledge about that this needs to be merged
@@ -120,7 +120,7 @@ export class ContentType {
     this.type._map.forEach(item => {
       if (!item.deleted) {
         item.delete(transaction)
-      } else {
+      } else if (item.id.clock < (transaction.beforeState.get(item.id.client) || 0)) {
         // same as above
         transaction._mergeStructs.push(item)
       }

src/structs/Item.js

@@ -389,9 +389,8 @@ export class Item extends AbstractStruct {
     }
     if ((this.left && this.left.constructor === GC) || (this.right && this.right.constructor === GC)) {
       this.parent = null
-    }
+    } else if (!this.parent) {
-    // only set parent if this shouldn't be garbage collected
+      // only set parent if this shouldn't be garbage collected
-    if (!this.parent) {
       if (this.left && this.left.constructor === Item) {
         this.parent = this.left.parent
         this.parentSub = this.left.parentSub

src/types/AbstractType.js

@@ -683,7 +683,7 @@ export const typeListInsertGenericsAfter = (transaction, parent, referenceItem,
   packJsonContent()
 }
 
-const lengthExceeded = error.create('Length exceeded!')
+const lengthExceeded = () => error.create('Length exceeded!')
 
 /**
  * @param {Transaction} transaction
@@ -696,7 +696,7 @@ const lengthExceeded = error.create('Length exceeded!')
  */
 export const typeListInsertGenerics = (transaction, parent, index, content) => {
   if (index > parent._length) {
-    throw lengthExceeded
+    throw lengthExceeded()
   }
   if (index === 0) {
     if (parent._searchMarker) {
@@ -798,7 +798,7 @@ export const typeListDelete = (transaction, parent, index, length) => {
     n = n.right
   }
   if (length > 0) {
-    throw lengthExceeded
+    throw lengthExceeded()
   }
   if (parent._searchMarker) {
     updateMarkerChanges(parent._searchMarker, startIndex, -startLength + length /* in case we remove the above exception */)

src/types/YMap.js

@@ -41,7 +41,7 @@ export class YMapEvent extends YEvent {
  * A shared Map implementation.
  *
  * @extends AbstractType<YMapEvent<MapType>>
- * @implements {Iterable<MapType>}
+ * @implements {Iterable<[string, MapType]>}
  */
 export class YMap extends AbstractType {
   /**
@@ -152,7 +152,7 @@ export class YMap extends AbstractType {
   /**
    * Returns the values for each element in the YMap Type.
    *
-   * @return {IterableIterator<any>}
+   * @return {IterableIterator<MapType>}
    */
   values () {
     return iterator.iteratorMap(createMapIterator(this._map), /** @param {any} v */ v => v[1].content.getContent()[v[1].length - 1])
@@ -161,10 +161,10 @@ export class YMap extends AbstractType {
   /**
    * Returns an Iterator of [key, value] pairs
    *
-   * @return {IterableIterator<any>}
+   * @return {IterableIterator<[string, MapType]>}
    */
   entries () {
-    return iterator.iteratorMap(createMapIterator(this._map), /** @param {any} v */ v => [v[0], v[1].content.getContent()[v[1].length - 1]])
+    return iterator.iteratorMap(createMapIterator(this._map), /** @param {any} v */ v => /** @type {any} */ ([v[0], v[1].content.getContent()[v[1].length - 1]]))
   }
 
   /**
@@ -183,7 +183,7 @@ export class YMap extends AbstractType {
   /**
    * Returns an Iterator of [key, value] pairs
    *
-   * @return {IterableIterator<any>}
+   * @return {IterableIterator<[string, MapType]>}
    */
   [Symbol.iterator] () {
     return this.entries()

src/utils/Doc.js

@@ -113,7 +113,7 @@ export class Doc extends Observable {
         this.whenSynced = provideSyncedPromise()
       }
       this.isSynced = isSynced === undefined || isSynced === true
-      if (!this.isLoaded) {
+      if (this.isSynced && !this.isLoaded) {
         this.emit('load', [])
       }
     })

src/utils/Transaction.js

@@ -166,18 +166,29 @@ export const addChangedTypeToTransaction = (transaction, type, parentSub) => {
 /**
  * @param {Array<AbstractStruct>} structs
  * @param {number} pos
+ * @return {number} # of merged structs
  */
-const tryToMergeWithLeft = (structs, pos) => {
+const tryToMergeWithLefts = (structs, pos) => {
-  const left = structs[pos - 1]
+  let right = structs[pos]
-  const right = structs[pos]
+  let left = structs[pos - 1]
-  if (left.deleted === right.deleted && left.constructor === right.constructor) {
+  let i = pos
-    if (left.mergeWith(right)) {
+  for (; i > 0; right = left, left = structs[--i - 1]) {
-      structs.splice(pos, 1)
+    if (left.deleted === right.deleted && left.constructor === right.constructor) {
-      if (right instanceof Item && right.parentSub !== null && /** @type {AbstractType<any>} */ (right.parent)._map.get(right.parentSub) === right) {
+      if (left.mergeWith(right)) {
-        /** @type {AbstractType<any>} */ (right.parent)._map.set(right.parentSub, /** @type {Item} */ (left))
+        if (right instanceof Item && right.parentSub !== null && /** @type {AbstractType<any>} */ (right.parent)._map.get(right.parentSub) === right) {
+          /** @type {AbstractType<any>} */ (right.parent)._map.set(right.parentSub, /** @type {Item} */ (left))
+        }
+        continue
       }
     }
+    break
   }
+  const merged = pos - i
+  if (merged) {
+    // remove all merged structs from the array
+    structs.splice(pos + 1 - merged, merged)
+  }
+  return merged
 }
 
 /**
@@ -224,9 +235,9 @@ const tryMergeDeleteSet = (ds, store) => {
       for (
         let si = mostRightIndexToCheck, struct = structs[si];
         si > 0 && struct.id.clock >= deleteItem.clock;
-        struct = structs[--si]
+        struct = structs[si]
       ) {
-        tryToMergeWithLeft(structs, si)
+        si -= 1 + tryToMergeWithLefts(structs, si)
       }
     }
   })
@@ -318,23 +329,25 @@ const cleanupTransactions = (transactionCleanups, i) => {
           const structs = /** @type {Array<GC|Item>} */ (store.clients.get(client))
           // we iterate from right to left so we can safely remove entries
           const firstChangePos = math.max(findIndexSS(structs, beforeClock), 1)
-          for (let i = structs.length - 1; i >= firstChangePos; i--) {
+          for (let i = structs.length - 1; i >= firstChangePos;) {
-            tryToMergeWithLeft(structs, i)
+            i -= 1 + tryToMergeWithLefts(structs, i)
           }
         }
       })
       // try to merge mergeStructs
       // @todo: it makes more sense to transform mergeStructs to a DS, sort it, and merge from right to left
       //        but at the moment DS does not handle duplicates
-      for (let i = 0; i < mergeStructs.length; i++) {
+      for (let i = mergeStructs.length - 1; i >= 0; i--) {
         const { client, clock } = mergeStructs[i].id
         const structs = /** @type {Array<GC|Item>} */ (store.clients.get(client))
         const replacedStructPos = findIndexSS(structs, clock)
         if (replacedStructPos + 1 < structs.length) {
-          tryToMergeWithLeft(structs, replacedStructPos + 1)
+          if (tryToMergeWithLefts(structs, replacedStructPos + 1) > 1) {
+            continue // no need to perform next check, both are already merged
+          }
         }
         if (replacedStructPos > 0) {
-          tryToMergeWithLeft(structs, replacedStructPos)
+          tryToMergeWithLefts(structs, replacedStructPos)
         }
       }
       if (!transaction.local && transaction.afterState.get(doc.clientID) !== transaction.beforeState.get(doc.clientID)) {

src/utils/UndoManager.js

@@ -15,6 +15,7 @@ import {
 
 import * as time from 'lib0/time'
 import * as array from 'lib0/array'
+import * as logging from 'lib0/logging'
 import { Observable } from 'lib0/observable'
 
 export class StackItem {
@@ -169,6 +170,7 @@ export class UndoManager extends Observable {
      * @type {Array<AbstractType<any>>}
      */
     this.scope = []
+    this.doc = doc
     this.addToScope(typeScope)
     this.deleteFilter = deleteFilter
     trackedOrigins.add(this)
@@ -189,7 +191,6 @@ export class UndoManager extends Observable {
      */
     this.undoing = false
     this.redoing = false
-    this.doc = doc
     this.lastChange = 0
     this.ignoreRemoteMapChanges = ignoreRemoteMapChanges
     this.captureTimeout = captureTimeout
@@ -263,6 +264,7 @@ export class UndoManager extends Observable {
     ytypes = array.isArray(ytypes) ? ytypes : [ytypes]
     ytypes.forEach(ytype => {
       if (this.scope.every(yt => yt !== ytype)) {
+        if (ytype.doc !== this.doc) logging.warn('[yjs#509] Not same Y.Doc') // use MultiDocUndoManager instead. also see https://github.com/yjs/yjs/issues/509
         this.scope.push(ytype)
       }
     })

src/utils/YEvent.js

@@ -6,6 +6,9 @@ import {
 
 import * as set from 'lib0/set'
 import * as array from 'lib0/array'
+import * as error from 'lib0/error'
+
+const errorComputeChanges = 'You must not compute changes after the event-handler fired.'
 
 /**
  * @template {AbstractType<any>} T
@@ -84,6 +87,9 @@ export class YEvent {
    */
   get keys () {
     if (this._keys === null) {
+      if (this.transaction.doc._transactionCleanups.length === 0) {
+        throw error.create(errorComputeChanges)
+      }
       const keys = new Map()
       const target = this.target
       const changed = /** @type Set<string|null> */ (this.transaction.changed.get(target))
@@ -167,6 +173,9 @@ export class YEvent {
   get changes () {
     let changes = this._changes
     if (changes === null) {
+      if (this.transaction.doc._transactionCleanups.length === 0) {
+        throw error.create(errorComputeChanges)
+      }
       const target = this.target
       const added = set.create()
       const deleted = set.create()

src/utils/encoding.js

@@ -251,7 +251,7 @@ const integrateStructs = (transaction, store, clientsStructRefs) => {
     return nextStructsTarget
   }
   let curStructsTarget = getNextStructTarget()
-  if (curStructsTarget === null && stack.length === 0) {
+  if (curStructsTarget === null) {
     return null
   }
 

tests/y-map.tests.js

@@ -8,6 +8,54 @@ import * as Y from '../src/index.js'
 import * as t from 'lib0/testing'
 import * as prng from 'lib0/prng'
 
+/**
+ * @param {t.TestCase} _tc
+ */
+export const testIterators = _tc => {
+  const ydoc = new Y.Doc()
+  /**
+   * @type {Y.Map<number>}
+   */
+  const ymap = ydoc.getMap()
+  // we are only checking if the type assumptions are correct
+  /**
+   * @type {Array<number>}
+   */
+  const vals = Array.from(ymap.values())
+  /**
+   * @type {Array<[string,number]>}
+   */
+  const entries = Array.from(ymap.entries())
+  /**
+   * @type {Array<string>}
+   */
+  const keys = Array.from(ymap.keys())
+  console.log(vals, entries, keys)
+}
+
+/**
+ * Computing event changes after transaction should result in an error. See yjs#539
+ *
+ * @param {t.TestCase} _tc
+ */
+export const testMapEventError = _tc => {
+  const doc = new Y.Doc()
+  const ymap = doc.getMap()
+  /**
+   * @type {any}
+   */
+  let event = null
+  ymap.observe((e) => {
+    event = e
+  })
+  t.fails(() => {
+    t.info(event.keys)
+  })
+  t.fails(() => {
+    t.info(event.keys)
+  })
+}
+
 /**
  * @param {t.TestCase} tc
  */