13.1.0

Implement unshift function in Y-Array

.github/FUNDING.yml

@@ -1,12 +0,0 @@
-# These are supported funding model platforms
-
-github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
-patreon: dmonad
-open_collective: # Replace with a single Open Collective username
-ko_fi: # Replace with a single Ko-fi username
-tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
-community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
-liberapay: # Replace with a single Liberapay username
-issuehunt: # Replace with a single IssueHunt username
-otechie: # Replace with a single Otechie username
-custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

README.md

@@ -25,6 +25,43 @@ build collaborative or distributed applications ping us at
 <yjs@tag1consulting.com>. Otherwise you can find help on our
 [discussion board](https://discuss.yjs.dev).
 
+## Sponsors
+
+I'm currently looking for sponsors that allow me to be less dependent on
+contracting work. These awesome backers already fund further development of
+Yjs:
+
+[![Vincent Waller](https://github.com/vwall.png?size=60)](https://github.com/vwall)
+[<img src="https://user-images.githubusercontent.com/5553757/83337333-a7bcb380-a2ba-11ea-837b-e404eb35d318.png"
+height="60px" />](https://input.com/)
+[![Duane Johnson](https://github.com/canadaduane.png?size=60)](https://github.com/canadaduane)
+[![Joe Reeve](https://github.com/ISNIT0.png?size=60)](https://github.com/ISNIT0)
+[<img src="https://room.sh/img/icons/android-chrome-192x192.png" height="60px" />](https://room.sh/)
+[![JourneyApps](https://github.com/journeyapps.png?size=60)](https://github.com/journeyapps)
+[![Adam Brunnmeier](https://github.com/adabru.png?size=60)](https://github.com/adabru)
+[![Nathanael Anderson](https://github.com/NathanaelA.png?size=60)](https://github.com/NathanaelA)
+
+Sponsorship also comes with special perks! [![Become a Sponsor](https://img.shields.io/static/v1?label=Become%20a%20Sponsor&message=%E2%9D%A4&logo=GitHub&style=flat&color=d42f2d)](https://github.com/sponsors/dmonad)
+
+## Who is using Yjs
+
+* [Relm](http://www.relm.us/) A collaborative gameworld for teamwork and
+  community. :star2:
+* [Input](https://input.com/) A collaborative note taking app. :star2:
+* [Room.sh](https://room.sh/) A meeting application with integrated
+  collaborative drawing, editing, and coding tools. :star:
+* [http://coronavirustechhandbook.com/](https://coronavirustechhandbook.com/)
+  A collaborative wiki that is edited by thousands of different people to work
+  on a rapid and sophisticated response to the coronavirus outbreak and
+  subsequent impacts. :star:
+* [Nimbus Note](https://nimbusweb.me/note.php) A note-taking app designed by
+  Nimbus Web.
+* [JoeDocs](https://joedocs.com/) An open collaborative wiki.
+* [Pluxbox RadioManager](https://pluxbox.com/) A web-based app to
+  collaboratively organize radio broadcasts.
+* [Cattaz](http://cattaz.io/) A wiki that can run custom applications in the
+  wiki pages.
+
 ## Table of Contents
 
 * [Overview](#Overview)
@@ -137,6 +174,54 @@ Now you understand how types are defined on a shared document. Next you can jump
 to the [demo repository](https://github.com/yjs/yjs-demos) or continue reading
 the API docs.
 
+### Example: Using and combining providers
+
+Any of the Yjs providers can be combined with each other. So you can sync data
+over different network technologies.
+
+In most cases you want to use a network provider (like y-websocket or y-webrtc)
+in combination with a persistence provider (y-indexeddb in the browser).
+Persistence allows you to load the document faster and to persist data that is
+created while offline.
+
+For the sake of this demo we combine two different network providers with a
+persistence provider.
+
+```js
+import * as Y from 'yjs'
+import { WebrtcProvider } from 'y-webrtc'
+import { WebsocketProvider } from 'y-websocket'
+import { IndexeddbPersistence } from 'y-indexeddb'
+
+const ydoc = new Y.Doc()
+
+// this allows you to instantly get the (cached) documents data
+const indexeddbProvider = new IndexeddbPersistence('count-demo', ydoc)
+idbP.whenSynced.then(() => {
+  console.log('loaded data from indexed db')
+})
+
+// Sync clients with the y-webrtc provider.
+const webrtcProvider = new WebrtcProvider('count-demo', ydoc)
+
+// Sync clients with the y-websocket provider
+const websocketProvider = new WebsocketProvider(
+  'wss://demos.yjs.dev', 'count-demo', ydoc
+)
+
+// array of numbers which produce a sum
+const yarray = ydoc.getArray('count')
+
+// observe changes of the sum
+yarray.observe(event => {
+  // print updates when the data changes
+  console.log('new sum: ' + yarray.toArray().reduce((a,b) => a + b))
+})
+
+// add 1 to the sum
+yarray.push([1]) // => "new sum: 1"
+```
+
 ## API
 
 ```js
@@ -158,11 +243,13 @@ necessary.
     <b><code>insert(index:number, content:Array&lt;object|boolean|Array|string|number|Uint8Array|Y.Type&gt;)</code></b>
     <dd>
 Insert content at <var>index</var>. Note that content is an array of elements.
-I.e. <code>array.insert(0, [1]</code> splices the list and inserts 1 at
+I.e. <code>array.insert(0, [1])</code> splices the list and inserts 1 at
 position 0.
     </dd>
     <b><code>push(Array&lt;Object|boolean|Array|string|number|Uint8Array|Y.Type&gt;)</code></b>
     <dd></dd>
+    <b><code>unshift(Array&lt;Object|boolean|Array|string|number|Uint8Array|Y.Type&gt;)</code></b>
+    <dd></dd>
     <b><code>delete(index:number, length:number)</code></b>
     <dd></dd>
     <b><code>get(index:number)</code></b>

package.json

@@ -1,11 +1,15 @@
 {
   "name": "yjs",
-  "version": "13.0.8",
+  "version": "13.1.0",
   "description": "Shared Editing Library",
   "main": "./dist/yjs.cjs",
   "module": "./dist/yjs.mjs",
   "types": "./dist/src/index.d.ts",
   "sideEffects": false,
+  "funding": {
+    "type": "GitHub Sponsors ❤",
+    "url": "https://github.com/sponsors/dmonad"
+  },
   "scripts": {
     "test": "npm run dist && node ./dist/tests.cjs --repitition-time 50",
     "test-extensive": "npm run lint && npm run dist && node ./dist/tests.cjs --production --repitition-time 10000",
@@ -56,20 +60,20 @@
   },
   "homepage": "https://yjs.dev",
   "dependencies": {
-    "lib0": "^0.2.26"
+    "lib0": "^0.2.27"
   },
   "devDependencies": {
-    "@rollup/plugin-commonjs": "^11.0.1",
+    "@rollup/plugin-commonjs": "^11.1.0",
-    "@rollup/plugin-node-resolve": "^7.0.0",
+    "@rollup/plugin-node-resolve": "^7.1.3",
     "concurrently": "^3.6.1",
-    "http-server": "^0.12.1",
+    "http-server": "^0.12.3",
-    "jsdoc": "^3.6.3",
+    "jsdoc": "^3.6.4",
-    "markdownlint-cli": "^0.19.0",
+    "markdownlint-cli": "^0.23.1",
-    "rollup": "^1.30.0",
+    "rollup": "^1.32.1",
     "rollup-cli": "^1.0.9",
-    "standard": "^14.0.0",
+    "standard": "^14.3.4",
     "tui-jsdoc-template": "^1.2.2",
-    "typescript": "^3.7.5",
+    "typescript": "^3.9.3",
     "y-protocols": "^0.2.3"
   }
 }

src/structs/AbstractStruct.js

@@ -12,11 +12,6 @@ export class AbstractStruct {
    * @param {number} length
    */
   constructor (id, length) {
-    /**
-     * The uniqe identifier of this struct.
-     * @type {ID}
-     * @readonly
-     */
     this.id = id
     this.length = length
     this.deleted = false
@@ -55,15 +50,11 @@ export class AbstractStructRef {
    * @param {ID} id
    */
   constructor (id) {
+    this.id = id
     /**
      * @type {Array<ID>}
      */
     this._missing = []
-    /**
-     * The uniqe identifier of this type.
-     * @type {ID}
-     */
-    this.id = id
   }
 
   /**

src/structs/ContentType.js

@@ -7,7 +7,7 @@ import {
   readYXmlFragment,
   readYXmlHook,
   readYXmlText,
-  StructStore, Transaction, Item, YEvent, AbstractType // eslint-disable-line
+  ID, StructStore, Transaction, Item, YEvent, AbstractType // eslint-disable-line
 } from '../internals.js'
 
 import * as encoding from 'lib0/encoding.js' // eslint-disable-line
@@ -115,7 +115,7 @@ export class ContentType {
         // We try to merge all deleted items after each transaction,
         // but we have no knowledge about that this needs to be merged
         // since it is not in transaction.ds. Hence we add it to transaction._mergeStructs
-        transaction._mergeStructs.add(item.id)
+        transaction._mergeStructs.push(item)
       }
       item = item.right
     }
@@ -124,7 +124,7 @@ export class ContentType {
         item.delete(transaction)
       } else {
         // same as above
-        transaction._mergeStructs.add(item.id)
+        transaction._mergeStructs.push(item)
       }
     })
     transaction.changed.delete(this.type)

src/structs/GC.js

@@ -2,7 +2,6 @@
 import {
   AbstractStructRef,
   AbstractStruct,
-  createID,
   addStruct,
   StructStore, Transaction, ID // eslint-disable-line
 } from '../internals.js'
@@ -78,8 +77,7 @@ export class GCRef extends AbstractStructRef {
    */
   toStruct (transaction, store, offset) {
     if (offset > 0) {
-      // @ts-ignore
+      this.id.clock += offset
-      this.id = createID(this.id.client, this.id.clock + offset)
       this.length -= offset
     }
     return new GC(

src/structs/Item.js

@@ -1,10 +1,9 @@
 
 import {
   readID,
-  createID,
   writeID,
   GC,
-  nextID,
+  getState,
   AbstractStructRef,
   AbstractStruct,
   replaceStruct,
@@ -21,6 +20,7 @@ import {
   readContentAny,
   readContentString,
   readContentEmbed,
+  createID,
   readContentFormat,
   readContentType,
   addChangedTypeToTransaction,
@@ -88,12 +88,12 @@ export const keepItem = (item, keep) => {
  * @private
  */
 export const splitItem = (transaction, leftItem, diff) => {
-  const id = leftItem.id
   // create rightItem
+  const { client, clock } = leftItem.id
   const rightItem = new Item(
-    createID(id.client, id.clock + diff),
+    createID(client, clock + diff),
     leftItem,
-    createID(id.client, id.clock + diff - 1),
+    createID(client, clock + diff - 1),
     leftItem.right,
     leftItem.rightOrigin,
     leftItem.parent,
@@ -116,7 +116,7 @@ export const splitItem = (transaction, leftItem, diff) => {
     rightItem.right.left = rightItem
   }
   // right is more specific.
-  transaction._mergeStructs.add(rightItem.id)
+  transaction._mergeStructs.push(rightItem)
   // update parent._map
   if (rightItem.parentSub !== null && rightItem.right === null) {
     rightItem.parent._map.set(rightItem.parentSub, rightItem)
@@ -137,8 +137,12 @@ export const splitItem = (transaction, leftItem, diff) => {
  * @private
  */
 export const redoItem = (transaction, item, redoitems) => {
-  if (item.redone !== null) {
+  const doc = transaction.doc
-    return getItemCleanStart(transaction, item.redone)
+  const store = doc.store
+  const ownClientID = doc.clientID
+  const redone = item.redone
+  if (redone !== null) {
+    return getItemCleanStart(transaction, redone)
   }
   let parentItem = item.parent._item
   /**
@@ -158,7 +162,7 @@ export const redoItem = (transaction, item, redoitems) => {
     left = item
     while (left.right !== null) {
       left = left.right
-      if (left.id.client !== transaction.doc.clientID) {
+      if (left.id.client !== ownClientID) {
         // It is not possible to redo this item because it conflicts with a
         // change from another client
         return null
@@ -212,15 +216,17 @@ export const redoItem = (transaction, item, redoitems) => {
       right = right.right
     }
   }
+  const nextClock = getState(store, ownClientID)
+  const nextId = createID(ownClientID, nextClock)
   const redoneItem = new Item(
-    nextID(transaction),
+    nextId,
-    left, left === null ? null : left.lastId,
+    left, left && left.lastId,
-    right, right === null ? null : right.id,
+    right, right && right.id,
     parentItem === null ? item.parent : /** @type {ContentType} */ (parentItem.content).type,
     item.parentSub,
     item.content.copy()
   )
-  item.redone = redoneItem.id
+  item.redone = nextId
   keepItem(redoneItem, true)
   redoneItem.integrate(transaction)
   return redoneItem
@@ -294,30 +300,35 @@ export class Item extends AbstractStruct {
      * @type {AbstractContent}
      */
     this.content = content
-    this.length = content.getLength()
-    this.countable = content.isCountable()
     /**
      * If true, do not garbage collect this Item.
      */
     this.keep = false
   }
 
+  get countable () {
+    return this.content.isCountable()
+  }
+
   /**
    * @param {Transaction} transaction
    */
   integrate (transaction) {
     const store = transaction.doc.store
-    const id = this.id
     const parent = this.parent
     const parentSub = this.parentSub
     const length = this.length
+    /**
+     * @type {Item|null}
+     */
+    let left = this.left
     /**
      * @type {Item|null}
      */
     let o
     // set o to the first conflicting item
-    if (this.left !== null) {
+    if (left !== null) {
-      o = this.left.right
+      o = left.right
     } else if (parentSub !== null) {
       o = parent._map.get(parentSub) || null
       while (o !== null && o.left !== null) {
@@ -343,14 +354,14 @@ export class Item extends AbstractStruct {
       conflictingItems.add(o)
       if (compareIDs(this.origin, o.origin)) {
         // case 1
-        if (o.id.client < id.client) {
+        if (o.id.client < this.id.client) {
-          this.left = o
+          left = o
           conflictingItems.clear()
         }
       } else if (o.origin !== null && itemsBeforeOrigin.has(getItem(store, o.origin))) {
         // case 2
         if (o.origin === null || !conflictingItems.has(getItem(store, o.origin))) {
-          this.left = o
+          left = o
           conflictingItems.clear()
         }
       } else {
@@ -358,11 +369,12 @@ export class Item extends AbstractStruct {
       }
       o = o.right
     }
+    this.left = left
     // reconnect left/right + update parent map/start if necessary
-    if (this.left !== null) {
+    if (left !== null) {
-      const right = this.left.right
+      const right = left.right
       this.right = right
-      this.left.right = this
+      left.right = this
     } else {
       let r
       if (parentSub !== null) {
@@ -381,9 +393,9 @@ export class Item extends AbstractStruct {
     } else if (parentSub !== null) {
       // set as current parent value if right === null and this is parentSub
       parent._map.set(parentSub, this)
-      if (this.left !== null) {
+      if (left !== null) {
         // this is the current attribute value of parent. delete right
-        this.left.delete(transaction)
+        left.delete(transaction)
       }
     }
     // adjust length of parent
@@ -522,7 +534,8 @@ export class Item extends AbstractStruct {
     }
     if (origin === null && rightOrigin === null) {
       const parent = this.parent
-      if (parent._item === null) {
+      const parentItem = parent._item
+      if (parentItem === null) {
         // parent type on y._map
         // find the correct key
         const ykey = findRootTypeKey(parent)
@@ -530,7 +543,7 @@ export class Item extends AbstractStruct {
         encoding.writeVarString(encoder, ykey)
       } else {
         encoding.writeVarUint(encoder, 0) // write parent id
-        writeID(encoder, parent._item.id)
+        writeID(encoder, parentItem.id)
       }
       if (parentSub !== null) {
         encoding.writeVarString(encoder, parentSub)
@@ -723,22 +736,18 @@ export class ItemRef extends AbstractStructRef {
    */
   toStruct (transaction, store, offset) {
     if (offset > 0) {
-      /**
+      this.id.clock += offset
-       * @type {ID}
-       */
-      const id = this.id
-      this.id = createID(id.client, id.clock + offset)
       this.left = createID(this.id.client, this.id.clock - 1)
       this.content = this.content.splice(offset)
-      this.length -= offset
     }
 
     const left = this.left === null ? null : getItemCleanEnd(transaction, store, this.left)
     const right = this.right === null ? null : getItemCleanStart(transaction, this.right)
+    const parentId = this.parent
     let parent = null
     let parentSub = this.parentSub
-    if (this.parent !== null) {
+    if (parentId !== null) {
-      const parentItem = getItem(store, this.parent)
+      const parentItem = getItem(store, parentId)
       // Edge case: toStruct is called with an offset > 0. In this case left is defined.
       // Depending in which order structs arrive, left may be GC'd and the parent not
       // deleted. This is why we check if left is GC'd. Strictly we don't have
@@ -767,9 +776,9 @@ export class ItemRef extends AbstractStructRef {
       : new Item(
         this.id,
         left,
-        this.left,
+        left && left.lastId,
         right,
-        this.right,
+        right && right.id,
         parent,
         parentSub,
         this.content

src/types/AbstractType.js

@@ -4,14 +4,14 @@ import {
   callEventHandlerListeners,
   addEventHandlerListener,
   createEventHandler,
-  nextID,
+  getState,
   isVisible,
   ContentType,
+  createID,
   ContentAny,
   ContentBinary,
-  createID,
   getItemCleanStart,
-  Doc, Snapshot, Transaction, EventHandler, YEvent, Item, // eslint-disable-line
+  ID, Doc, Snapshot, Transaction, EventHandler, YEvent, Item, // eslint-disable-line
 } from '../internals.js'
 
 import * as map from 'lib0/map.js'
@@ -375,6 +375,9 @@ export const typeListGet = (type, index) => {
  */
 export const typeListInsertGenericsAfter = (transaction, parent, referenceItem, content) => {
   let left = referenceItem
+  const doc = transaction.doc
+  const ownClientId = doc.clientID
+  const store = doc.store
   const right = referenceItem === null ? parent._start : referenceItem.right
   /**
    * @type {Array<Object|Array<any>|number>}
@@ -382,7 +385,7 @@ export const typeListInsertGenericsAfter = (transaction, parent, referenceItem,
   let jsonContent = []
   const packJsonContent = () => {
     if (jsonContent.length > 0) {
-      left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, new ContentAny(jsonContent))
+      left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentAny(jsonContent))
       left.integrate(transaction)
       jsonContent = []
     }
@@ -401,12 +404,12 @@ export const typeListInsertGenericsAfter = (transaction, parent, referenceItem,
         switch (c.constructor) {
           case Uint8Array:
           case ArrayBuffer:
-            left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, new ContentBinary(new Uint8Array(/** @type {Uint8Array} */ (c))))
+            left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentBinary(new Uint8Array(/** @type {Uint8Array} */ (c))))
             left.integrate(transaction)
             break
           default:
             if (c instanceof AbstractType) {
-              left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, new ContentType(c))
+              left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentType(c))
               left.integrate(transaction)
             } else {
               throw new Error('Unexpected content type in insert operation')
@@ -509,6 +512,8 @@ export const typeMapDelete = (transaction, parent, key) => {
  */
 export const typeMapSet = (transaction, parent, key, value) => {
   const left = parent._map.get(key) || null
+  const doc = transaction.doc
+  const ownClientId = doc.clientID
   let content
   if (value == null) {
     content = new ContentAny([value])
@@ -532,7 +537,7 @@ export const typeMapSet = (transaction, parent, key, value) => {
         }
     }
   }
-  new Item(nextID(transaction), left, left === null ? null : left.lastId, null, null, parent, key, content).integrate(transaction)
+  new Item(createID(ownClientId, getState(doc.store, ownClientId)), left, left && left.lastId, null, null, parent, key, content).integrate(transaction)
 }
 
 /**

src/types/YArray.js

@@ -40,7 +40,7 @@ export class YArrayEvent extends YEvent {
  * A shared Array implementation.
  * @template T
  * @extends AbstractType<YArrayEvent<T>>
- * @implements {IterableIterator<T>}
+ * @implements {Iterable<T>}
  */
 export class YArray extends AbstractType {
   constructor () {
@@ -121,6 +121,15 @@ export class YArray extends AbstractType {
     this.insert(this.length, content)
   }
 
+  /**
+   * Preppends content to this YArray.
+   *
+   * @param {Array<T>} content Array of content to preppend.
+   */
+  unshift (content) {
+    this.insert(0, content)
+  }
+
   /**
    * Deletes elements starting from an index.
    *

src/types/YMap.js

@@ -42,7 +42,7 @@ export class YMapEvent extends YEvent {
  * A shared Map implementation.
  *
  * @extends AbstractType<YMapEvent<T>>
- * @implements {IterableIterator}
+ * @implements {Iterable<T>}
  */
 export class YMap extends AbstractType {
   constructor () {

src/types/YText.js

@@ -6,10 +6,10 @@
 import {
   YEvent,
   AbstractType,
-  nextID,
-  createID,
   getItemCleanStart,
+  getState,
   isVisible,
+  createID,
   YTextRefID,
   callTypeObservers,
   transact,
@@ -150,8 +150,10 @@ const insertNegatedAttributes = (transaction, parent, left, right, negatedAttrib
     left = right
     right = right.right
   }
+  const doc = transaction.doc
+  const ownClientId = doc.clientID
   for (const [key, val] of negatedAttributes) {
-    left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, new ContentFormat(key, val))
+    left = new Item(createID(ownClientId, getState(doc.store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentFormat(key, val))
     left.integrate(transaction)
   }
   return { left, right }
@@ -215,6 +217,8 @@ const minimizeAttributeChanges = (left, right, currentAttributes, attributes) =>
  * @function
  **/
 const insertAttributes = (transaction, parent, left, right, currentAttributes, attributes) => {
+  const doc = transaction.doc
+  const ownClientId = doc.clientID
   const negatedAttributes = new Map()
   // insert format-start items
   for (const key in attributes) {
@@ -223,7 +227,7 @@ const insertAttributes = (transaction, parent, left, right, currentAttributes, a
     if (!equalAttrs(currentVal, val)) {
       // save negated attribute (set null if currentVal undefined)
       negatedAttributes.set(key, currentVal)
-      left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, new ContentFormat(key, val))
+      left = new Item(createID(ownClientId, getState(doc.store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentFormat(key, val))
       left.integrate(transaction)
     }
   }
@@ -249,13 +253,15 @@ const insertText = (transaction, parent, left, right, currentAttributes, text, a
       attributes[key] = null
     }
   }
+  const doc = transaction.doc
+  const ownClientId = doc.clientID
   const minPos = minimizeAttributeChanges(left, right, currentAttributes, attributes)
   const insertPos = insertAttributes(transaction, parent, minPos.left, minPos.right, currentAttributes, attributes)
   left = insertPos.left
   right = insertPos.right
   // insert content
   const content = text.constructor === String ? new ContentString(/** @type {string} */ (text)) : new ContentEmbed(text)
-  left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, content)
+  left = new Item(createID(ownClientId, getState(doc.store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, content)
   left.integrate(transaction)
   return insertNegatedAttributes(transaction, parent, left, insertPos.right, insertPos.negatedAttributes)
 }
@@ -274,6 +280,8 @@ const insertText = (transaction, parent, left, right, currentAttributes, text, a
  * @function
  */
 const formatText = (transaction, parent, left, right, currentAttributes, length, attributes) => {
+  const doc = transaction.doc
+  const ownClientId = doc.clientID
   const minPos = minimizeAttributeChanges(left, right, currentAttributes, attributes)
   const insertPos = insertAttributes(transaction, parent, minPos.left, minPos.right, currentAttributes, attributes)
   const negatedAttributes = insertPos.negatedAttributes
@@ -318,7 +326,7 @@ const formatText = (transaction, parent, left, right, currentAttributes, length,
     for (; length > 0; length--) {
       newlines += '\n'
     }
-    left = new Item(nextID(transaction), left, left === null ? null : left.lastId, right, right === null ? null : right.id, parent, null, new ContentString(newlines))
+    left = new Item(createID(ownClientId, getState(doc.store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentString(newlines))
     left.integrate(transaction)
   }
   return insertNegatedAttributes(transaction, parent, left, right, negatedAttributes)

src/types/YXmlFragment.js

@@ -48,7 +48,7 @@ import * as decoding from 'lib0/decoding.js' // eslint-disable-line
  * Can be created with {@link YXmlFragment#createTreeWalker}
  *
  * @public
- * @implements {IterableIterator}
+ * @implements {Iterable<YXmlElement|YXmlText|YXmlElement|YXmlHook>}
  */
 export class YXmlTreeWalker {
   /**
@@ -81,10 +81,10 @@ export class YXmlTreeWalker {
      * @type {Item|null}
      */
     let n = this._currentNode
-    let type = /** @type {ContentType} */ (n.content).type
+    let type = /** @type {any} */ (n.content).type
     if (n !== null && (!this._firstCall || n.deleted || !this._filter(type))) { // if first call, we check if we can use the first item
       do {
-        type = /** @type {ContentType} */ (n.content).type
+        type = /** @type {any} */ (n.content).type
         if (!n.deleted && (type.constructor === YXmlElement || type.constructor === YXmlFragment) && type._start !== null) {
           // walk down in the tree
           n = type._start

src/utils/DeleteSet.js

@@ -1,11 +1,11 @@
 
 import {
   findIndexSS,
-  createID,
   getState,
   splitItem,
+  createID,
   iterateStructs,
-  Item, GC, StructStore, Transaction, ID // eslint-disable-line
+  Item, AbstractStruct, GC, StructStore, Transaction, ID // eslint-disable-line
 } from '../internals.js'
 
 import * as array from 'lib0/array.js'

src/utils/RelativePosition.js

@@ -1,12 +1,12 @@
 
 import {
-  createID,
   writeID,
   readID,
   compareIDs,
   getState,
   findRootTypeKey,
   Item,
+  createID,
   ContentType,
   followRedone,
   ID, Doc, AbstractType // eslint-disable-line
@@ -107,7 +107,7 @@ export const createRelativePosition = (type, item) => {
   if (type._item === null) {
     tname = findRootTypeKey(type)
   } else {
-    typeid = type._item.id
+    typeid = createID(type._item.id.client, type._item.id.clock)
   }
   return new RelativePosition(typeid, tname, item)
 }

src/utils/Snapshot.js

@@ -4,13 +4,13 @@ import {
   createDeleteSetFromStructStore,
   getStateVector,
   getItemCleanStart,
-  createID,
   iterateDeletedStructs,
   writeDeleteSet,
   writeStateVector,
   readDeleteSet,
   readStateVector,
   createDeleteSet,
+  createID,
   getState,
   Transaction, Doc, DeleteSet, Item // eslint-disable-line
 } from '../internals.js'

src/utils/StructStore.js

@@ -2,7 +2,7 @@
 import {
   GC,
   splitItem,
-  GCRef, ItemRef, Transaction, ID, Item // eslint-disable-line
+  AbstractStruct, GCRef, ItemRef, Transaction, ID, Item // eslint-disable-line
 } from '../internals.js'
 
 import * as math from 'lib0/math.js'
@@ -114,7 +114,7 @@ export const addStruct = (store, struct) => {
 
 /**
  * Perform a binary search on a sorted array
- * @param {Array<any>} structs
+ * @param {Array<Item|GC>} structs
  * @param {number} clock
  * @return {number}
  *
@@ -163,16 +163,10 @@ export const find = (store, id) => {
 
 /**
  * Expects that id is actually in store. This function throws or is an infinite loop otherwise.
- *
- * @param {StructStore} store
- * @param {ID} id
- * @return {Item}
- *
  * @private
  * @function
  */
-// @ts-ignore
+export const getItem = /** @type {function(StructStore,ID):Item} */ (find)
-export const getItem = (store, id) => find(store, id)
 
 /**
  * @param {Transaction} transaction

src/utils/Transaction.js

@@ -1,7 +1,6 @@
 
 import {
   getState,
-  createID,
   writeStructsFromTransaction,
   writeDeleteSet,
   DeleteSet,
@@ -11,7 +10,8 @@ import {
   callEventHandlerListeners,
   Item,
   generateNewClientId,
-  StructStore, ID, AbstractType, AbstractStruct, YEvent, Doc // eslint-disable-line
+  createID,
+  GC, StructStore, ID, AbstractType, AbstractStruct, YEvent, Doc // eslint-disable-line
 } from '../internals.js'
 
 import * as encoding from 'lib0/encoding.js'
@@ -86,9 +86,9 @@ export class Transaction {
      */
     this.changedParentTypes = new Map()
     /**
-     * @type {Set<ID>}
+     * @type {Array<AbstractStruct>}
      */
-    this._mergeStructs = new Set()
+    this._mergeStructs = []
     /**
      * @type {any}
      */
@@ -170,7 +170,7 @@ const tryToMergeWithLeft = (structs, pos) => {
  */
 const tryGcDeleteSet = (ds, store, gcFilter) => {
   for (const [client, deleteItems] of ds.clients) {
-    const structs = /** @type {Array<AbstractStruct>} */ (store.clients.get(client))
+    const structs = /** @type {Array<GC|Item>} */ (store.clients.get(client))
     for (let di = deleteItems.length - 1; di >= 0; di--) {
       const deleteItem = deleteItems[di]
       const endDeleteItemClock = deleteItem.clock + deleteItem.len
@@ -199,7 +199,7 @@ const tryMergeDeleteSet = (ds, store) => {
   // try to merge deleted / gc'd items
   // merge from right to left for better efficiecy and so we don't miss any merge targets
   for (const [client, deleteItems] of ds.clients) {
-    const structs = /** @type {Array<AbstractStruct>} */ (store.clients.get(client))
+    const structs = /** @type {Array<GC|Item>} */ (store.clients.get(client))
     for (let di = deleteItems.length - 1; di >= 0; di--) {
       const deleteItem = deleteItems[di]
       // start with merging the item next to the last deleted item
@@ -235,6 +235,7 @@ const cleanupTransactions = (transactionCleanups, i) => {
     const doc = transaction.doc
     const store = doc.store
     const ds = transaction.deleteSet
+    const mergeStructs = transaction._mergeStructs
     try {
       sortAndMergeDeleteSet(ds)
       transaction.afterState = getStateVector(transaction.doc.store)
@@ -292,7 +293,7 @@ const cleanupTransactions = (transactionCleanups, i) => {
       for (const [client, clock] of transaction.afterState) {
         const beforeClock = transaction.beforeState.get(client) || 0
         if (beforeClock !== clock) {
-          const structs = /** @type {Array<AbstractStruct>} */ (store.clients.get(client))
+          const structs = /** @type {Array<GC|Item>} */ (store.clients.get(client))
           // we iterate from right to left so we can safely remove entries
           const firstChangePos = math.max(findIndexSS(structs, beforeClock), 1)
           for (let i = structs.length - 1; i >= firstChangePos; i--) {
@@ -303,10 +304,9 @@ const cleanupTransactions = (transactionCleanups, i) => {
       // try to merge mergeStructs
       // @todo: it makes more sense to transform mergeStructs to a DS, sort it, and merge from right to left
       //        but at the moment DS does not handle duplicates
-      for (const mid of transaction._mergeStructs) {
+      for (let i = 0; i < mergeStructs.length; i++) {
-        const client = mid.client
+        const { client, clock } = mergeStructs[i].id
-        const clock = mid.clock
+        const structs = /** @type {Array<GC|Item>} */ (store.clients.get(client))
-        const structs = /** @type {Array<AbstractStruct>} */ (store.clients.get(client))
         const replacedStructPos = findIndexSS(structs, clock)
         if (replacedStructPos + 1 < structs.length) {
           tryToMergeWithLeft(structs, replacedStructPos + 1)

src/utils/UndoManager.js

@@ -3,14 +3,14 @@ import {
   iterateDeletedStructs,
   keepItem,
   transact,
+  createID,
   redoItem,
   iterateStructs,
   isParentOf,
-  createID,
   followRedone,
   getItemCleanStart,
   getState,
-  Transaction, Doc, Item, GC, DeleteSet, AbstractType // eslint-disable-line
+  ID, Transaction, Doc, Item, GC, DeleteSet, AbstractType // eslint-disable-line
 } from '../internals.js'
 
 import * as time from 'lib0/time.js'

src/utils/encoding.js

@@ -19,15 +19,15 @@ import {
   GCRef,
   ItemRef,
   writeID,
-  createID,
   readID,
   getState,
+  createID,
   getStateVector,
   readAndApplyDeleteSet,
   writeDeleteSet,
   createDeleteSetFromStructStore,
   transact,
-  Doc, Transaction, AbstractStruct, StructStore, ID // eslint-disable-line
+  Doc, Transaction, GC, Item, StructStore, ID // eslint-disable-line
 } from '../internals.js'
 
 import * as encoding from 'lib0/encoding.js'
@@ -36,7 +36,7 @@ import * as binary from 'lib0/binary.js'
 
 /**
  * @param {encoding.Encoder} encoder
- * @param {Array<AbstractStruct>} structs All structs by `client`
+ * @param {Array<GC|Item>} structs All structs by `client`
  * @param {number} client
  * @param {number} clock write structs starting with `ID(client,clock)`
  *
@@ -50,35 +50,12 @@ const writeStructs = (encoder, structs, client, clock) => {
   writeID(encoder, createID(client, clock))
   const firstStruct = structs[startNewStructs]
   // write first struct with an offset
-  firstStruct.write(encoder, clock - firstStruct.id.clock, 0)
+  firstStruct.write(encoder, clock - firstStruct.id.clock)
   for (let i = startNewStructs + 1; i < structs.length; i++) {
-    structs[i].write(encoder, 0, 0)
+    structs[i].write(encoder, 0)
   }
 }
 
-/**
- * @param {decoding.Decoder} decoder
- * @param {number} numOfStructs
- * @param {ID} nextID
- * @return {Array<GCRef|ItemRef>}
- *
- * @private
- * @function
- */
-const readStructRefs = (decoder, numOfStructs, nextID) => {
-  /**
-   * @type {Array<GCRef|ItemRef>}
-   */
-  const refs = []
-  for (let i = 0; i < numOfStructs; i++) {
-    const info = decoding.readUint8(decoder)
-    const ref = (binary.BITS5 & info) === 0 ? new GCRef(decoder, nextID, info) : new ItemRef(decoder, nextID, info)
-    nextID = createID(nextID.client, nextID.clock + ref.length)
-    refs.push(ref)
-  }
-  return refs
-}
-
 /**
  * @param {encoding.Encoder} encoder
  * @param {StructStore} store
@@ -111,22 +88,30 @@ export const writeClientsStructs = (encoder, store, _sm) => {
 
 /**
  * @param {decoding.Decoder} decoder The decoder object to read data from.
+ * @param {Map<number,Array<GCRef|ItemRef>>} clientRefs
  * @return {Map<number,Array<GCRef|ItemRef>>}
  *
  * @private
  * @function
  */
-export const readClientsStructRefs = decoder => {
+export const readClientsStructRefs = (decoder, clientRefs) => {
-  /**
-   * @type {Map<number,Array<GCRef|ItemRef>>}
-   */
-  const clientRefs = new Map()
   const numOfStateUpdates = decoding.readVarUint(decoder)
   for (let i = 0; i < numOfStateUpdates; i++) {
     const numberOfStructs = decoding.readVarUint(decoder)
     const nextID = readID(decoder)
-    const refs = readStructRefs(decoder, numberOfStructs, nextID)
+    const nextIdClient = nextID.client
-    clientRefs.set(nextID.client, refs)
+    let nextIdClock = nextID.clock
+    /**
+     * @type {Array<GCRef|ItemRef>}
+     */
+    const refs = []
+    clientRefs.set(nextIdClient, refs)
+    for (let i = 0; i < numberOfStructs; i++) {
+      const info = decoding.readUint8(decoder)
+      const ref = (binary.BITS5 & info) === 0 ? new GCRef(decoder, createID(nextIdClient, nextIdClock), info) : new ItemRef(decoder, createID(nextIdClient, nextIdClock), info)
+      refs.push(ref)
+      nextIdClock += ref.length
+    }
   }
   return clientRefs
 }
@@ -171,16 +156,18 @@ const resumeStructIntegration = (transaction, store) => {
     }
     const ref = stack[stack.length - 1]
     const m = ref._missing
-    const client = ref.id.client
+    const refID = ref.id
+    const client = refID.client
+    const refClock = refID.clock
     const localClock = getState(store, client)
-    const offset = ref.id.clock < localClock ? localClock - ref.id.clock : 0
+    const offset = refClock < localClock ? localClock - refClock : 0
-    if (ref.id.clock + offset !== localClock) {
+    if (refClock + offset !== localClock) {
       // A previous message from this client is missing
       // check if there is a pending structRef with a smaller clock and switch them
       const structRefs = clientsStructRefs.get(client)
       if (structRefs !== undefined) {
         const r = structRefs.refs[structRefs.i]
-        if (r.id.clock < ref.id.clock) {
+        if (r.id.clock < refClock) {
           // put ref with smaller clock on stack instead and continue
           structRefs.refs[structRefs.i] = ref
           stack[stack.length - 1] = r
@@ -282,7 +269,8 @@ const mergeReadStructsIntoPendingReads = (store, clientsStructsRefs) => {
  * @function
  */
 export const readStructs = (decoder, transaction, store) => {
-  const clientsStructRefs = readClientsStructRefs(decoder)
+  const clientsStructRefs = new Map()
+  readClientsStructRefs(decoder, clientsStructRefs)
   mergeReadStructsIntoPendingReads(store, clientsStructRefs)
   resumeStructIntegration(transaction, store)
   tryResumePendingDeleteReaders(transaction, store)

tests/testHelper.js

@@ -330,6 +330,7 @@ export const compareStructStores = (ss1, ss2) => {
         s1.constructor !== s2.constructor ||
         !Y.compareIDs(s1.id, s2.id) ||
         s1.deleted !== s2.deleted ||
+        // @ts-ignore
         s1.length !== s2.length
       ) {
         t.fail('Structs dont match')

tests/y-text.tests.js

@@ -205,6 +205,23 @@ export const testFormattingRemovedInMidText = tc => {
   t.assert(Y.getTypeChildren(text0).length === 3)
 }
 
+/**
+ * @param {t.TestCase} tc
+ *
+export const testLargeFragmentedDocument = tc => {
+  const { text0, text1, testConnector } = init(tc, { users: 2 })
+  // @ts-ignore
+  text0.doc.transact(() => {
+    for (let i = 0; i < 1000000; i++) {
+      text0.insert(0, '0')
+    }
+  })
+  t.measureTime('time to apply', () => {
+    testConnector.flushAllMessages()
+  })
+}
+*/
+
 // RANDOM TESTS
 
 let charCounter = 0