13.5.46

Add test for Y.Array.from

.circleci/config.yml

@@ -1,7 +0,0 @@
-version: 2.1
-orbs:
-  node: circleci/node@3.0.0
-workflows:
-  node-tests:
-    jobs:
-      - node/test

INTERNALS.md

@@ -66,7 +66,7 @@ fields, respectively. These are used when peers concurrently insert at the same
 location in a document. Though quite rare in practice, Yjs needs to make sure
 the list items always resolve to the same order on all peers. The actual logic
 is relatively simple - its only a couple dozen lines of code and it lives in
-the `Item#integrate()` method. The YATA paper has much more detail on the this
+the `Item#integrate()` method. The YATA paper has much more detail on this
 algorithm.
 
 ### Item Storage
@@ -152,8 +152,8 @@ concepts that can be used to create a custom network protocol:
 an incremental document updates that allows clients to sync with each other.
 The update object is an Uint8Array that efficiently encodes `Item` objects and
 the delete set.
-* `state vector`: A state vector defines the know state of each user (a set of
-  tubles `(client, clock)`). This object is also efficiently encoded as a
+* `state vector`: A state vector defines the known state of each user (a set of
+  tuples `(client, clock)`). This object is also efficiently encoded as a
 Uint8Array.
 
 The client can ask a remote client for missing document updates by sending
@@ -168,7 +168,7 @@ An implementation of the syncing process is in
 ## Snapshots
 
 A snapshot can be used to restore an old document state. It is a `state vector`
-\+ `delete set`. I client can restore an old document state by iterating through
+\+ `delete set`. A client can restore an old document state by iterating through
 the sequence CRDT and ignoring all Items that have an `id.clock >
 stateVector[id.client].clock`. Instead of using `item.deleted` the client will
 use the delete set to find out if an item was deleted or not.

README.md

@@ -15,58 +15,51 @@ suited for even large documents.
 
 * Demos: [https://github.com/yjs/yjs-demos](https://github.com/yjs/yjs-demos)
 * Discuss: [https://discuss.yjs.dev](https://discuss.yjs.dev)
+* Chat: [Gitter](https://gitter.im/Yjs/community) | [Discord](https://discord.gg/T3nqMT6qbM)
 * Benchmark Yjs vs. Automerge:
   [https://github.com/dmonad/crdt-benchmarks](https://github.com/dmonad/crdt-benchmarks)
 * Podcast [**"Yjs Deep Dive into real time collaborative editing solutions":**](https://www.tag1consulting.com/blog/deep-dive-real-time-collaborative-editing-solutions-tagteamtalk-001-0)
 * Podcast [**"Google Docs-style editing in Gutenberg with the YJS framework":**](https://publishpress.com/blog/yjs/)
 
-:construction_worker_woman: If you are looking for professional (paid) support to
-build collaborative or distributed applications ping us at
-<yjs@tag1consulting.com>. Otherwise you can find help on our
-[discussion board](https://discuss.yjs.dev).
+:construction_worker_woman: If you are looking for professional support, please
+consider supporting this project via a "support contract" on
+[GitHub Sponsors](https://github.com/sponsors/dmonad). I will attend your issues
+quicker and we can discuss questions and problems in regular video conferences.
+Otherwise you can find help on our community [discussion board](https://discuss.yjs.dev).
 
-## Sponsors
+## Sponsorship
 
-I'm currently looking for sponsors that allow me to be less dependent on
-contracting work. These awesome backers already fund further development of
-Yjs:
-
-[![davidhq](https://github.com/davidhq.png?size=60)](https://github.com/davidhq)
-[![Ifiok Jr.](https://github.com/ifiokjr.png?size=60)](https://github.com/ifiokjr)
-[![Burke Libbey](https://github.com/burke.png?size=60)](https://github.com/burke)
-[![Beni Cherniavsky-Paskin](https://github.com/cben.png?size=60)](https://github.com/cben)
-[![Tom Moor](https://github.com/tommoor.png?size=60)](https://github.com/tommoor)
-[![Michael Meyers](https://github.com/michaelemeyers.png?size=60)](https://github.com/michaelemeyers)
-[![Cristiano Benjamin](https://github.com/csbenjamin.png?size=60)](https://github.com/csbenjamin)
-[![Braden](https://github.com/AdventureBeard.png?size=60)](https://github.com/AdventureBeard)
-[![nimbuswebinc](https://nimbusweb.me/new-style-img/note-icon.svg)](https://github.com/nimbuswebinc)
-[![JourneyApps](https://github.com/journeyapps.png?size=60)](https://github.com/journeyapps)
-[![Adam Brunnmeier](https://github.com/adabru.png?size=60)](https://github.com/adabru)
-[![Nathanael Anderson](https://github.com/NathanaelA.png?size=60)](https://github.com/NathanaelA)
-[<img src="https://room.sh/img/icons/android-chrome-192x192.png" height="60px" />](https://room.sh/)
-
-Sponsorship also comes with special perks! [![Become a Sponsor](https://img.shields.io/static/v1?label=Become%20a%20Sponsor&message=%E2%9D%A4&logo=GitHub&style=flat&color=d42f2d)](https://github.com/sponsors/dmonad)
+Please contribute to the project financially - especially if your company relies
+on Yjs. [![Become a Sponsor](https://img.shields.io/static/v1?label=Become%20a%20Sponsor&message=%E2%9D%A4&logo=GitHub&style=flat&color=d42f2d)](https://github.com/sponsors/dmonad)
 
 ## Who is using Yjs
 
-* [Serenity Notes](https://www.serenity.re/en/notes) End-to-end encrypted
-  collaborative notes app.
+* [AFFiNE](https://affine.pro/) A local-first, privacy-first, open source
+  knowledge base. 🏅
+* [Dynaboard](https://dynaboard.com/) Build web apps collaboratively. :star2:
+* [Sana](https://sanalabs.com/) A learning platform with collaborative text
+  editing powered by Yjs.
 * [Relm](https://www.relm.us/) A collaborative gameworld for teamwork and
-  community. :star2:
-* [Input](https://input.com/) A collaborative note taking app. :star2:
+  community. :star:
 * [Room.sh](https://room.sh/) A meeting application with integrated
   collaborative drawing, editing, and coding tools. :star:
-* [https://coronavirustechhandbook.com/](https://coronavirustechhandbook.com/)
-  A collaborative wiki that is edited by thousands of different people to work
-  on a rapid and sophisticated response to the coronavirus outbreak and
-  subsequent impacts. :star:
 * [Nimbus Note](https://nimbusweb.me/note.php) A note-taking app designed by
-  Nimbus Web.
-* [JoeDocs](https://joedocs.com/) An open collaborative wiki.
-* [Pluxbox RadioManager](https://pluxbox.com/) A web-based app to
-  collaboratively organize radio broadcasts.
+  Nimbus Web. :star:
+* [Pluxbox RadioManager](https://getradiomanager.com/) A web-based app to
+  collaboratively organize radio broadcasts. :star:
+* [Serenity Notes](https://www.serenity.re/en/notes) End-to-end encrypted
+  collaborative notes app.
+* [PRSM](https://prsm.uk/) Collaborative mind-mapping and system visualisation. *[(source)](https://github.com/micrology/prsm)*
 * [Alldone](https://alldone.app/) A next-gen project management and
   collaboration platform.
+* [Living Spec](https://livingspec.com/) A modern way for product teams to collaborate.
+* [Slidebeamer](https://slidebeamer.com/) Presentation app.
+* [BlockSurvey](https://blocksurvey.io) End-to-end encryption for your forms/surveys.
+* [Skiff](https://skiff.org/) Private, decentralized workspace.
+* [Hyperquery](https://hyperquery.ai/) A collaborative data workspace for
+  sharing analyses, documentation, spreadsheets, and dashboards.
+* [Nosgestesclimat](https://nosgestesclimat.fr/groupe) The french carbon
+  footprint calculator has a group P2P mode based on yjs
 
 ## Table of Contents
 
@@ -99,6 +92,8 @@ are implemented in separate modules.
 | [Monaco](https://microsoft.github.io/monaco-editor/) | ✔ | [y-monaco](https://github.com/yjs/y-monaco) | [demo](https://demos.yjs.dev/monaco/monaco.html) |
 | [Slate](https://github.com/ianstormtaylor/slate) | ✔ | [slate-yjs](https://github.com/bitphinix/slate-yjs) | [demo](https://bitphinix.github.io/slate-yjs-example) |
 | [valtio](https://github.com/pmndrs/valtio) |  | [valtio-yjs](https://github.com/dai-shi/valtio-yjs) | [demo](https://codesandbox.io/s/valtio-yjs-demo-ox3iy) |
+| [immer](https://github.com/immerjs/immer) |  | [immer-yjs](https://github.com/sep2/immer-yjs) | [demo](https://codesandbox.io/s/immer-yjs-demo-6e0znb) |
+| React / Vue / Svelte / MobX | | [SyncedStore](https://syncedstore.org) | [demo](https://syncedstore.org/docs/react) |
 
 ### Providers
 
@@ -127,6 +122,12 @@ leveldb database.
 Efficiently persists document updates to the browsers indexeddb database.
 The document is immediately available and only diffs need to be synced through the
 network provider.
+  </dd>
+  <dt><a href="https://github.com/marcopolo/y-libp2p">y-libp2p</a></dt>
+  <dd>
+Uses <a href="https://libp2p.io/">libp2p</a> to propagate updates via
+<a href="https://github.com/libp2p/specs/tree/master/pubsub/gossipsub">GossipSub</a>.
+Also includes a peer-sync mechanism to catch up on missed updates.
   </dd>
   <dt><a href="https://github.com/yjs/y-dat">y-dat</a></dt>
   <dd>
@@ -134,6 +135,20 @@ network provider.
 <a href="https://github.com/kappa-db/multifeed">multifeed</a>. Each client has
 an append-only log of CRDT local updates (hypercore). Multifeed manages and sync
 hypercores and y-dat listens to changes and applies them to the Yjs document.
+</dd>
+  <dt><a href="https://github.com/yousefED/matrix-crdt">Matrix-CRDT</a></dt>
+  <dd>
+Use <a href="https://www.matrix.org">Matrix</a> as an off-the-shelf backend for
+Yjs by using the <a href="https://github.com/yousefED/matrix-crdt">MatrixProvider</a>.
+Use Matrix as transport and storage of Yjs updates, so you can focus building
+your client app and Matrix can provide powerful features like Authentication,
+Authorization, Federation, hosting (self-hosting or SaaS) and even End-to-End
+Encryption (E2EE).
+</dd>
+  <dt><a href="https://github.com/MaxNoetzold/y-mongodb-provider">y-mongodb-provider</a></dt>
+  <dd>
+Adds persistent storage to a server with MongoDB. Can be used with the
+y-websocket provider.
 </dd>
 </dl>
 
@@ -482,6 +497,8 @@ or any of its children.
     <dd>Clone this type into a fresh Yjs type.</dd>
     <b><code>toArray():Array&lt;Y.XmlElement|Y.XmlText&gt;</code></b>
     <dd>Copies the children to a new Array.</dd>
+    <b><code>toDOM():DocumentFragment</code></b>
+    <dd>Transforms this type and all children to new DOM elements.</dd>
     <b><code>toString():string</code></b>
     <dd>Get the XML serialization of all descendants.</dd>
     <b><code>toJSON():string</code></b>
@@ -555,6 +572,8 @@ content and be actually XML compliant.
     <dd>Clone this type into a fresh Yjs type.</dd>
     <b><code>toArray():Array&lt;Y.XmlElement|Y.XmlText&gt;</code></b>
     <dd>Copies the children to a new Array.</dd>
+    <b><code>toDOM():Element</code></b>
+    <dd>Transforms this type and all children to a new DOM element.</dd>
     <b><code>toString():string</code></b>
     <dd>Get the XML serialization of all descendants.</dd>
     <b><code>toJSON():string</code></b>
@@ -623,6 +642,8 @@ type. Doesn't log types that have not been defined (using
   <dd>Define a shared Y.Array type. Is equivalent to <code>y.get(string, Y.Array)</code>.</dd>
   <b><code>getMap(string):Y.Map</code></b>
   <dd>Define a shared Y.Map type. Is equivalent to <code>y.get(string, Y.Map)</code>.</dd>
+  <b><code>getText(string):Y.Text</code></b>
+  <dd>Define a shared Y.Text type. Is equivalent to <code>y.get(string, Y.Text)</code>.</dd>
   <b><code>getXmlFragment(string):Y.XmlFragment</code></b>
   <dd>Define a shared Y.XmlFragment type. Is equivalent to <code>y.get(string, Y.XmlFragment)</code>.</dd>
   <b><code>on(string, function)</code></b>
@@ -708,7 +729,7 @@ Y.applyUpdate(ydoc1, diff2)
 Y.applyUpdate(ydoc2, diff1)
 ```
 
-### Example: Syncing clients without loading the Y.Doc
+#### Example: Syncing clients without loading the Y.Doc
 
 It is possible to sync clients and compute delta updates without loading the Yjs
 document to memory. Yjs exposes an API to compute the differences directly on the
@@ -732,6 +753,17 @@ currentState1 = Y.mergeUpdates([currentState1, diff2])
 currentState1 = Y.mergeUpdates([currentState1, diff1])
 ```
 
+#### Using V2 update format
+
+Yjs implements two update formats. By default you are using the V1 update format.
+You can opt-in into the V2 update format wich provides much better compression.
+It is not yet used by all providers. However, you can already use it if
+you are building your own provider. All below functions are available with the
+suffix "V2". E.g. `Y.applyUpdate` ⇒ `Y.applyUpdateV2`. We also support conversion
+functions between both formats: `Y.convertUpdateFormatV1ToV2` & `Y.convertUpdateFormatV2ToV1`.
+
+#### Update API
+
 <dl>
   <b><code>Y.applyUpdate(Y.Doc, update:Uint8Array, [transactionOrigin:any])</code></b>
   <dd>
@@ -764,10 +796,33 @@ Encode the missing differences to another update message. This function works
 similarly to <code>Y.encodeStateAsUpdate(ydoc, stateVector)</code> but works
 on updates instead.
   </dd>
+  <b><code>convertUpdateFormatV1ToV2</code></b>
+  <dd>
+Convert V1 update format to the V2 update format.
+  </dd>
+  <b><code>convertUpdateFormatV2ToV1</code></b>
+  <dd>
+Convert V2 update format to the V1 update format.
+  </dd>
 </dl>
 
 ### Relative Positions
 
+When working with collaborative documents, we often need to work with positions.
+Positions may represent cursor locations, selection ranges, or even assign a
+comment to a range of text. Normal index-positions (expressed as integers) are
+not convenient to use because the index-range is invalidated as soon as a remote
+change manipulates the document. Relative positions give you a powerful API to
+express positions.
+
+A relative position is fixated to an element in the shared document and is not
+affected by remote changes. I.e. given the document `"a|c"`, the relative
+position is attached to `c`. When a remote user modifies the document by
+inserting a character before the cursor, the cursor will stay attached to the
+character `c`. `insert(1, 'x')("a|c") = "ax|c"`. When the relative position is
+set to the end of the document, it will stay attached to the end of the
+document.
+
 #### Example: Transform to RelativePosition and back
 
 ```js
@@ -802,14 +857,35 @@ pos.index === 2 // => true
 ```
 
 <dl>
-  <b><code>Y.createRelativePositionFromTypeIndex(Uint8Array|Y.Type, number)</code></b>
-  <dd></dd>
-  <b><code>Y.createAbsolutePositionFromRelativePosition(RelativePosition, Y.Doc)</code></b>
-  <dd></dd>
-  <b><code>Y.encodeRelativePosition(RelativePosition):Uint8Array</code></b>
-  <dd></dd>
+  <b><code>
+Y.createRelativePositionFromTypeIndex(type:Uint8Array|Y.Type, index: number
+[, assoc=0])
+  </code></b>
+  <dd>
+Create a relative position fixated to the i-th element in any sequence-like
+shared type (if <code>assoc >= 0</code>). By default, the position associates
+with the character that comes after the specified index position. If
+<code>assoc < 0</code>, then the relative position associates with the character
+before the specified index position.
+  </dd>
+  <b><code>
+Y.createAbsolutePositionFromRelativePosition(RelativePosition, Y.Doc):
+{ type: Y.AbstractType, index: number, assoc: number } | null
+  </code></b>
+  <dd>
+Create an absolute position from a relative position. If the relative position
+cannot be referenced, or the type is deleted, then the result is null.
+  </dd>
+  <b><code>
+Y.encodeRelativePosition(RelativePosition):Uint8Array
+  </code></b>
+  <dd>
+Encode a relative position to an Uint8Array. Binary data is the preferred
+encoding format for document updates. If you prefer JSON encoding, you can
+simply JSON.stringify / JSON.parse the relative position instead.
+  </dd>
   <b><code>Y.decodeRelativePosition(Uint8Array):RelativePosition</code></b>
-  <dd></dd>
+  <dd>Decode a binary-encoded relative position to a RelativePositon object.</dd>
 </dl>
 
 ### Y.UndoManager
@@ -850,6 +926,16 @@ undo- or the redo-stack.
   </dd>
   <b>
     <code>
+on('stack-item-updated', { stackItem: { meta: Map&lt;any,any&gt; }, type: 'undo'
+| 'redo' })
+    </code>
+  </b>
+  <dd>
+Register an event that is called when an existing <code>StackItem</code> is updated.
+This happens when two changes happen within a "captureInterval".
+  </dd>
+  <b>
+    <code>
 on('stack-item-popped', { stackItem: { meta: Map&lt;any,any&gt; }, type: 'undo'
 | 'redo' })
     </code>
@@ -858,6 +944,14 @@ on('stack-item-popped', { stackItem: { meta: Map&lt;any,any&gt; }, type: 'undo'
 Register an event that is called when a <code>StackItem</code> is popped from
 the undo- or the redo-stack.
   </dd>
+  <b>
+    <code>
+on('stack-cleared', { undoStackCleared: boolean, redoStackCleared: boolean })
+    </code>
+  </b>
+  <dd>
+Register an event that is called when the undo- and/or the redo-stack is cleared.
+  </dd>
 </dl>
 
 #### Example: Stop Capturing
@@ -950,7 +1044,7 @@ undoManager.on('stack-item-popped', event => {
 
 *Conflict-free replicated data types* (CRDT) for collaborative editing are an
 alternative approach to *operational transformation* (OT). A very simple
-differenciation between the two approaches is that OT attempts to transform
+differentiation between the two approaches is that OT attempts to transform
 index positions to ensure convergence (all clients end up with the same
 content), while CRDTs use mathematical models that usually do not involve index
 transformations, like linked lists. OT is currently the de-facto standard for
@@ -969,16 +1063,17 @@ More information about the specific implementation is available in
 [INTERNALS.md](./INTERNALS.md) and in
 [this walkthrough of the Yjs codebase](https://youtu.be/0l5XgnQ6rB4).
 
-CRDTs that suitable for shared text editing suffer from the fact that they only grow
-in size. There are CRDTs that do not grow in size, but they do not have the
-characteristics that are benificial for shared text editing (like intention
-preservation). Yjs implements many improvements to the original algorithm that
-diminish the trade-off that the document only grows in size. We can't garbage
-collect deleted structs (tombstones) while ensuring a unique order of the
-structs. But we can 1. merge preceeding structs into a single struct to reduce
-the amount of meta information, 2. we can delete content from the struct if it
-is deleted, and 3. we can garbage collect tombstones if we don't care about the
-order of the structs anymore (e.g. if the parent was deleted).
+CRDTs that are suitable for shared text editing suffer from the fact that they
+only grow in size. There are CRDTs that do not grow in size, but they do not
+have the characteristics that are benificial for shared text editing (like
+intention preservation). Yjs implements many improvements to the original
+algorithm that diminish the trade-off that the document only grows in size. We
+can't garbage collect deleted structs (tombstones) while ensuring a unique
+order of the structs. But we can 1. merge preceeding structs into a single
+struct to reduce the amount of meta information, 2. we can delete content from
+the struct if it is deleted, and 3. we can garbage collect tombstones if we
+don't care about the order of the structs anymore (e.g. if the parent was
+deleted).
 
 **Examples:**
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "yjs",
-  "version": "14.0.0-1",
+  "version": "13.5.46",
   "description": "Shared Editing Library",
   "main": "./dist/yjs.cjs",
   "module": "./dist/yjs.mjs",
@@ -19,14 +19,15 @@
     "lint": "markdownlint README.md && standard && tsc",
     "docs": "rm -rf docs; jsdoc --configure ./.jsdoc.json --verbose --readme ./README.md --package ./package.json || true",
     "serve-docs": "npm run docs && http-server ./docs/",
-    "preversion": "npm run lint && PRODUCTION=1 npm run dist && npm run docs && test -e dist/src/index.d.ts && test -e dist/yjs.cjs && test -e dist/yjs.cjs",
+    "preversion": "npm run lint && PRODUCTION=1 npm run dist && npm run docs && node ./dist/tests.cjs --repetition-time 1000 && test -e dist/src/index.d.ts && test -e dist/yjs.cjs && test -e dist/yjs.cjs",
     "debug": "concurrently 'http-server -o test.html' 'npm run watch'",
     "trace-deopt": "clear && rollup -c  && node --trace-deopt dist/test.cjs",
-    "trace-opt": "clear && rollup -c  && node --trace-opt dist/test.cjs",
-    "postinstall": "node ./sponsor-y.js"
+    "trace-opt": "clear && rollup -c  && node --trace-opt dist/test.cjs"
   },
   "exports": {
     ".": {
+      "types": "./dist/src/index.d.ts",
+      "module": "./dist/yjs.mjs",
       "import": "./dist/yjs.mjs",
       "require": "./dist/yjs.cjs"
     },
@@ -74,19 +75,19 @@
   },
   "homepage": "https://docs.yjs.dev",
   "dependencies": {
-    "lib0": "^0.2.43"
+    "lib0": "^0.2.49"
   },
   "devDependencies": {
     "@rollup/plugin-commonjs": "^17.0.0",
     "@rollup/plugin-node-resolve": "^11.2.1",
     "concurrently": "^3.6.1",
+    "typescript": "^4.9.5",
     "http-server": "^0.12.3",
     "jsdoc": "^3.6.7",
     "markdownlint-cli": "^0.23.2",
     "rollup": "^2.60.0",
     "standard": "^16.0.4",
     "tui-jsdoc-template": "^1.2.2",
-    "typescript": "^4.4.4",
     "y-protocols": "^1.0.5"
   }
 }

sponsor-y.js

@@ -1,12 +0,0 @@
-
-try {
-  const log = require('lib0/dist/logging.cjs')
-
-  log.print()
-  log.print(log.BOLD, log.GREEN, log.BOLD, 'Thank you for using Yjs ', log.RED, '❤\n')
-  log.print(
-    log.GREY,
-    'The project has grown considerably in the past year. Too much for me to maintain\nin my spare time. Several companies built their products with Yjs.\nYet, this project receives very little funding. Yjs is far from done. I want to\ncreate more awesome extensions and work on the growing number of open issues.\n', log.BOLD, 'Dear user, the future of this project entirely depends on you.\n')
-  log.print(log.BLUE, log.BOLD, 'Please start funding the project now: https://github.com/sponsors/dmonad \n')
-  log.print(log.GREY, '(This message will be removed when I achieved my funding goal)\n\n')
-} catch (e) { }

src/index.js

@@ -48,6 +48,7 @@ export {
   findRootTypeKey,
   findIndexSS,
   getItem,
+  typeListToArraySnapshot,
   typeMapGetSnapshot,
   createDocFromSnapshot,
   iterateDeletedStructs,
@@ -66,6 +67,8 @@ export {
   decodeStateVector,
   logUpdate,
   logUpdateV2,
+  decodeUpdate,
+  decodeUpdateV2,
   relativePositionToJSON,
   isDeleted,
   isParentOf,
@@ -84,19 +87,26 @@ export {
   encodeRelativePosition,
   decodeRelativePosition,
   diffUpdate,
-  diffUpdateV2
+  diffUpdateV2,
+  convertUpdateFormatV1ToV2,
+  convertUpdateFormatV2ToV1,
+  UpdateEncoderV1
 } from './internals.js'
 
-const glo = /** @type {any} */ (typeof window !== 'undefined'
-  ? window
-  : typeof global !== 'undefined' ? global : {})
+const glo = /** @type {any} */ (typeof globalThis !== 'undefined'
+  ? globalThis
+  : typeof window !== 'undefined'
+    ? window
+    // @ts-ignore
+    : typeof global !== 'undefined' ? global : {})
+
 const importIdentifier = '__ $YJS$ __'
 
 if (glo[importIdentifier] === true) {
   /**
-   * Dear reader of this warning message. Please take this seriously.
+   * Dear reader of this message. Please take this seriously.
    *
-   * If you see this message, please make sure that you only import one version of Yjs. In many cases,
+   * If you see this message, make sure that you only import one version of Yjs. In many cases,
    * your package manager installs two versions of Yjs that are used by different packages within your project.
    * Another reason for this message is that some parts of your project use the commonjs version of Yjs
    * and others use the EcmaScript version of Yjs.
@@ -104,7 +114,9 @@ if (glo[importIdentifier] === true) {
    * This often leads to issues that are hard to debug. We often need to perform constructor checks,
    * e.g. `struct instanceof GC`. If you imported different versions of Yjs, it is impossible for us to
    * do the constructor checks anymore - which might break the CRDT algorithm.
+   *
+   * https://github.com/yjs/yjs/issues/438
    */
-  console.warn('Yjs was already imported. Importing different versions of Yjs often leads to issues.')
+  console.error('Yjs was already imported. This breaks constructor checks and will lead to issues! - https://github.com/yjs/yjs/issues/438')
 }
 glo[importIdentifier] = true

src/internals.js

@@ -8,7 +8,6 @@ export * from './utils/encoding.js'
 export * from './utils/EventHandler.js'
 export * from './utils/ID.js'
 export * from './utils/isParentOf.js'
-export * from './utils/ListCursor.js'
 export * from './utils/logging.js'
 export * from './utils/PermanentUserData.js'
 export * from './utils/RelativePosition.js'
@@ -39,7 +38,6 @@ export * from './structs/ContentFormat.js'
 export * from './structs/ContentJSON.js'
 export * from './structs/ContentAny.js'
 export * from './structs/ContentString.js'
-export * from './structs/ContentMove.js'
 export * from './structs/ContentType.js'
 export * from './structs/Item.js'
 export * from './structs/Skip.js'

src/structs/ContentFormat.js

@@ -100,4 +100,4 @@ export class ContentFormat {
  * @param {UpdateDecoderV1 | UpdateDecoderV2} decoder
  * @return {ContentFormat}
  */
-export const readContentFormat = decoder => new ContentFormat(decoder.readString(), decoder.readJSON())
+export const readContentFormat = decoder => new ContentFormat(decoder.readKey(), decoder.readJSON())

src/structs/ContentMove.js

@@ -1,296 +0,0 @@
-
-import * as error from 'lib0/error'
-import * as decoding from 'lib0/decoding'
-import * as encoding from 'lib0/encoding'
-import * as math from 'lib0/math'
-import {
-  writeID,
-  readID,
-  ID, AbstractType, ContentType, RelativePosition, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Transaction, Item, StructStore, getItem, getItemCleanStart, getItemCleanEnd, // eslint-disable-line
-  addsStruct
-} from '../internals.js'
-
-/**
- * @param {ContentMove | { start: RelativePosition, end: RelativePosition }} moved
- * @param {Transaction} tr
- * @param {boolean} split
- * @return {{ start: Item, end: Item }} $start (inclusive) is the beginning and $end (inclusive) is the end of the moved area
- */
-export const getMovedCoords = (moved, tr, split) => {
-  const store = tr.doc.store
-  const startItem = moved.start.item
-  const endItem = moved.end.item
-  let start // this (inclusive) is the beginning of the moved area
-  let end // this (exclusive) is the first item after start that is not part of the moved area
-  if (startItem) {
-    if (moved.start.assoc < 0) {
-      // We know that the items have already been split, hence getItem suffices.
-      start = split ? getItemCleanEnd(tr, startItem) : getItem(store, startItem)
-      start = start.right
-    } else {
-      start = split ? getItemCleanStart(tr, startItem) : getItem(store, startItem)
-    }
-  } else if (moved.start.tname != null) {
-    start = tr.doc.get(moved.start.tname)._start
-  } else if (moved.start.type) {
-    start = /** @type {ContentType} */ (getItem(store, moved.start.type).content).type._start
-  } else {
-    error.unexpectedCase()
-  }
-  if (endItem) {
-    if (moved.end.assoc < 0) {
-      end = split ? getItemCleanEnd(tr, endItem) : getItem(store, endItem)
-      end = end.right
-    } else {
-      end = split ? getItemCleanStart(tr, endItem) : getItem(store, endItem)
-    }
-  } else {
-    error.unexpectedCase()
-  }
-  return { start: /** @type {Item} */ (start), end: /** @type {Item} */ (end) }
-}
-
-/**
- * @param {Transaction} tr
- * @param {ContentMove} moved
- * @param {Item} movedItem
- * @param {Set<Item>} trackedMovedItems
- * @return {boolean} true if there is a loop
- */
-export const findMoveLoop = (tr, moved, movedItem, trackedMovedItems) => {
-  if (trackedMovedItems.has(movedItem)) {
-    return true
-  }
-  trackedMovedItems.add(movedItem)
-  /**
-   * @type {{ start: Item | null, end: Item | null }}
-   */
-  let { start, end } = getMovedCoords(moved, tr, false)
-  while (start !== end && start != null) {
-    if (
-      !start.deleted &&
-      start.moved === movedItem &&
-      start.content.constructor === ContentMove &&
-      findMoveLoop(tr, start.content, start, trackedMovedItems)
-    ) {
-      return true
-    }
-    start = start.right
-  }
-  return false
-}
-
-/**
- * @private
- */
-export class ContentMove {
-  /**
-   * @param {RelativePosition} start
-   * @param {RelativePosition} end
-   * @param {number} priority if we want to move content that is already moved, we need to assign a higher priority to this move operation.
-   */
-  constructor (start, end, priority) {
-    this.start = start
-    this.end = end
-    this.priority = priority
-    /**
-     * We store which Items+ContentMove we override. Once we delete
-     * this ContentMove, we need to re-integrate the overridden items.
-     *
-     * This representation can be improved if we ever run into memory issues because of too many overrides.
-     * Ideally, we should probably just re-iterate the document and re-integrate all moved items.
-     * This is fast enough and reduces memory footprint significantly.
-     *
-     * @type {Set<Item>}
-     */
-    this.overrides = new Set()
-  }
-
-  /**
-   * @return {number}
-   */
-  getLength () {
-    return 1
-  }
-
-  /**
-   * @return {Array<any>}
-   */
-  getContent () {
-    return [null]
-  }
-
-  /**
-   * @return {boolean}
-   */
-  isCountable () {
-    return false
-  }
-
-  /**
-   * @return {ContentMove}
-   */
-  copy () {
-    return new ContentMove(this.start, this.end, this.priority)
-  }
-
-  /**
-   * @param {number} offset
-   * @return {ContentMove}
-   */
-  splice (offset) {
-    return this
-  }
-
-  /**
-   * @param {ContentMove} right
-   * @return {boolean}
-   */
-  mergeWith (right) {
-    return false
-  }
-
-  /**
-   * @param {Transaction} transaction
-   * @param {Item} item
-   */
-  integrate (transaction, item) {
-    const sm = /** @type {AbstractType<any>} */ (item.parent)._searchMarker
-    if (sm) sm.length = 0
-    const movedCoords = getMovedCoords(this, transaction, true)
-    /**
-     * @type {{ start: Item | null, end: item | null }}
-     */
-    let { start, end } = movedCoords
-    let maxPriority = 0
-    // If this ContentMove was created locally, we set prio = -1. This indicates
-    // that we want to set prio to the current prio-maximum of the moved range.
-    const adaptPriority = this.priority < 0
-    while (start !== end && start != null) {
-      const prevMove = start.moved // this is the same as prevMove
-      const nextPrio = prevMove ? /** @type {ContentMove} */ (prevMove.content).priority : -1
-      if (adaptPriority || nextPrio < this.priority || (prevMove != null && nextPrio === this.priority && (prevMove.id.client < item.id.client || (prevMove.id.client === item.id.client && prevMove.id.clock < item.id.clock)))) {
-        if (prevMove !== null) {
-          if (/** @type {ContentMove} */ (prevMove.content).isCollapsed()) {
-            prevMove.deleteAsCleanup(transaction, adaptPriority)
-          }
-          this.overrides.add(prevMove)
-          if (start !== movedCoords.start) {
-            // only add this to mergeStructs if this is not the first item
-            transaction._mergeStructs.push(start)
-          }
-        }
-        maxPriority = math.max(maxPriority, nextPrio)
-        // was already moved
-        if (prevMove && !transaction.prevMoved.has(start) && !addsStruct(transaction, prevMove)) {
-          // only override prevMoved if the prevMoved item is not new
-          // we need to know which item previously moved an item
-          transaction.prevMoved.set(start, prevMove)
-        }
-        start.moved = item
-        if (!start.deleted && start.content.constructor === ContentMove && findMoveLoop(transaction, start.content, start, new Set([item]))) {
-          item.deleteAsCleanup(transaction, adaptPriority)
-          return
-        }
-      } else if (prevMove != null) {
-        /** @type {ContentMove} */ (prevMove.content).overrides.add(item)
-      }
-      start = start.right
-    }
-    if (adaptPriority) {
-      this.priority = maxPriority + 1
-    }
-  }
-
-  /**
-   * @param {Transaction} transaction
-   * @param {Item} item
-   */
-  delete (transaction, item) {
-    /**
-     * @type {{ start: Item | null, end: Item | null }}
-     */
-    let { start, end } = getMovedCoords(this, transaction, false)
-    while (start !== end && start != null) {
-      if (start.moved === item) {
-        const prevMoved = transaction.prevMoved.get(start)
-        if (addsStruct(transaction, item)) {
-          if (prevMoved === item) {
-            // Edge case: Item has been moved by this move op and it has been created & deleted in the same transaction (hence no effect that should be emitted by the change computation)
-            transaction.prevMoved.delete(start)
-          }
-        } else if (prevMoved == null) { // && !addsStruct(tr, item)
-          // Normal case: item has been moved by this move and it has not been created & deleted in the same transaction
-          transaction.prevMoved.set(start, item)
-        }
-        start.moved = null
-      }
-      start = start.right
-    }
-    /**
-     * @param {Item} reIntegrateItem
-     */
-    const reIntegrate = reIntegrateItem => {
-      const content = /** @type {ContentMove} */ (reIntegrateItem.content)
-      // content is not yet transformed to a ContentDeleted
-      if (content.getRef() === 11) {
-        if (reIntegrateItem.deleted) {
-          // potentially we can integrate the items that reIntegrateItem overrides
-          content.overrides.forEach(reIntegrate)
-        } else {
-          content.integrate(transaction, reIntegrateItem)
-        }
-      }
-    }
-    this.overrides.forEach(reIntegrate)
-  }
-
-  /**
-   * @param {StructStore} store
-   */
-  gc (store) {}
-
-  /**
-   * @param {UpdateEncoderV1 | UpdateEncoderV2} encoder
-   * @param {number} offset
-   */
-  write (encoder, offset) {
-    const isCollapsed = this.isCollapsed()
-    encoding.writeVarUint(encoder.restEncoder, (isCollapsed ? 1 : 0) | (this.start.assoc >= 0 ? 2 : 0) | (this.end.assoc >= 0 ? 4 : 0) | this.priority << 6)
-    writeID(encoder.restEncoder, /** @type {ID} */ (this.start.item))
-    if (!isCollapsed) {
-      writeID(encoder.restEncoder, /** @type {ID} */ (this.end.item))
-    }
-  }
-
-  /**
-   * @return {number}
-   */
-  getRef () {
-    return 11
-  }
-
-  isCollapsed () {
-    return this.start.item === this.end.item && this.start.item !== null
-  }
-}
-
-/**
- * @private
- *
- * @param {UpdateDecoderV1 | UpdateDecoderV2} decoder
- * @return {ContentMove}
- */
-export const readContentMove = decoder => {
-  const info = decoding.readVarUint(decoder.restDecoder)
-  const isCollapsed = (info & 1) === 1
-  const startAssoc = (info & 2) === 2 ? 0 : -1
-  const endAssoc = (info & 4) === 4 ? 0 : -1
-  // @TODO use BIT3 & BIT4 to indicate the case `null` is the start/end
-  // BIT5 is reserved for future extensions
-  const priority = info >>> 6
-  const startId = readID(decoder.restDecoder)
-  const start = new RelativePosition(null, null, startId, startAssoc)
-  const end = new RelativePosition(null, null, isCollapsed ? startId : readID(decoder.restDecoder), endAssoc)
-  return new ContentMove(start, end, priority)
-}

src/structs/ContentType.js

@@ -39,7 +39,7 @@ export const YXmlTextRefID = 6
  */
 export class ContentType {
   /**
-   * @param {AbstractType<YEvent>} type
+   * @param {AbstractType<any>} type
    */
   constructor (type) {
     /**
@@ -109,7 +109,7 @@ export class ContentType {
       if (!item.deleted) {
         item.delete(transaction)
       } else {
-        // Whis will be gc'd later and we want to merge it if possible
+        // This will be gc'd later and we want to merge it if possible
         // We try to merge all deleted items after each transaction,
         // but we have no knowledge about that this needs to be merged
         // since it is not in transaction.ds. Hence we add it to transaction._mergeStructs

src/structs/Item.js

@@ -21,14 +21,13 @@ import {
   createID,
   readContentFormat,
   readContentType,
-  readContentMove,
   addChangedTypeToTransaction,
-  UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, ContentType, ContentDeleted, StructStore, ID, AbstractType, Transaction // eslint-disable-line
+  isDeleted,
+  DeleteSet, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, ContentType, ContentDeleted, StructStore, ID, AbstractType, Transaction // eslint-disable-line
 } from '../internals.js'
 
 import * as error from 'lib0/error'
 import * as binary from 'lib0/binary'
-import { ContentMove } from './ContentMove.js'
 
 /**
  * @todo This should return several items
@@ -118,13 +117,6 @@ export const splitItem = (transaction, leftItem, diff) => {
     /** @type {AbstractType<any>} */ (rightItem.parent)._map.set(rightItem.parentSub, rightItem)
   }
   leftItem.length = diff
-  if (leftItem.moved) {
-    rightItem.moved = leftItem.moved
-    const m = transaction.prevMoved.get(leftItem)
-    if (m) {
-      transaction.prevMoved.set(rightItem, m)
-    }
-  }
   return rightItem
 }
 
@@ -134,13 +126,14 @@ export const splitItem = (transaction, leftItem, diff) => {
  * @param {Transaction} transaction The Yjs instance.
  * @param {Item} item
  * @param {Set<Item>} redoitems
- * @param {Array<Item>} itemsToDelete
+ * @param {DeleteSet} itemsToDelete
+ * @param {boolean} ignoreRemoteMapChanges
  *
  * @return {Item|null}
  *
  * @private
  */
-export const redoItem = (transaction, item, redoitems, itemsToDelete) => {
+export const redoItem = (transaction, item, redoitems, itemsToDelete, ignoreRemoteMapChanges) => {
   const doc = transaction.doc
   const store = doc.store
   const ownClientID = doc.clientID
@@ -152,42 +145,27 @@ export const redoItem = (transaction, item, redoitems, itemsToDelete) => {
   /**
    * @type {Item|null}
    */
-  let left
+  let left = null
   /**
    * @type {Item|null}
    */
   let right
+  // make sure that parent is redone
+  if (parentItem !== null && parentItem.deleted === true) {
+    // try to undo parent if it will be undone anyway
+    if (parentItem.redone === null && (!redoitems.has(parentItem) || redoItem(transaction, parentItem, redoitems, itemsToDelete, ignoreRemoteMapChanges) === null)) {
+      return null
+    }
+    while (parentItem.redone !== null) {
+      parentItem = getItemCleanStart(transaction, parentItem.redone)
+    }
+  }
+  const parentType = parentItem === null ? /** @type {AbstractType<any>} */ (item.parent) : /** @type {ContentType} */ (parentItem.content).type
+
   if (item.parentSub === null) {
     // Is an array item. Insert at the old position
     left = item.left
     right = item
-  } else {
-    // Is a map item. Insert as current value
-    left = item
-    while (left.right !== null) {
-      left = left.right
-      if (left.id.client !== ownClientID) {
-        // It is not possible to redo this item because it conflicts with a
-        // change from another client
-        return null
-      }
-    }
-    if (left.right !== null) {
-      left = /** @type {Item} */ (/** @type {AbstractType<any>} */ (item.parent)._map.get(item.parentSub))
-    }
-    right = null
-  }
-  // make sure that parent is redone
-  if (parentItem !== null && parentItem.deleted === true && parentItem.redone === null) {
-    // try to undo parent if it will be undone anyway
-    if (!redoitems.has(parentItem) || redoItem(transaction, parentItem, redoitems, itemsToDelete) === null) {
-      return null
-    }
-  }
-  if (parentItem !== null && parentItem.redone !== null) {
-    while (parentItem.redone !== null) {
-      parentItem = getItemCleanStart(transaction, parentItem.redone)
-    }
     // find next cloned_redo items
     while (left !== null) {
       /**
@@ -219,10 +197,27 @@ export const redoItem = (transaction, item, redoitems, itemsToDelete) => {
       }
       right = right.right
     }
-    // Iterate right while right is in itemsToDelete
-    // If it is intended to delete right while item is redone, we can expect that item should replace right.
-    while (left !== null && left.right !== null && left.right !== right && itemsToDelete.findIndex(d => d === /** @type {Item} */ (left).right) >= 0) {
-      left = left.right
+  } else {
+    right = null
+    if (item.right && !ignoreRemoteMapChanges) {
+      left = item
+      // Iterate right while right is in itemsToDelete
+      // If it is intended to delete right while item is redone, we can expect that item should replace right.
+      while (left !== null && left.right !== null && isDeleted(itemsToDelete, left.right.id)) {
+        left = left.right
+      }
+      // follow redone
+      // trace redone until parent matches
+      while (left !== null && left.redone !== null) {
+        left = getItemCleanStart(transaction, left.redone)
+      }
+      if (left && left.right !== null) {
+        // It is not possible to redo this item because it conflicts with a
+        // change from another client
+        return null
+      }
+    } else {
+      left = parentType._map.get(item.parentSub) || null
     }
   }
   const nextClock = getState(store, ownClientID)
@@ -231,7 +226,7 @@ export const redoItem = (transaction, item, redoitems, itemsToDelete) => {
     nextId,
     left, left && left.lastId,
     right, right && right.id,
-    parentItem === null ? item.parent : /** @type {ContentType} */ (parentItem.content).type,
+    parentType,
     item.parentSub,
     item.content.copy()
   )
@@ -290,18 +285,11 @@ export class Item extends AbstractStruct {
      */
     this.parentSub = parentSub
     /**
-     * If this type's effect is reundone this type refers to the type-id that undid
+     * If this type's effect is redone this type refers to the type that undid
      * this operation.
-     *
      * @type {ID | null}
      */
     this.redone = null
-    /**
-     * This property is reused by the moved prop. In this case this property refers to an Item.
-     *
-     * @type {Item | null}
-     */
-    this.moved = null
     /**
      * @type {AbstractContent}
      */
@@ -383,21 +371,11 @@ export class Item extends AbstractStruct {
     if (this.parent && this.parent.constructor === ID && this.id.client !== this.parent.client && this.parent.clock >= getState(store, this.parent.client)) {
       return this.parent.client
     }
-    if (this.content.constructor === ContentMove) {
-      const c = /** @type {ContentMove} */ (this.content)
-      const start = c.start.item
-      const end = c.isCollapsed() ? null : c.end.item
-      if (start && start.clock >= getState(store, start.client)) {
-        return start.client
-      }
-      if (end && end.clock >= getState(store, end.client)) {
-        return end.client
-      }
-    }
 
     // We have all missing ids, now find the items
+
     if (this.origin) {
-      this.left = getItemCleanEnd(transaction, this.origin)
+      this.left = getItemCleanEnd(transaction, store, this.origin)
       this.origin = this.left.lastId
     }
     if (this.rightOrigin) {
@@ -425,7 +403,6 @@ export class Item extends AbstractStruct {
         this.parent = /** @type {ContentType} */ (parentItem.content).type
       }
     }
-
     return null
   }
 
@@ -436,7 +413,7 @@ export class Item extends AbstractStruct {
   integrate (transaction, offset) {
     if (offset > 0) {
       this.id.clock += offset
-      this.left = getItemCleanEnd(transaction, createID(this.id.client, this.id.clock - 1))
+      this.left = getItemCleanEnd(transaction, transaction.doc.store, createID(this.id.client, this.id.clock - 1))
       this.origin = this.left.lastId
       this.content = this.content.splice(offset)
       this.length -= offset
@@ -535,24 +512,6 @@ export class Item extends AbstractStruct {
       if (this.parentSub === null && this.countable && !this.deleted) {
         /** @type {AbstractType<any>} */ (this.parent)._length += this.length
       }
-      // check if this item is in a moved range
-      if ((this.left && this.left.moved) || (this.right && this.right.moved)) {
-        const leftMoved = this.left && this.left.moved && /** @type {ContentMove} */ (this.left.moved.content)
-        const rightMoved = this.right && this.right.moved && /** @type {ContentMove} */ (this.right.moved.content)
-        if (leftMoved === rightMoved) {
-          this.moved = /** @type {Item} */ (this.left).moved
-        } else if (
-          (leftMoved != null && !leftMoved.isCollapsed()) ||
-          (rightMoved != null && !rightMoved.isCollapsed())
-        ) {
-          // We know that this item is on the edge of a moved range.
-          // @todo Instead, we could check to which moved-range this item belongs
-          // This approach (reintegration) is pretty expensive in some scenarios
-          leftMoved && leftMoved.integrate(transaction, /** @type {any} */ (this.left).moved)
-          rightMoved && rightMoved.integrate(transaction, /** @type {any} */ (this.right).moved)
-        }
-      }
-
       addStruct(transaction.doc.store, this)
       this.content.integrate(transaction, this)
       // add parent to transaction.changed
@@ -614,22 +573,21 @@ export class Item extends AbstractStruct {
       this.deleted === right.deleted &&
       this.redone === null &&
       right.redone === null &&
-      this.moved === right.moved &&
       this.content.constructor === right.content.constructor &&
       this.content.mergeWith(right.content)
     ) {
-      if (right.marker) {
-        // Right will be "forgotten", so we delete all
-        // search markers that reference right.
-        const searchMarker = /** @type {AbstractType<any>} */ (this.parent)._searchMarker
-        if (searchMarker) {
-          for (let i = searchMarker.length - 1; i >= 0; i--) {
-            if (searchMarker[i].nextItem === right) {
-              // @todo do something more efficient than splicing..
-              searchMarker.splice(i, 1)
+      const searchMarker = /** @type {AbstractType<any>} */ (this.parent)._searchMarker
+      if (searchMarker) {
+        searchMarker.forEach(marker => {
+          if (marker.p === right) {
+            // right is going to be "forgotten" so we need to update the marker
+            marker.p = this
+            // adjust marker index
+            if (!this.deleted && this.countable) {
+              marker.index -= this.length
             }
           }
-        }
+        })
       }
       if (right.keep) {
         this.keep = true
@@ -659,23 +617,7 @@ export class Item extends AbstractStruct {
       this.markDeleted()
       addToDeleteSet(transaction.deleteSet, this.id.client, this.id.clock, this.length)
       addChangedTypeToTransaction(transaction, parent, this.parentSub)
-      this.content.delete(transaction, this)
-    }
-  }
-
-  /**
-   * Similar to `this.delete(tr)`, but additionally ensures
-   * that the deleted range is broadcasted using a different
-   * origin/source in a separate update event, so that
-   * the providers don't filter this message.
-   *
-   * @param {Transaction} transaction
-   * @param {boolean} isLocal
-   */
-  deleteAsCleanup (transaction, isLocal) {
-    this.delete(transaction)
-    if (!isLocal) {
-      addToDeleteSet(transaction.cleanupDeletions, this.id.client, this.id.clock, this.length)
+      this.content.delete(transaction)
     }
   }
 
@@ -762,7 +704,7 @@ export const readItemContent = (decoder, info) => contentRefs[info & binary.BITS
  * @type {Array<function(UpdateDecoderV1 | UpdateDecoderV2):AbstractContent>}
  */
 export const contentRefs = [
-  error.unexpectedCase, // GC is not ItemContent
+  () => { error.unexpectedCase() }, // GC is not ItemContent
   readContentDeleted, // 1
   readContentJSON, // 2
   readContentBinary, // 3
@@ -772,8 +714,7 @@ export const contentRefs = [
   readContentType, // 7
   readContentAny, // 8
   readContentDoc, // 9
-  error.unexpectedCase, // 10 - Skip is not ItemContent
-  readContentMove // 11
+  () => { error.unexpectedCase() } // 10 - Skip is not ItemContent
 ]
 
 /**
@@ -840,9 +781,8 @@ export class AbstractContent {
 
   /**
    * @param {Transaction} transaction
-   * @param {Item} item
    */
-  delete (transaction, item) {
+  delete (transaction) {
     throw error.methodUnimplemented()
   }
 

src/types/AbstractType.js

@@ -10,8 +10,8 @@ import {
   createID,
   ContentAny,
   ContentBinary,
-  ListCursor,
-  ContentDoc, UpdateEncoderV1, UpdateEncoderV2, Doc, Snapshot, Transaction, EventHandler, YEvent, Item, // eslint-disable-line
+  getItemCleanStart,
+  ContentDoc, YText, YArray, UpdateEncoderV1, UpdateEncoderV2, Doc, Snapshot, Transaction, EventHandler, YEvent, Item, // eslint-disable-line
 } from '../internals.js'
 
 import * as map from 'lib0/map'
@@ -19,8 +19,68 @@ import * as iterator from 'lib0/iterator'
 import * as error from 'lib0/error'
 import * as math from 'lib0/math'
 
-const maxSearchMarker = 300
-const freshSearchMarkerDistance = 30
+const maxSearchMarker = 80
+
+/**
+ * A unique timestamp that identifies each marker.
+ *
+ * Time is relative,.. this is more like an ever-increasing clock.
+ *
+ * @type {number}
+ */
+let globalSearchMarkerTimestamp = 0
+
+export class ArraySearchMarker {
+  /**
+   * @param {Item} p
+   * @param {number} index
+   */
+  constructor (p, index) {
+    p.marker = true
+    this.p = p
+    this.index = index
+    this.timestamp = globalSearchMarkerTimestamp++
+  }
+}
+
+/**
+ * @param {ArraySearchMarker} marker
+ */
+const refreshMarkerTimestamp = marker => { marker.timestamp = globalSearchMarkerTimestamp++ }
+
+/**
+ * This is rather complex so this function is the only thing that should overwrite a marker
+ *
+ * @param {ArraySearchMarker} marker
+ * @param {Item} p
+ * @param {number} index
+ */
+const overwriteMarker = (marker, p, index) => {
+  marker.p.marker = false
+  marker.p = p
+  p.marker = true
+  marker.index = index
+  marker.timestamp = globalSearchMarkerTimestamp++
+}
+
+/**
+ * @param {Array<ArraySearchMarker>} searchMarker
+ * @param {Item} p
+ * @param {number} index
+ */
+const markPosition = (searchMarker, p, index) => {
+  if (searchMarker.length >= maxSearchMarker) {
+    // override oldest marker (we don't want to create more objects)
+    const marker = searchMarker.reduce((a, b) => a.timestamp < b.timestamp ? a : b)
+    overwriteMarker(marker, p, index)
+    return marker
+  } else {
+    // create new marker
+    const pm = new ArraySearchMarker(p, index)
+    searchMarker.push(pm)
+    return pm
+  }
+}
 
 /**
  * Search marker help us to find positions in the associative array faster.
@@ -29,64 +89,82 @@ const freshSearchMarkerDistance = 30
  *
  * A maximum of `maxSearchMarker` objects are created.
  *
- * @template T
- * @param {Transaction} tr
+ * This function always returns a refreshed marker (updated timestamp)
+ *
  * @param {AbstractType<any>} yarray
  * @param {number} index
- * @param {function(ListCursor):T} f
- * @return T
  */
-export const useSearchMarker = (tr, yarray, index, f) => {
-  const searchMarker = yarray._searchMarker
-  if (searchMarker === null || yarray._start === null || index < freshSearchMarkerDistance) {
-    return f(new ListCursor(yarray).forward(tr, index, true))
+export const findMarker = (yarray, index) => {
+  if (yarray._start === null || index === 0 || yarray._searchMarker === null) {
+    return null
   }
-  if (searchMarker.length === 0) {
-    const sm = new ListCursor(yarray).forward(tr, index, true)
-    searchMarker.push(sm)
-    if (sm.nextItem) sm.nextItem.marker = true
+  const marker = yarray._searchMarker.length === 0 ? null : yarray._searchMarker.reduce((a, b) => math.abs(index - a.index) < math.abs(index - b.index) ? a : b)
+  let p = yarray._start
+  let pindex = 0
+  if (marker !== null) {
+    p = marker.p
+    pindex = marker.index
+    refreshMarkerTimestamp(marker) // we used it, we might need to use it again
   }
-  const sm = searchMarker.reduce(
-    (a, b, arrayIndex) => math.abs(index - a.index) < math.abs(index - b.index) ? a : b
-  )
-  const newIsCheaper = math.abs(sm.index - index) >= index
-  const createFreshMarker = searchMarker.length < maxSearchMarker && (math.abs(sm.index - index) > freshSearchMarkerDistance || newIsCheaper)
-  const fsm = createFreshMarker ? (newIsCheaper ? new ListCursor(yarray) : sm.clone()) : sm
-  const prevItem = /** @type {Item} */ (sm.nextItem)
-  if (createFreshMarker) {
-    searchMarker.push(fsm)
+  // iterate to right if possible
+  while (p.right !== null && pindex < index) {
+    if (!p.deleted && p.countable) {
+      if (index < pindex + p.length) {
+        break
+      }
+      pindex += p.length
+    }
+    p = p.right
   }
-  const diff = fsm.index - index
-  if (diff > 0) {
-    fsm.backward(tr, diff)
+  // iterate to left if necessary (might be that pindex > index)
+  while (p.left !== null && pindex > index) {
+    p = p.left
+    if (!p.deleted && p.countable) {
+      pindex -= p.length
+    }
+  }
+  // we want to make sure that p can't be merged with left, because that would screw up everything
+  // in that cas just return what we have (it is most likely the best marker anyway)
+  // iterate to left until p can't be merged with left
+  while (p.left !== null && p.left.id.client === p.id.client && p.left.id.clock + p.left.length === p.id.clock) {
+    p = p.left
+    if (!p.deleted && p.countable) {
+      pindex -= p.length
+    }
+  }
+
+  // @todo remove!
+  // assure position
+  // {
+  //   let start = yarray._start
+  //   let pos = 0
+  //   while (start !== p) {
+  //     if (!start.deleted && start.countable) {
+  //       pos += start.length
+  //     }
+  //     start = /** @type {Item} */ (start.right)
+  //   }
+  //   if (pos !== pindex) {
+  //     debugger
+  //     throw new Error('Gotcha position fail!')
+  //   }
+  // }
+  // if (marker) {
+  //   if (window.lengthes == null) {
+  //     window.lengthes = []
+  //     window.getLengthes = () => window.lengthes.sort((a, b) => a - b)
+  //   }
+  //   window.lengthes.push(marker.index - pindex)
+  //   console.log('distance', marker.index - pindex, 'len', p && p.parent.length)
+  // }
+  if (marker !== null && math.abs(marker.index - pindex) < /** @type {YText|YArray<any>} */ (p.parent).length / maxSearchMarker) {
+    // adjust existing marker
+    overwriteMarker(marker, p, pindex)
+    return marker
   } else {
-    fsm.forward(tr, -diff, true)
+    // create new marker
+    return markPosition(yarray._searchMarker, p, pindex)
   }
-  const result = f(fsm)
-  if (fsm.reachedEnd) {
-    fsm.reachedEnd = false
-    const nextItem = /** @type {Item} */ (fsm.nextItem)
-    if (nextItem.countable && !nextItem.deleted) {
-      fsm.index -= nextItem.length
-    }
-    fsm.rel = 0
-  }
-  fsm.index -= fsm.rel
-  fsm.rel = 0
-  if (!createFreshMarker) {
-    // reused old marker and we moved to a different position
-    prevItem.marker = false
-  }
-  const fsmItem = fsm.nextItem
-  if (fsmItem) {
-    if (fsmItem.marker) {
-      // already marked, forget current iterator
-      searchMarker.splice(searchMarker.findIndex(m => m === fsm), 1)
-    } else {
-      fsmItem.marker = true
-    }
-  }
-  return result
 }
 
 /**
@@ -94,25 +172,39 @@ export const useSearchMarker = (tr, yarray, index, f) => {
  *
  * This should be called before doing a deletion!
  *
- * @param {Array<ListCursor>} searchMarker
+ * @param {Array<ArraySearchMarker>} searchMarker
  * @param {number} index
  * @param {number} len If insertion, len is positive. If deletion, len is negative.
- * @param {ListCursor|null} origSearchMarker Do not update this searchmarker because it is the one we used to manipulate. @todo !=null for improved perf in ytext
  */
-export const updateMarkerChanges = (searchMarker, index, len, origSearchMarker) => {
+export const updateMarkerChanges = (searchMarker, index, len) => {
   for (let i = searchMarker.length - 1; i >= 0; i--) {
-    const marker = searchMarker[i]
-    if (marker !== origSearchMarker) {
-      if (len > 0 && index === marker.index) {
-        // inserting at a marked position deletes the marked position because we can't do a simple transformation
-        // (we don't know whether to insert directly before or directly after the position)
+    const m = searchMarker[i]
+    if (len > 0) {
+      /**
+       * @type {Item|null}
+       */
+      let p = m.p
+      p.marker = false
+      // Ideally we just want to do a simple position comparison, but this will only work if
+      // search markers don't point to deleted items for formats.
+      // Iterate marker to prev undeleted countable position so we know what to do when updating a position
+      while (p && (p.deleted || !p.countable)) {
+        p = p.left
+        if (p && !p.deleted && p.countable) {
+          // adjust position. the loop should break now
+          m.index -= p.length
+        }
+      }
+      if (p === null || p.marker === true) {
+        // remove search marker if updated position is null or if position is already marked
         searchMarker.splice(i, 1)
-        if (marker.nextItem) marker.nextItem.marker = false
         continue
       }
-      if (index < marker.index) { // a simple index <= m.index check would actually suffice
-        marker.index = math.max(index, marker.index + len)
-      }
+      m.p = p
+      p.marker = true
+    }
+    if (index < m.index || (len > 0 && index === m.index)) { // a simple index <= m.index check would actually suffice
+      m.index = math.max(index, m.index + len)
     }
   }
 }
@@ -186,20 +278,13 @@ export class AbstractType {
     this._eH = createEventHandler()
     /**
      * Deep event handlers
-     * @type {EventHandler<Array<YEvent>,Transaction>}
+     * @type {EventHandler<Array<YEvent<any>>,Transaction>}
      */
     this._dEH = createEventHandler()
     /**
-     * @type {null | Array<ListCursor>}
+     * @type {null | Array<ArraySearchMarker>}
      */
     this._searchMarker = null
-    /**
-     * You can store custom stuff here.
-     * This might be useful to associate your application state to this shared type.
-     *
-     * @type {Map<any, any>}
-     */
-    this.meta = new Map()
   }
 
   /**
@@ -239,9 +324,9 @@ export class AbstractType {
   }
 
   /**
-   * @param {UpdateEncoderV1 | UpdateEncoderV2} encoder
+   * @param {UpdateEncoderV1 | UpdateEncoderV2} _encoder
    */
-  _write (encoder) { }
+  _write (_encoder) { }
 
   /**
    * The first non-deleted item
@@ -259,9 +344,9 @@ export class AbstractType {
    * Must be implemented by each type.
    *
    * @param {Transaction} transaction
-   * @param {Set<null|string>} parentSubs Keys changed on this type. `null` if list was modified.
+   * @param {Set<null|string>} _parentSubs Keys changed on this type. `null` if list was modified.
    */
-  _callObserver (transaction, parentSubs) {
+  _callObserver (transaction, _parentSubs) {
     if (!transaction.local && this._searchMarker) {
       this._searchMarker.length = 0
     }
@@ -279,7 +364,7 @@ export class AbstractType {
   /**
    * Observe all events that are created by this type and its children.
    *
-   * @param {function(Array<YEvent>,Transaction):void} f Observer function
+   * @param {function(Array<YEvent<any>>,Transaction):void} f Observer function
    */
   observeDeep (f) {
     addEventHandlerListener(this._dEH, f)
@@ -297,7 +382,7 @@ export class AbstractType {
   /**
    * Unregister an observer function.
    *
-   * @param {function(Array<YEvent>,Transaction):void} f Observer function
+   * @param {function(Array<YEvent<any>>,Transaction):void} f Observer function
    */
   unobserveDeep (f) {
     removeEventHandlerListener(this._dEH, f)
@@ -369,6 +454,171 @@ export const typeListToArray = type => {
   return cs
 }
 
+/**
+ * @param {AbstractType<any>} type
+ * @param {Snapshot} snapshot
+ * @return {Array<any>}
+ *
+ * @private
+ * @function
+ */
+export const typeListToArraySnapshot = (type, snapshot) => {
+  const cs = []
+  let n = type._start
+  while (n !== null) {
+    if (n.countable && isVisible(n, snapshot)) {
+      const c = n.content.getContent()
+      for (let i = 0; i < c.length; i++) {
+        cs.push(c[i])
+      }
+    }
+    n = n.right
+  }
+  return cs
+}
+
+/**
+ * Executes a provided function on once on overy element of this YArray.
+ *
+ * @param {AbstractType<any>} type
+ * @param {function(any,number,any):void} f A function to execute on every element of this YArray.
+ *
+ * @private
+ * @function
+ */
+export const typeListForEach = (type, f) => {
+  let index = 0
+  let n = type._start
+  while (n !== null) {
+    if (n.countable && !n.deleted) {
+      const c = n.content.getContent()
+      for (let i = 0; i < c.length; i++) {
+        f(c[i], index++, type)
+      }
+    }
+    n = n.right
+  }
+}
+
+/**
+ * @template C,R
+ * @param {AbstractType<any>} type
+ * @param {function(C,number,AbstractType<any>):R} f
+ * @return {Array<R>}
+ *
+ * @private
+ * @function
+ */
+export const typeListMap = (type, f) => {
+  /**
+   * @type {Array<any>}
+   */
+  const result = []
+  typeListForEach(type, (c, i) => {
+    result.push(f(c, i, type))
+  })
+  return result
+}
+
+/**
+ * @param {AbstractType<any>} type
+ * @return {IterableIterator<any>}
+ *
+ * @private
+ * @function
+ */
+export const typeListCreateIterator = type => {
+  let n = type._start
+  /**
+   * @type {Array<any>|null}
+   */
+  let currentContent = null
+  let currentContentIndex = 0
+  return {
+    [Symbol.iterator] () {
+      return this
+    },
+    next: () => {
+      // find some content
+      if (currentContent === null) {
+        while (n !== null && n.deleted) {
+          n = n.right
+        }
+        // check if we reached the end, no need to check currentContent, because it does not exist
+        if (n === null) {
+          return {
+            done: true,
+            value: undefined
+          }
+        }
+        // we found n, so we can set currentContent
+        currentContent = n.content.getContent()
+        currentContentIndex = 0
+        n = n.right // we used the content of n, now iterate to next
+      }
+      const value = currentContent[currentContentIndex++]
+      // check if we need to empty currentContent
+      if (currentContent.length <= currentContentIndex) {
+        currentContent = null
+      }
+      return {
+        done: false,
+        value
+      }
+    }
+  }
+}
+
+/**
+ * Executes a provided function on once on overy element of this YArray.
+ * Operates on a snapshotted state of the document.
+ *
+ * @param {AbstractType<any>} type
+ * @param {function(any,number,AbstractType<any>):void} f A function to execute on every element of this YArray.
+ * @param {Snapshot} snapshot
+ *
+ * @private
+ * @function
+ */
+export const typeListForEachSnapshot = (type, f, snapshot) => {
+  let index = 0
+  let n = type._start
+  while (n !== null) {
+    if (n.countable && isVisible(n, snapshot)) {
+      const c = n.content.getContent()
+      for (let i = 0; i < c.length; i++) {
+        f(c[i], index++, type)
+      }
+    }
+    n = n.right
+  }
+}
+
+/**
+ * @param {AbstractType<any>} type
+ * @param {number} index
+ * @return {any}
+ *
+ * @private
+ * @function
+ */
+export const typeListGet = (type, index) => {
+  const marker = findMarker(type, index)
+  let n = type._start
+  if (marker !== null) {
+    n = marker.p
+    index -= marker.index
+  }
+  for (; n !== null; n = n.right) {
+    if (!n.deleted && n.countable) {
+      if (index < n.length) {
+        return n.content.getContent()[index]
+      }
+      index -= n.length
+    }
+  }
+}
+
 /**
  * @param {Transaction} transaction
  * @param {AbstractType<any>} parent
@@ -433,6 +683,128 @@ export const typeListInsertGenericsAfter = (transaction, parent, referenceItem,
   packJsonContent()
 }
 
+const lengthExceeded = error.create('Length exceeded!')
+
+/**
+ * @param {Transaction} transaction
+ * @param {AbstractType<any>} parent
+ * @param {number} index
+ * @param {Array<Object<string,any>|Array<any>|number|null|string|Uint8Array>} content
+ *
+ * @private
+ * @function
+ */
+export const typeListInsertGenerics = (transaction, parent, index, content) => {
+  if (index > parent._length) {
+    throw lengthExceeded
+  }
+  if (index === 0) {
+    if (parent._searchMarker) {
+      updateMarkerChanges(parent._searchMarker, index, content.length)
+    }
+    return typeListInsertGenericsAfter(transaction, parent, null, content)
+  }
+  const startIndex = index
+  const marker = findMarker(parent, index)
+  let n = parent._start
+  if (marker !== null) {
+    n = marker.p
+    index -= marker.index
+    // we need to iterate one to the left so that the algorithm works
+    if (index === 0) {
+      // @todo refactor this as it actually doesn't consider formats
+      n = n.prev // important! get the left undeleted item so that we can actually decrease index
+      index += (n && n.countable && !n.deleted) ? n.length : 0
+    }
+  }
+  for (; n !== null; n = n.right) {
+    if (!n.deleted && n.countable) {
+      if (index <= n.length) {
+        if (index < n.length) {
+          // insert in-between
+          getItemCleanStart(transaction, createID(n.id.client, n.id.clock + index))
+        }
+        break
+      }
+      index -= n.length
+    }
+  }
+  if (parent._searchMarker) {
+    updateMarkerChanges(parent._searchMarker, startIndex, content.length)
+  }
+  return typeListInsertGenericsAfter(transaction, parent, n, content)
+}
+
+/**
+ * Pushing content is special as we generally want to push after the last item. So we don't have to update
+ * the serach marker.
+ *
+ * @param {Transaction} transaction
+ * @param {AbstractType<any>} parent
+ * @param {Array<Object<string,any>|Array<any>|number|null|string|Uint8Array>} content
+ *
+ * @private
+ * @function
+ */
+export const typeListPushGenerics = (transaction, parent, content) => {
+  // Use the marker with the highest index and iterate to the right.
+  const marker = (parent._searchMarker || []).reduce((maxMarker, currMarker) => currMarker.index > maxMarker.index ? currMarker : maxMarker, { index: 0, p: parent._start })
+  let n = marker.p
+  if (n) {
+    while (n.right) {
+      n = n.right
+    }
+  }
+  return typeListInsertGenericsAfter(transaction, parent, n, content)
+}
+
+/**
+ * @param {Transaction} transaction
+ * @param {AbstractType<any>} parent
+ * @param {number} index
+ * @param {number} length
+ *
+ * @private
+ * @function
+ */
+export const typeListDelete = (transaction, parent, index, length) => {
+  if (length === 0) { return }
+  const startIndex = index
+  const startLength = length
+  const marker = findMarker(parent, index)
+  let n = parent._start
+  if (marker !== null) {
+    n = marker.p
+    index -= marker.index
+  }
+  // compute the first item to be deleted
+  for (; n !== null && index > 0; n = n.right) {
+    if (!n.deleted && n.countable) {
+      if (index < n.length) {
+        getItemCleanStart(transaction, createID(n.id.client, n.id.clock + index))
+      }
+      index -= n.length
+    }
+  }
+  // delete all items until done
+  while (length > 0 && n !== null) {
+    if (!n.deleted) {
+      if (length < n.length) {
+        getItemCleanStart(transaction, createID(n.id.client, n.id.clock + length))
+      }
+      n.delete(transaction)
+      length -= n.length
+    }
+    n = n.right
+  }
+  if (length > 0) {
+    throw lengthExceeded
+  }
+  if (parent._searchMarker) {
+    updateMarkerChanges(parent._searchMarker, startIndex, -startLength + length /* in case we remove the above exception */)
+  }
+}
+
 /**
  * @param {Transaction} transaction
  * @param {AbstractType<any>} parent

src/types/YArray.js

@@ -5,19 +5,25 @@
 import {
   YEvent,
   AbstractType,
+  typeListGet,
+  typeListToArray,
+  typeListForEach,
+  typeListCreateIterator,
+  typeListInsertGenerics,
+  typeListPushGenerics,
+  typeListDelete,
+  typeListMap,
   YArrayRefID,
   callTypeObservers,
   transact,
-  ListCursor,
-  useSearchMarker,
-  createRelativePositionFromTypeIndex,
-  UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Doc, Transaction, Item, // eslint-disable-line
-  getMinimalListViewRanges
+  ArraySearchMarker, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Doc, Transaction, Item // eslint-disable-line
 } from '../internals.js'
+import { typeListSlice } from './AbstractType.js'
 
 /**
  * Event that describes the changes on a YArray
  * @template T
+ * @extends YEvent<YArray<T>>
  */
 export class YArrayEvent extends YEvent {
   /**
@@ -45,18 +51,21 @@ export class YArray extends AbstractType {
      */
     this._prelimContent = []
     /**
-     * @type {Array<ListCursor>}
+     * @type {Array<ArraySearchMarker>}
      */
     this._searchMarker = []
   }
 
   /**
    * Construct a new YArray containing the specified items.
-   * @template T
+   * @template {Object<string,any>|Array<any>|number|null|string|Uint8Array} T
    * @param {Array<T>} items
    * @return {YArray<T>}
    */
   static from (items) {
+    /**
+     * @type {YArray<T>}
+     */
     const a = new YArray()
     a.push(items)
     return a
@@ -78,6 +87,9 @@ export class YArray extends AbstractType {
     this._prelimContent = null
   }
 
+  /**
+   * @return {YArray<T>}
+   */
   _copy () {
     return new YArray()
   }
@@ -86,9 +98,12 @@ export class YArray extends AbstractType {
    * @return {YArray<T>}
    */
   clone () {
+    /**
+     * @type {YArray<T>}
+     */
     const arr = new YArray()
     arr.insert(0, this.toArray().map(el =>
-      el instanceof AbstractType ? el.clone() : el
+      el instanceof AbstractType ? /** @type {typeof el} */ (el.clone()) : el
     ))
     return arr
   }
@@ -125,83 +140,12 @@ export class YArray extends AbstractType {
    * @param {Array<T>} content The array of content
    */
   insert (index, content) {
-    if (content.length > 0) {
-      if (this.doc !== null) {
-        transact(this.doc, transaction => {
-          useSearchMarker(transaction, this, index, walker =>
-            walker.insertArrayValue(transaction, content)
-          )
-        })
-      } else {
-        /** @type {Array<any>} */ (this._prelimContent).splice(index, 0, ...content)
-      }
-    }
-  }
-
-  /**
-   * Move a single item from $index to $target.
-   *
-   * If the original item is to the left of $target, then the index of the item will decrement.
-   *
-   * ```js
-   * yarray.insert(0, [1, 2, 3])
-   * yarray.move(0, 3) // move "1" to index 3
-   * yarray.toArray() // => [2, 3, 1]
-   * yarray.move(2, 0) // move "1" to index 0
-   * yarray.toArray() // => [1, 2, 3]
-   * ```
-   *
-   * @param {number} index
-   * @param {number} target
-   */
-  move (index, target) {
-    if (index === target || index + 1 === target || index >= this.length) {
-      // It doesn't make sense to move a range into the same range (it's basically a no-op).
-      return
-    }
     if (this.doc !== null) {
       transact(this.doc, transaction => {
-        const start = createRelativePositionFromTypeIndex(this, index, 1)
-        const end = start.clone()
-        end.assoc = -1
-        useSearchMarker(transaction, this, target, walker => {
-          walker.insertMove(transaction, [{ start, end }])
-        })
+        typeListInsertGenerics(transaction, this, index, /** @type {any} */ (content))
       })
     } else {
-      const content = /** @type {Array<any>} */ (this._prelimContent).splice(index, 1)
-      ;/** @type {Array<any>} */ (this._prelimContent).splice(target, 0, ...content)
-    }
-  }
-
-  /**
-   * @experimental
-   *
-   * @param {number} startIndex Inclusive move-start
-   * @param {number} endIndex Inclusive move-end
-   * @param {number} target
-   * @param {number} assocStart >=0 if start should be associated with the right character. See relative-position assoc parameter.
-   * @param {number} assocEnd >= 0 if end should be associated with the right character.
-   */
-  moveRange (startIndex, endIndex, target, assocStart = 1, assocEnd = -1) {
-    if (
-      (startIndex <= target && target <= endIndex) || // It doesn't make sense to move a range into the same range (it's basically a no-op).
-      endIndex - startIndex < 0 // require length of >= 0
-    ) {
-      return
-    }
-    if (this.doc !== null) {
-      transact(this.doc, transaction => {
-        const ranges = useSearchMarker(transaction, this, startIndex, walker =>
-          getMinimalListViewRanges(transaction, walker, endIndex - startIndex + 1)
-        )
-        useSearchMarker(transaction, this, target, walker => {
-          walker.insertMove(transaction, ranges)
-        })
-      })
-    } else {
-      const content = /** @type {Array<any>} */ (this._prelimContent).splice(startIndex, endIndex - startIndex + 1)
-      ;/** @type {Array<any>} */ (this._prelimContent).splice(target, 0, ...content)
+      /** @type {Array<any>} */ (this._prelimContent).splice(index, 0, ...content)
     }
   }
 
@@ -209,9 +153,17 @@ export class YArray extends AbstractType {
    * Appends content to this YArray.
    *
    * @param {Array<T>} content Array of content to append.
+   *
+   * @todo Use the following implementation in all types.
    */
   push (content) {
-    this.insert(this.length, content)
+    if (this.doc !== null) {
+      transact(this.doc, transaction => {
+        typeListPushGenerics(transaction, this, /** @type {any} */ (content))
+      })
+    } else {
+      /** @type {Array<any>} */ (this._prelimContent).push(...content)
+    }
   }
 
   /**
@@ -232,9 +184,7 @@ export class YArray extends AbstractType {
   delete (index, length = 1) {
     if (this.doc !== null) {
       transact(this.doc, transaction => {
-        useSearchMarker(transaction, this, index, walker =>
-          walker.delete(transaction, length)
-        )
+        typeListDelete(transaction, this, index, length)
       })
     } else {
       /** @type {Array<any>} */ (this._prelimContent).splice(index, length)
@@ -248,11 +198,7 @@ export class YArray extends AbstractType {
    * @return {T}
    */
   get (index) {
-    return transact(/** @type {Doc} */ (this.doc), transaction =>
-      useSearchMarker(transaction, this, index, walker =>
-        walker.slice(transaction, 1)[0]
-      )
-    )
+    return typeListGet(this, index)
   }
 
   /**
@@ -261,9 +207,7 @@ export class YArray extends AbstractType {
    * @return {Array<T>}
    */
   toArray () {
-    return transact(/** @type {Doc} */ (this.doc), tr =>
-      new ListCursor(this).slice(tr, this.length)
-    )
+    return typeListToArray(this)
   }
 
   /**
@@ -274,11 +218,7 @@ export class YArray extends AbstractType {
    * @return {Array<T>}
    */
   slice (start = 0, end = this.length) {
-    return transact(/** @type {Doc} */ (this.doc), transaction =>
-      useSearchMarker(transaction, this, start, walker =>
-        walker.slice(transaction, end < 0 ? this.length + end - start : end - start)
-      )
-    )
+    return typeListSlice(this, start, end)
   }
 
   /**
@@ -300,9 +240,7 @@ export class YArray extends AbstractType {
    *                 callback function
    */
   map (f) {
-    return transact(/** @type {Doc} */ (this.doc), tr =>
-      new ListCursor(this).map(tr, f)
-    )
+    return typeListMap(this, /** @type {any} */ (f))
   }
 
   /**
@@ -311,17 +249,14 @@ export class YArray extends AbstractType {
    * @param {function(T,number,YArray<T>):void} f A function to execute on every element of this YArray.
    */
   forEach (f) {
-    return transact(/** @type {Doc} */ (this.doc), tr =>
-      new ListCursor(this).forEach(tr, f)
-    )
+    typeListForEach(this, f)
   }
 
   /**
    * @return {IterableIterator<T>}
    */
   [Symbol.iterator] () {
-    // @todo, this could be optimized using a real iterator
-    return this.toArray().values()
+    return typeListCreateIterator(this)
   }
 
   /**
@@ -333,9 +268,9 @@ export class YArray extends AbstractType {
 }
 
 /**
- * @param {UpdateDecoderV1 | UpdateDecoderV2} decoder
+ * @param {UpdateDecoderV1 | UpdateDecoderV2} _decoder
  *
  * @private
  * @function
  */
-export const readYArray = decoder => new YArray()
+export const readYArray = _decoder => new YArray()

src/types/YMap.js

@@ -21,6 +21,7 @@ import * as iterator from 'lib0/iterator'
 
 /**
  * @template T
+ * @extends YEvent<YMap<T>>
  * Event that describes the changes on a YMap.
  */
 export class YMapEvent extends YEvent {
@@ -80,6 +81,9 @@ export class YMap extends AbstractType {
     this._prelimContent = null
   }
 
+  /**
+   * @return {YMap<MapType>}
+   */
   _copy () {
     return new YMap()
   }
@@ -88,9 +92,12 @@ export class YMap extends AbstractType {
    * @return {YMap<MapType>}
    */
   clone () {
+    /**
+     * @type {YMap<MapType>}
+     */
     const map = new YMap()
     this.forEach((value, key) => {
-      map.set(key, value instanceof AbstractType ? value.clone() : value)
+      map.set(key, value instanceof AbstractType ? /** @type {typeof value} */ (value.clone()) : value)
     })
     return map
   }
@@ -166,20 +173,17 @@ export class YMap extends AbstractType {
    * @param {function(MapType,string,YMap<MapType>):void} f A function to execute on every element of this YArray.
    */
   forEach (f) {
-    /**
-     * @type {Object<string,MapType>}
-     */
-    const map = {}
     this._map.forEach((item, key) => {
       if (!item.deleted) {
         f(item.content.getContent()[item.length - 1], key, this)
       }
     })
-    return map
   }
 
   /**
-   * @return {IterableIterator<MapType>}
+   * Returns an Iterator of [key, value] pairs
+   *
+   * @return {IterableIterator<any>}
    */
   [Symbol.iterator] () {
     return this.entries()
@@ -209,7 +213,7 @@ export class YMap extends AbstractType {
   set (key, value) {
     if (this.doc !== null) {
       transact(this.doc, transaction => {
-        typeMapSet(transaction, this, key, value)
+        typeMapSet(transaction, this, key, /** @type {any} */ (value))
       })
     } else {
       /** @type {Map<string, any>} */ (this._prelimContent).set(key, value)
@@ -243,7 +247,7 @@ export class YMap extends AbstractType {
   clear () {
     if (this.doc !== null) {
       transact(this.doc, transaction => {
-        this.forEach(function (value, key, map) {
+        this.forEach(function (_value, key, map) {
           typeMapDelete(transaction, map, key)
         })
       })
@@ -261,9 +265,9 @@ export class YMap extends AbstractType {
 }
 
 /**
- * @param {UpdateDecoderV1 | UpdateDecoderV2} decoder
+ * @param {UpdateDecoderV1 | UpdateDecoderV2} _decoder
  *
  * @private
  * @function
  */
-export const readYMap = decoder => new YMap()
+export const readYMap = _decoder => new YMap()

src/types/YText.js

@@ -20,15 +20,14 @@ import {
   splitSnapshotAffectedStructs,
   iterateDeletedStructs,
   iterateStructs,
+  findMarker,
   typeMapDelete,
   typeMapSet,
   typeMapGet,
   typeMapGetAll,
   updateMarkerChanges,
   ContentType,
-  useSearchMarker,
-  findIndexCleanStart,
-  ListCursor, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, ID, Doc, Item, Snapshot, Transaction // eslint-disable-line
+  ArraySearchMarker, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, ID, Doc, Item, Snapshot, Transaction // eslint-disable-line
 } from '../internals.js'
 
 import * as object from 'lib0/object'
@@ -126,30 +125,10 @@ const findNextPosition = (transaction, pos, count) => {
  */
 const findPosition = (transaction, parent, index) => {
   const currentAttributes = new Map()
-  if (parent._searchMarker) {
-    return useSearchMarker(transaction, parent, index, listIter => {
-      let left, right
-      if (listIter.rel > 0) {
-        // must exist because rel > 0
-        const nextItem = /** @type {Item} */ (listIter.nextItem)
-        if (listIter.rel === nextItem.length) {
-          left = nextItem
-          right = left.right
-        } else {
-          const structs = /** @type {Array<Item|GC>} */ (transaction.doc.store.clients.get(nextItem.id.client))
-          const after = /** @type {Item} */ (structs[findIndexCleanStart(transaction, structs, nextItem.id.clock + listIter.rel)])
-          listIter.nextItem = after
-          listIter.rel = 0
-          left = listIter.left
-          right = listIter.right
-        }
-      } else {
-        left = listIter.left
-        right = listIter.right
-      }
-      // @todo this should simply split if .rel > 0
-      return new ItemTextListPosition(left, right, index, currentAttributes)
-    })
+  const marker = findMarker(parent, index)
+  if (marker) {
+    const pos = new ItemTextListPosition(marker.p.left, marker.p, marker.index, currentAttributes)
+    return findNextPosition(transaction, pos, index - marker.index)
   } else {
     const pos = new ItemTextListPosition(null, parent._start, 0, currentAttributes)
     return findNextPosition(transaction, pos, index)
@@ -272,7 +251,7 @@ const insertAttributes = (transaction, parent, currPos, attributes) => {
  * @function
  **/
 const insertText = (transaction, parent, currPos, text, attributes) => {
-  currPos.currentAttributes.forEach((val, key) => {
+  currPos.currentAttributes.forEach((_val, key) => {
     if (attributes[key] === undefined) {
       attributes[key] = null
     }
@@ -285,7 +264,7 @@ const insertText = (transaction, parent, currPos, text, attributes) => {
   const content = text.constructor === String ? new ContentString(/** @type {string} */ (text)) : (text instanceof AbstractType ? new ContentType(text) : new ContentEmbed(text))
   let { left, right, index } = currPos
   if (parent._searchMarker) {
-    updateMarkerChanges(parent._searchMarker, currPos.index, content.getLength(), null)
+    updateMarkerChanges(parent._searchMarker, currPos.index, content.getLength())
   }
   right = new Item(createID(ownClientId, getState(doc.store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, content)
   right.integrate(transaction, 0)
@@ -312,7 +291,17 @@ const formatText = (transaction, parent, currPos, length, attributes) => {
   const negatedAttributes = insertAttributes(transaction, parent, currPos, attributes)
   // iterate until first non-format or null is found
   // delete all formats with attributes[format.key] != null
-  while (length > 0 && currPos.right !== null) {
+  // also check the attributes after the first non-format as we do not want to insert redundant negated attributes there
+  // eslint-disable-next-line no-labels
+  iterationLoop: while (
+    currPos.right !== null &&
+    (length > 0 ||
+      (
+        negatedAttributes.size > 0 &&
+        (currPos.right.deleted || currPos.right.content.constructor === ContentFormat)
+      )
+    )
+  ) {
     if (!currPos.right.deleted) {
       switch (currPos.right.content.constructor) {
         case ContentFormat: {
@@ -322,9 +311,16 @@ const formatText = (transaction, parent, currPos, length, attributes) => {
             if (equalAttrs(attr, value)) {
               negatedAttributes.delete(key)
             } else {
+              if (length === 0) {
+                // no need to further extend negatedAttributes
+                // eslint-disable-next-line no-labels
+                break iterationLoop
+              }
               negatedAttributes.set(key, value)
             }
             currPos.right.delete(transaction)
+          } else {
+            currPos.currentAttributes.set(key, value)
           }
           break
         }
@@ -359,14 +355,16 @@ const formatText = (transaction, parent, currPos, length, attributes) => {
  *
  * @param {Transaction} transaction
  * @param {Item} start
- * @param {Item|null} end exclusive end, automatically iterates to the next Content Item
+ * @param {Item|null} curr exclusive end, automatically iterates to the next Content Item
  * @param {Map<string,any>} startAttributes
- * @param {Map<string,any>} endAttributes This attribute is modified!
+ * @param {Map<string,any>} currAttributes
  * @return {number} The amount of formatting Items deleted.
  *
  * @function
  */
-const cleanupFormattingGap = (transaction, start, end, startAttributes, endAttributes) => {
+const cleanupFormattingGap = (transaction, start, curr, startAttributes, currAttributes) => {
+  let end = curr
+  const endAttributes = map.copy(currAttributes)
   while (end && (!end.countable || end.deleted)) {
     if (!end.deleted && end.content.constructor === ContentFormat) {
       updateCurrentAttributes(endAttributes, /** @type {ContentFormat} */ (end.content))
@@ -374,16 +372,28 @@ const cleanupFormattingGap = (transaction, start, end, startAttributes, endAttri
     end = end.right
   }
   let cleanups = 0
+  let reachedEndOfCurr = false
   while (start !== end) {
+    if (curr === start) {
+      reachedEndOfCurr = true
+    }
     if (!start.deleted) {
       const content = start.content
       switch (content.constructor) {
         case ContentFormat: {
           const { key, value } = /** @type {ContentFormat} */ (content)
-          if ((endAttributes.get(key) || null) !== value || (startAttributes.get(key) || null) === value) {
+          const startAttrValue = startAttributes.get(key) || null
+          if ((endAttributes.get(key) || null) !== value || startAttrValue === value) {
             // Either this format is overwritten or it is not necessary because the attribute already existed.
             start.delete(transaction)
             cleanups++
+            if (!reachedEndOfCurr && (currAttributes.get(key) || null) === value && (startAttributes.get(key) || null) !== value) {
+              if (startAttrValue === null) {
+                currAttributes.delete(key)
+              } else {
+                currAttributes.set(key, startAttrValue)
+              }
+            }
           }
           break
         }
@@ -486,11 +496,11 @@ const deleteText = (transaction, currPos, length) => {
     currPos.forward()
   }
   if (start) {
-    cleanupFormattingGap(transaction, start, currPos.right, startAttrs, map.copy(currPos.currentAttributes))
+    cleanupFormattingGap(transaction, start, currPos.right, startAttrs, currPos.currentAttributes)
   }
   const parent = /** @type {AbstractType<any>} */ (/** @type {Item} */ (currPos.left || currPos.right).parent)
   if (parent._searchMarker) {
-    updateMarkerChanges(parent._searchMarker, currPos.index, -startLength + length, null)
+    updateMarkerChanges(parent._searchMarker, currPos.index, -startLength + length)
   }
   return currPos
 }
@@ -523,6 +533,7 @@ const deleteText = (transaction, currPos, length) => {
   */
 
 /**
+ * @extends YEvent<YText>
  * Event that describes the changes on a YText type.
  */
 export class YTextEvent extends YEvent {
@@ -705,7 +716,7 @@ export class YTextEvent extends YEvent {
                     } else {
                       attributes[key] = value
                     }
-                  } else {
+                  } else if (value !== null) {
                     item.delete(transaction)
                   }
                 }
@@ -731,7 +742,7 @@ export class YTextEvent extends YEvent {
                     } else {
                       attributes[key] = value
                     }
-                  } else {
+                  } else if (attr !== null) { // this will be cleaned up automatically by the contextless cleanup function
                     item.delete(transaction)
                   }
                 }
@@ -785,7 +796,7 @@ export class YText extends AbstractType {
      */
     this._pending = string !== undefined ? [() => this.insert(0, string)] : []
     /**
-     * @type {Array<ListCursor>}
+     * @type {Array<ArraySearchMarker>}
      */
     this._searchMarker = []
   }
@@ -921,7 +932,7 @@ export class YText extends AbstractType {
    * Apply a {@link Delta} on this shared YText type.
    *
    * @param {any} delta The changes to apply on this element.
-   * @param {object}  [opts]
+   * @param {object}  opts
    * @param {boolean} [opts.sanitize] Sanitize input delta. Removes ending newlines if set to true.
    *
    *
@@ -1012,12 +1023,12 @@ export class YText extends AbstractType {
             case ContentString: {
               const cur = currentAttributes.get('ychange')
               if (snapshot !== undefined && !isVisible(n, snapshot)) {
-                if (cur === undefined || cur.user !== n.id.client || cur.state !== 'removed') {
+                if (cur === undefined || cur.user !== n.id.client || cur.type !== 'removed') {
                   packStr()
                   currentAttributes.set('ychange', computeYChange ? computeYChange('removed', n.id) : { type: 'removed' })
                 }
               } else if (prevSnapshot !== undefined && !isVisible(n, prevSnapshot)) {
-                if (cur === undefined || cur.user !== n.id.client || cur.state !== 'added') {
+                if (cur === undefined || cur.user !== n.id.client || cur.type !== 'added') {
                   packStr()
                   currentAttributes.set('ychange', computeYChange ? computeYChange('added', n.id) : { type: 'added' })
                 }
@@ -1058,7 +1069,7 @@ export class YText extends AbstractType {
         n = n.right
       }
       packStr()
-    }, splitSnapshotAffectedStructs)
+    }, 'cleanup')
     return ops
   }
 
@@ -1223,12 +1234,11 @@ export class YText extends AbstractType {
    *
    * @note Xml-Text nodes don't have attributes. You can use this feature to assign properties to complete text-blocks.
    *
-   * @param {Snapshot} [snapshot]
    * @return {Object<string, any>} A JSON Object that describes the attributes.
    *
    * @public
    */
-  getAttributes (snapshot) {
+  getAttributes () {
     return typeMapGetAll(this)
   }
 
@@ -1241,10 +1251,10 @@ export class YText extends AbstractType {
 }
 
 /**
- * @param {UpdateDecoderV1 | UpdateDecoderV2} decoder
+ * @param {UpdateDecoderV1 | UpdateDecoderV2} _decoder
  * @return {YText}
  *
  * @private
  * @function
  */
-export const readYText = decoder => new YText()
+export const readYText = _decoder => new YText()

src/types/YXmlElement.js

@@ -7,8 +7,9 @@ import {
   typeMapSet,
   typeMapGet,
   typeMapGetAll,
+  typeListForEach,
   YXmlElementRefID,
-  YXmlText, ContentType, AbstractType, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Snapshot, Doc, Item // eslint-disable-line
+  YXmlText, ContentType, AbstractType, UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Doc, Item // eslint-disable-line
 } from '../internals.js'
 
 /**
@@ -175,15 +176,44 @@ export class YXmlElement extends YXmlFragment {
   /**
    * Returns all attribute name/value pairs in a JSON Object.
    *
-   * @param {Snapshot} [snapshot]
    * @return {Object<string, any>} A JSON Object that describes the attributes.
    *
    * @public
    */
-  getAttributes (snapshot) {
+  getAttributes () {
     return typeMapGetAll(this)
   }
 
+  /**
+   * Creates a Dom Element that mirrors this YXmlElement.
+   *
+   * @param {Document} [_document=document] The document object (you must define
+   *                                        this when calling this method in
+   *                                        nodejs)
+   * @param {Object<string, any>} [hooks={}] Optional property to customize how hooks
+   *                                             are presented in the DOM
+   * @param {any} [binding] You should not set this property. This is
+   *                               used if DomBinding wants to create a
+   *                               association to the created DOM type.
+   * @return {Node} The {@link https://developer.mozilla.org/en-US/docs/Web/API/Element|Dom Element}
+   *
+   * @public
+   */
+  toDOM (_document = document, hooks = {}, binding) {
+    const dom = _document.createElement(this.nodeName)
+    const attrs = this.getAttributes()
+    for (const key in attrs) {
+      dom.setAttribute(key, attrs[key])
+    }
+    typeListForEach(this, yxml => {
+      dom.appendChild(yxml.toDOM(_document, hooks, binding))
+    })
+    if (binding !== undefined) {
+      binding._createAssociation(dom, this)
+    }
+    return dom
+  }
+
   /**
    * Transform the properties of this type to binary and write it to an
    * BinaryEncoder.

src/types/YXmlEvent.js

@@ -5,6 +5,7 @@ import {
 } from '../internals.js'
 
 /**
+ * @extends YEvent<YXmlElement|YXmlText|YXmlFragment>
  * An Event that describes changes on a YXml Element or Yxml Fragment
  */
 export class YXmlEvent extends YEvent {

src/types/YXmlFragment.js

@@ -6,15 +6,18 @@ import {
   YXmlEvent,
   YXmlElement,
   AbstractType,
+  typeListMap,
+  typeListForEach,
+  typeListInsertGenerics,
   typeListInsertGenericsAfter,
+  typeListDelete,
   typeListToArray,
   YXmlFragmentRefID,
   callTypeObservers,
   transact,
+  typeListGet,
   typeListSlice,
-  useSearchMarker,
-  UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Doc, ContentType, Transaction, Item, YXmlText, YXmlHook, Snapshot, // eslint-disable-line
-  ListCursor
+  UpdateDecoderV1, UpdateDecoderV2, UpdateEncoderV1, UpdateEncoderV2, Doc, ContentType, Transaction, Item, YXmlText, YXmlHook // eslint-disable-line
 } from '../internals.js'
 
 import * as error from 'lib0/error'
@@ -253,10 +256,7 @@ export class YXmlFragment extends AbstractType {
    * @return {string} The string representation of all children.
    */
   toString () {
-    if (this.doc != null) {
-      return transact(this.doc, tr => new ListCursor(this).map(tr, xml => xml.toString()).join(''))
-    }
-    return ''
+    return typeListMap(this, xml => xml.toString()).join('')
   }
 
   /**
@@ -266,6 +266,32 @@ export class YXmlFragment extends AbstractType {
     return this.toString()
   }
 
+  /**
+   * Creates a Dom Element that mirrors this YXmlElement.
+   *
+   * @param {Document} [_document=document] The document object (you must define
+   *                                        this when calling this method in
+   *                                        nodejs)
+   * @param {Object<string, any>} [hooks={}] Optional property to customize how hooks
+   *                                             are presented in the DOM
+   * @param {any} [binding] You should not set this property. This is
+   *                               used if DomBinding wants to create a
+   *                               association to the created DOM type.
+   * @return {Node} The {@link https://developer.mozilla.org/en-US/docs/Web/API/Element|Dom Element}
+   *
+   * @public
+   */
+  toDOM (_document = document, hooks = {}, binding) {
+    const fragment = _document.createDocumentFragment()
+    if (binding !== undefined) {
+      binding._createAssociation(fragment, this)
+    }
+    typeListForEach(this, xmlType => {
+      fragment.insertBefore(xmlType.toDOM(_document, hooks, binding), null)
+    })
+    return fragment
+  }
+
   /**
    * Inserts new content at an index.
    *
@@ -278,11 +304,9 @@ export class YXmlFragment extends AbstractType {
    */
   insert (index, content) {
     if (this.doc !== null) {
-      return transact(this.doc, transaction =>
-        useSearchMarker(transaction, this, index, walker =>
-          walker.insertArrayValue(transaction, content)
-        )
-      )
+      transact(this.doc, transaction => {
+        typeListInsertGenerics(transaction, this, index, content)
+      })
     } else {
       // @ts-ignore _prelimContent is defined because this is not yet integrated
       this._prelimContent.splice(index, 0, ...content)
@@ -323,11 +347,9 @@ export class YXmlFragment extends AbstractType {
    */
   delete (index, length = 1) {
     if (this.doc !== null) {
-      transact(/** @type {Doc} */ (this.doc), transaction =>
-        useSearchMarker(transaction, this, index, walker =>
-          walker.delete(transaction, length)
-        )
-      )
+      transact(this.doc, transaction => {
+        typeListDelete(transaction, this, index, length)
+      })
     } else {
       // @ts-ignore _prelimContent is defined because this is not yet integrated
       this._prelimContent.splice(index, length)
@@ -368,11 +390,7 @@ export class YXmlFragment extends AbstractType {
    * @return {YXmlElement|YXmlText}
    */
   get (index) {
-    return transact(/** @type {Doc} */ (this.doc), transaction =>
-      useSearchMarker(transaction, this, index, walker =>
-        walker.slice(transaction, 1)[0]
-      )
-    )
+    return typeListGet(this, index)
   }
 
   /**
@@ -386,6 +404,15 @@ export class YXmlFragment extends AbstractType {
     return typeListSlice(this, start, end)
   }
 
+  /**
+   * Executes a provided function on once on overy child element.
+   *
+   * @param {function(YXmlElement|YXmlText,number, typeof self):void} f A function to execute on every element of this YArray.
+   */
+  forEach (f) {
+    typeListForEach(this, f)
+  }
+
   /**
    * Transform the properties of this type to binary and write it to an
    * BinaryEncoder.
@@ -400,10 +427,10 @@ export class YXmlFragment extends AbstractType {
 }
 
 /**
- * @param {UpdateDecoderV1 | UpdateDecoderV2} decoder
+ * @param {UpdateDecoderV1 | UpdateDecoderV2} _decoder
  * @return {YXmlFragment}
  *
  * @private
  * @function
  */
-export const readYXmlFragment = decoder => new YXmlFragment()
+export const readYXmlFragment = _decoder => new YXmlFragment()

src/types/YXmlHook.js

@@ -40,6 +40,36 @@ export class YXmlHook extends YMap {
     return el
   }
 
+  /**
+   * Creates a Dom Element that mirrors this YXmlElement.
+   *
+   * @param {Document} [_document=document] The document object (you must define
+   *                                        this when calling this method in
+   *                                        nodejs)
+   * @param {Object.<string, any>} [hooks] Optional property to customize how hooks
+   *                                             are presented in the DOM
+   * @param {any} [binding] You should not set this property. This is
+   *                               used if DomBinding wants to create a
+   *                               association to the created DOM type
+   * @return {Element} The {@link https://developer.mozilla.org/en-US/docs/Web/API/Element|Dom Element}
+   *
+   * @public
+   */
+  toDOM (_document = document, hooks = {}, binding) {
+    const hook = hooks[this.hookName]
+    let dom
+    if (hook !== undefined) {
+      dom = hook.createDom(this)
+    } else {
+      dom = document.createElement(this.hookName)
+    }
+    dom.setAttribute('data-yjs-hook', this.hookName)
+    if (binding !== undefined) {
+      binding._createAssociation(dom, this)
+    }
+    return dom
+  }
+
   /**
    * Transform the properties of this type to binary and write it to an
    * BinaryEncoder.

src/types/YXmlText.js

@@ -39,6 +39,29 @@ export class YXmlText extends YText {
     return text
   }
 
+  /**
+   * Creates a Dom Element that mirrors this YXmlText.
+   *
+   * @param {Document} [_document=document] The document object (you must define
+   *                                        this when calling this method in
+   *                                        nodejs)
+   * @param {Object<string, any>} [hooks] Optional property to customize how hooks
+   *                                             are presented in the DOM
+   * @param {any} [binding] You should not set this property. This is
+   *                               used if DomBinding wants to create a
+   *                               association to the created DOM type.
+   * @return {Text} The {@link https://developer.mozilla.org/en-US/docs/Web/API/Element|Dom Element}
+   *
+   * @public
+   */
+  toDOM (_document = document, hooks, binding) {
+    const dom = _document.createTextNode(this.toString())
+    if (binding !== undefined) {
+      binding._createAssociation(dom, this)
+    }
+    return dom
+  }
+
   toString () {
     // @ts-ignore
     return this.toDelta().map(delta => {

src/utils/Doc.js

@@ -38,7 +38,7 @@ export const generateNewClientId = random.uint32
  */
 export class Doc extends Observable {
   /**
-   * @param {DocOpts} [opts] configuration
+   * @param {DocOpts} opts configuration
    */
   constructor ({ guid = random.uuidv4(), collectionid = null, gc = true, gcFilter = () => true, meta = null, autoLoad = false, shouldLoad = true } = {}) {
     super()
@@ -48,7 +48,7 @@ export class Doc extends Observable {
     this.guid = guid
     this.collectionid = collectionid
     /**
-     * @type {Map<string, AbstractType<YEvent>>}
+     * @type {Map<string, AbstractType<YEvent<any>>>}
      */
     this.share = new Map()
     this.store = new StructStore()
@@ -72,13 +72,57 @@ export class Doc extends Observable {
     this.shouldLoad = shouldLoad
     this.autoLoad = autoLoad
     this.meta = meta
+    /**
+     * This is set to true when the persistence provider loaded the document from the database or when the `sync` event fires.
+     * Note that not all providers implement this feature. Provider authors are encouraged to fire the `load` event when the doc content is loaded from the database.
+     *
+     * @type {boolean}
+     */
     this.isLoaded = false
+    /**
+     * This is set to true when the connection provider has successfully synced with a backend.
+     * Note that when using peer-to-peer providers this event may not provide very useful.
+     * Also note that not all providers implement this feature. Provider authors are encouraged to fire
+     * the `sync` event when the doc has been synced (with `true` as a parameter) or if connection is
+     * lost (with false as a parameter).
+     */
+    this.isSynced = false
+    /**
+     * Promise that resolves once the document has been loaded from a presistence provider.
+     */
     this.whenLoaded = promise.create(resolve => {
       this.on('load', () => {
         this.isLoaded = true
         resolve(this)
       })
     })
+    const provideSyncedPromise = () => promise.create(resolve => {
+      /**
+       * @param {boolean} isSynced
+       */
+      const eventHandler = (isSynced) => {
+        if (isSynced === undefined || isSynced === true) {
+          this.off('sync', eventHandler)
+          resolve()
+        }
+      }
+      this.on('sync', eventHandler)
+    })
+    this.on('sync', isSynced => {
+      if (isSynced === false && this.isSynced) {
+        this.whenSynced = provideSyncedPromise()
+      }
+      this.isSynced = isSynced === undefined || isSynced === true
+      if (!this.isLoaded) {
+        this.emit('load', [])
+      }
+    })
+    /**
+     * Promise that resolves once the document has been synced with a backend.
+     * This promise is recreated when the connection is lost.
+     * Note the documentation about the `isSynced` property.
+     */
+    this.whenSynced = provideSyncedPromise()
   }
 
   /**

src/utils/ListCursor.js

@@ -1,740 +0,0 @@
-import * as error from 'lib0/error'
-
-import {
-  getItemCleanStart,
-  createID,
-  getMovedCoords,
-  updateMarkerChanges,
-  getState,
-  ContentAny,
-  ContentBinary,
-  ContentType,
-  ContentDoc,
-  Doc,
-  compareIDs,
-  createRelativePosition,
-  RelativePosition, ID, AbstractContent, ContentMove, Transaction, Item, AbstractType // eslint-disable-line
-} from '../internals.js'
-import { compareRelativePositions } from './RelativePosition.js'
-import * as array from 'lib0/array'
-
-const lengthExceeded = error.create('Length exceeded!')
-
-/**
- * We keep the moved-stack across several transactions. Local or remote changes can invalidate
- * "moved coords" on the moved-stack.
- *
- * The reason for this is that if assoc < 0, then getMovedCoords will return the target.right item.
- * While the computed item is on the stack, it is possible that a user inserts something between target
- * and the item on the stack. Then we expect that the newly inserted item is supposed to be on the new
- * computed item.
- *
- * @param {Transaction} tr
- * @param {ListCursor} li
- */
-const popMovedStack = (tr, li) => {
-  let { start, end, move } = li.movedStack.pop() || { start: null, end: null, move: null }
-  if (move) {
-    const moveContent = /** @type {ContentMove} */ (move.content)
-    if (
-      (
-        moveContent.start.assoc < 0 && (
-          (start === null && moveContent.start.item !== null) ||
-          (start !== null && !compareIDs(/** @type {Item} */ (start.left).lastId, moveContent.start.item))
-        )
-      ) || (
-        moveContent.end.assoc < 0 && (
-          (end === null && moveContent.end.item !== null) ||
-          (end !== null && !compareIDs(/** @type {Item} */ (end.left).lastId, moveContent.end.item))
-        )
-      )
-    ) {
-      const coords = getMovedCoords(moveContent, tr, false)
-      start = coords.start
-      end = coords.end
-    }
-  }
-  li.currMove = move
-  li.currMoveStart = start
-  li.currMoveEnd = end
-  li.reachedEnd = false
-}
-
-/**
- * Structure that helps to iterate through list-like structures. This is a useful abstraction that keeps track of move operations.
- */
-export class ListCursor {
-  /**
-   * @param {AbstractType<any>} type
-   */
-  constructor (type) {
-    this.type = type
-    /**
-     * Current index-position
-     */
-    this.index = 0
-    /**
-     * Relative position to the current item (if item.content.length > 1)
-     */
-    this.rel = 0
-    /**
-     * This refers to the current right item, unless reachedEnd is true. Then it refers to the left item.
-     *
-     * @public
-     * @type {Item | null}
-     */
-    this.nextItem = type._start
-    this.reachedEnd = type._start === null
-    /**
-     * @type {Item | null}
-     */
-    this.currMove = null
-    /**
-     * @type {Item | null}
-     */
-    this.currMoveStart = null
-    /**
-     * @type {Item | null}
-     */
-    this.currMoveEnd = null
-    /**
-     * @type {Array<{ start: Item | null, end: Item | null, move: Item }>}
-     */
-    this.movedStack = []
-  }
-
-  clone () {
-    const iter = new ListCursor(this.type)
-    iter.index = this.index
-    iter.rel = this.rel
-    iter.nextItem = this.nextItem
-    iter.reachedEnd = this.reachedEnd
-    iter.currMove = this.currMove
-    iter.currMoveStart = this.currMoveStart
-    iter.currMoveEnd = this.currMoveEnd
-    iter.movedStack = this.movedStack.slice()
-    return iter
-  }
-
-  /**
-   * @type {Item | null}
-   */
-  get left () {
-    if (this.reachedEnd) {
-      return this.nextItem
-    } else {
-      return this.nextItem && this.nextItem.left
-    }
-  }
-
-  /**
-   * @type {Item | null}
-   */
-  get right () {
-    if (this.reachedEnd) {
-      return null
-    } else {
-      return this.nextItem
-    }
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {number} index
-   */
-  moveTo (tr, index) {
-    const diff = index - this.index
-    if (diff > 0) {
-      this.forward(tr, diff, true)
-    } else if (diff < 0) {
-      this.backward(tr, -diff)
-    }
-  }
-
-  /**
-   * When using skipUncountables=false within a "useSearchMarker" call, it is recommended
-   * to move the marker to the end. @todo do this after each useSearchMarkerCall
-   *
-   * @param {Transaction} tr
-   * @param {number} len
-   * @param {boolean} skipUncountables Iterate as much as possible iterating over uncountables until we find the next item.
-   */
-  forward (tr, len, skipUncountables) {
-    if (len === 0 && this.nextItem == null) {
-      return this
-    }
-    if (this.index + len > this.type._length || this.nextItem == null) {
-      throw lengthExceeded
-    }
-    let item = /** @type {Item} */ (this.nextItem)
-    this.index += len
-    // eslint-disable-next-line no-unmodified-loop-condition
-    while ((!this.reachedEnd || this.currMove !== null) && (len > 0 || (skipUncountables && len === 0 && item && (!item.countable || item.deleted || item === this.currMoveEnd || (this.reachedEnd && this.currMoveEnd === null) || item.moved !== this.currMove)))) {
-      if (item === this.currMoveEnd || (this.currMoveEnd === null && this.reachedEnd && this.currMove)) {
-        item = /** @type {Item} */ (this.currMove) // we iterate to the right after the current condition
-        popMovedStack(tr, this)
-      } else if (item === null) {
-        error.unexpectedCase() // should never happen
-      } else if (item.countable && !item.deleted && item.moved === this.currMove && len > 0) {
-        len -= item.length
-        if (len < 0) {
-          this.rel = item.length + len
-          len = 0
-          break
-        }
-      } else if (item.content.constructor === ContentMove && item.moved === this.currMove) {
-        if (this.currMove) {
-          this.movedStack.push({ start: this.currMoveStart, end: this.currMoveEnd, move: this.currMove })
-        }
-        const { start, end } = getMovedCoords(item.content, tr, false)
-        this.currMove = item
-        this.currMoveStart = start
-        this.currMoveEnd = end
-        item = start
-        continue
-      }
-      if (this.reachedEnd) {
-        throw error.unexpectedCase
-      }
-      if (item.right) {
-        item = item.right
-      } else {
-        this.reachedEnd = true
-      }
-    }
-    this.index -= len
-    this.nextItem = item
-    return this
-  }
-
-  /**
-   * We prefer to insert content outside of a moved range.
-   * Try to escape the moved range by walking to the left over deleted items.
-   *
-   * @param {Transaction} tr
-   */
-  reduceMoveDepth (tr) {
-    let nextItem = this.nextItem
-    if (nextItem !== null) {
-      while (this.currMove) {
-        if (nextItem === this.currMoveStart) {
-          nextItem = /** @type {Item} */ (this.currMove) // we iterate to the left after the current condition
-          popMovedStack(tr, this)
-          continue
-        }
-        // check if we can iterate to the left while stepping over deleted items until we find an item === this.currMoveStart
-        /**
-         * @type {Item} nextItem
-         */
-        let item = nextItem
-        while (item.deleted && item.moved === this.currMove && item !== this.currMoveStart) {
-          item = /** @type {Item} */ (item.left) // this must exist otherwise we miscalculated the move
-        }
-        if (item === this.currMoveStart) {
-          // we only want to iterate over deleted items if we can escape a move
-          nextItem = item
-        } else {
-          break
-        }
-      }
-      this.nextItem = nextItem
-    }
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {number} len
-   * @return {ListCursor}
-   */
-  backward (tr, len) {
-    if (this.index - len < 0) {
-      throw lengthExceeded
-    }
-    this.index -= len
-    if (this.reachedEnd) {
-      const nextItem = /** @type {Item} */ (this.nextItem)
-      this.rel = nextItem.countable && !nextItem.deleted ? nextItem.length : 0
-      this.reachedEnd = false
-    }
-    if (this.rel >= len) {
-      this.rel -= len
-      return this
-    }
-    let item = this.nextItem
-    if (item && item.content.constructor === ContentMove) {
-      item = item.left
-    } else {
-      len += ((item && item.countable && !item.deleted && item.moved === this.currMove) ? item.length : 0) - this.rel
-    }
-    this.rel = 0
-    while (item && len > 0) {
-      if (item.countable && !item.deleted && item.moved === this.currMove) {
-        len -= item.length
-        if (len < 0) {
-          this.rel = -len
-          len = 0
-        }
-        if (len === 0) {
-          break
-        }
-      } else if (item.content.constructor === ContentMove && item.moved === this.currMove) {
-        if (this.currMove) {
-          this.movedStack.push({ start: this.currMoveStart, end: this.currMoveEnd, move: this.currMove })
-        }
-        const { start, end } = getMovedCoords(item.content, tr, false)
-        this.currMove = item
-        this.currMoveStart = start
-        this.currMoveEnd = end
-        item = start
-        continue
-      }
-      if (item === this.currMoveStart) {
-        item = /** @type {Item} */ (this.currMove) // we iterate to the left after the current condition
-        popMovedStack(tr, this)
-      }
-      item = item.left
-    }
-    this.nextItem = item
-    return this
-  }
-
-  /**
-   * @template {{length: number}} T
-   * @param {Transaction} tr
-   * @param {number} len
-   * @param {T} value the initial content
-   * @param {function(AbstractContent, number, number):T} slice
-   * @param {function(T, T): T} concat
-   */
-  _slice (tr, len, value, slice, concat) {
-    if (this.index + len > this.type._length) {
-      throw lengthExceeded
-    }
-    this.index += len
-    /**
-     * We store nextItem in a variable because this version cannot be null.
-     */
-    let nextItem = /** @type {Item} */ (this.nextItem)
-    while (len > 0 && !this.reachedEnd) {
-      while (nextItem.countable && !this.reachedEnd && len > 0 && nextItem !== this.currMoveEnd) {
-        if (!nextItem.deleted && nextItem.moved === this.currMove) {
-          const slicedContent = slice(nextItem.content, this.rel, len)
-          len -= slicedContent.length
-          value = concat(value, slicedContent)
-          if (this.rel + slicedContent.length === nextItem.length) {
-            this.rel = 0
-          } else {
-            this.rel += slicedContent.length
-            continue // do not iterate to item.right
-          }
-        }
-        if (nextItem.right) {
-          nextItem = nextItem.right
-        } else {
-          this.reachedEnd = true
-        }
-      }
-      if ((!this.reachedEnd || this.currMove !== null) && len > 0) {
-        // always set nextItem before any method call
-        this.nextItem = nextItem
-        this.forward(tr, 0, true)
-        nextItem = this.nextItem
-      }
-    }
-    this.nextItem = nextItem
-    if (len < 0) {
-      this.index -= len
-    }
-    return value
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {number} len
-   */
-  delete (tr, len) {
-    const startLength = len
-    const sm = this.type._searchMarker
-    let item = this.nextItem
-    if (this.index + len > this.type._length) {
-      throw lengthExceeded
-    }
-    while (len > 0) {
-      while (item && !item.deleted && item.countable && !this.reachedEnd && len > 0 && item.moved === this.currMove && item !== this.currMoveEnd) {
-        if (this.rel > 0) {
-          item = getItemCleanStart(tr, createID(item.id.client, item.id.clock + this.rel))
-          this.rel = 0
-        }
-        if (len < item.length) {
-          getItemCleanStart(tr, createID(item.id.client, item.id.clock + len))
-        }
-        len -= item.length
-        item.delete(tr)
-        if (item.right) {
-          item = item.right
-        } else {
-          this.reachedEnd = true
-        }
-      }
-      if (len > 0) {
-        this.nextItem = item
-        this.forward(tr, 0, true)
-        item = this.nextItem
-      }
-    }
-    this.nextItem = item
-    if (sm) {
-      updateMarkerChanges(sm, this.index, -startLength + len, this)
-    }
-  }
-
-  /**
-   * @param {Transaction} tr
-   */
-  _splitRel (tr) {
-    if (this.rel > 0) {
-      /**
-       * @type {ID}
-       */
-      const itemid = /** @type {Item} */ (this.nextItem).id
-      this.nextItem = getItemCleanStart(tr, createID(itemid.client, itemid.clock + this.rel))
-      this.rel = 0
-    }
-  }
-
-  /**
-   * Important: you must update markers after calling this method!
-   *
-   * @param {Transaction} tr
-   * @param {Array<AbstractContent>} content
-   */
-  insertContents (tr, content) {
-    this.reduceMoveDepth(tr)
-    this._splitRel(tr)
-    const parent = this.type
-    const store = tr.doc.store
-    const ownClientId = tr.doc.clientID
-    /**
-     * @type {Item | null}
-     */
-    const right = this.right
-
-    /**
-     * @type {Item | null}
-     */
-    let left = this.left
-    content.forEach(c => {
-      left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, c)
-      left.integrate(tr, 0)
-    })
-    if (right === null) {
-      this.nextItem = left
-      this.reachedEnd = true
-    } else {
-      this.nextItem = right
-    }
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {Array<{ start: RelativePosition, end: RelativePosition }>} ranges
-   */
-  insertMove (tr, ranges) {
-    this.insertContents(tr, ranges.map(range => new ContentMove(range.start, range.end, -1)))
-    // @todo is there a better alrogirthm to update searchmarkers? We could simply remove the markers that are in the updated range.
-    // Also note that searchmarkers are updated in insertContents as well.
-    const sm = this.type._searchMarker
-    if (sm) sm.length = 0 // @todo instead, iterate through sm and delete all marked properties on items
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {Array<Object<string,any>|Array<any>|boolean|number|null|string|Uint8Array>} values
-   */
-  insertArrayValue (tr, values) {
-    this._splitRel(tr)
-    const sm = this.type._searchMarker
-    /**
-     * @type {Array<AbstractContent>}
-     */
-    const contents = []
-    /**
-     * @type {Array<Object|Array<any>|number|null>}
-     */
-    let jsonContent = []
-    const packJsonContent = () => {
-      if (jsonContent.length > 0) {
-        contents.push(new ContentAny(jsonContent))
-        jsonContent = []
-      }
-    }
-    values.forEach(c => {
-      if (c === null) {
-        jsonContent.push(c)
-      } else {
-        switch (c.constructor) {
-          case Number:
-          case Object:
-          case Boolean:
-          case Array:
-          case String:
-            jsonContent.push(c)
-            break
-          default:
-            packJsonContent()
-            switch (c.constructor) {
-              case Uint8Array:
-              case ArrayBuffer:
-                contents.push(new ContentBinary(new Uint8Array(/** @type {Uint8Array} */ (c))))
-                break
-              case Doc:
-                contents.push(new ContentDoc(/** @type {Doc} */ (c)))
-                break
-              default:
-                if (c instanceof AbstractType) {
-                  contents.push(new ContentType(c))
-                } else {
-                  throw new Error('Unexpected content type in insert operation')
-                }
-            }
-        }
-      }
-    })
-    packJsonContent()
-    this.insertContents(tr, contents)
-    this.index += values.length
-    if (sm) {
-      updateMarkerChanges(sm, this.index - values.length, values.length, this)
-    }
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {number} len
-   */
-  slice (tr, len) {
-    return this._slice(tr, len, [], sliceArrayContent, concatArrayContent)
-  }
-
-  /**
-   * @param {Transaction} tr
-   * @param {function(any, number, any):void} f
-   */
-  forEach (tr, f) {
-    for (const val of this.values(tr)) {
-      // decrease index because retrieving value will increase index
-      f(val, this.index - 1, this.type)
-    }
-  }
-
-  /**
-   * @template T
-   * @param {Transaction} tr
-   * @param {function(any, number, any):T} f
-   * @return {Array<T>}
-   */
-  map (tr, f) {
-    const arr = new Array(this.type._length - this.index)
-    let i = 0
-    for (const val of this.values(tr)) {
-      arr[i++] = f(val, this.index - 1, this.type)
-    }
-    return arr
-  }
-
-  /**
-   * @param {Transaction} tr
-   */
-  values (tr) {
-    return {
-      [Symbol.iterator] () {
-        return this
-      },
-      next: () => {
-        if (this.reachedEnd || this.index === this.type._length) {
-          return { done: true }
-        }
-        const [value] = this.slice(tr, 1)
-        return {
-          done: false,
-          value: value
-        }
-      }
-    }
-  }
-}
-
-/**
- * @param {AbstractContent} itemcontent
- * @param {number} start
- * @param {number} len
- */
-const sliceArrayContent = (itemcontent, start, len) => {
-  const content = itemcontent.getContent()
-  return content.length <= len && start === 0 ? content : content.slice(start, start + len)
-}
-/**
- * @param {Array<any>} content
- * @param {Array<any>} added
- */
-const concatArrayContent = (content, added) => {
-  content.push(...added)
-  return content
-}
-
-/**
- * Move-ranges must not cross each other.
- *
- * This function computes the minimal amount of ranges to move a range of content to
- * a different place.
- *
- * Algorithm:
- * * Store the current stack in $preStack and $preItem = walker.nextItem
- * * Iterate forward $len items.
- * * The current stack is stored is $afterStack and $
- * * Delete the stack-items that both of them have in common
- *
- * @param {Transaction} tr
- * @param {ListCursor} walker
- * @param {number} len
- * @return {Array<{ start: RelativePosition, end: RelativePosition }>}
- */
-export const getMinimalListViewRanges = (tr, walker, len) => {
-  if (len === 0) return []
-  if (walker.index + len > walker.type._length) {
-    throw lengthExceeded
-  }
-  // stepping outside the current move-range as much as possible
-  walker.reduceMoveDepth(tr)
-
-  /**
-   * @type {Array<{ start: RelativePosition, end: RelativePosition, move: Item | null }>}
-   */
-  const ranges = []
-  // store relevant information for the beginning, before we iterate forward
-  /**
-   * @type {Array<Item>}
-   */
-  const preStack = walker.movedStack.map(si => si.move)
-  const preMove = walker.currMove
-  const preItem = /** @type {Item} */ (walker.nextItem)
-  const preRel = walker.rel
-
-  walker.forward(tr, len, false)
-
-  // store the same information for the end, after we iterate forward
-  /**
-   * @type {Array<Item>}
-   */
-  const afterStack = walker.movedStack.map(si => si.move)
-  const afterMove = walker.currMove
-  /**
-  const nextIsCurrMoveStart = walker.nextItem === walker.currMoveStart
-    const afterItem = /** @type {Item} / (nextIsCurrMoveStart
-    ? walker.currMove
-    : (walker.rel > 0 || walker.reachedEnd)
-        ? walker.nextItem
-        : /** @type {Item} / (walker.nextItem).left
-  ) */
-  const afterItem = /** @type {Item} */ (
-    (walker.rel > 0 || walker.reachedEnd)
-      ? walker.nextItem
-      : /** @type {Item} */ (walker.nextItem).left
-  )
-  /**
-   * afterRel is always > 0
-   */
-  const afterRel = walker.rel > 0
-    ? walker.rel
-    : afterItem.length
-
-  walker.forward(tr, 0, false) // @todo remove once this is done is useSearchMarker
-
-  let start = createRelativePosition(walker.type, createID(preItem.id.client, preItem.id.clock + preRel), 0)
-  let end = createRelativePosition(
-    walker.type,
-    createID(afterItem.id.client, afterItem.id.clock + afterRel - 1),
-    -1
-  )
-
-  if (preMove) {
-    preStack.push(preMove)
-  }
-  if (afterMove) {
-    afterStack.push(afterMove)
-  }
-
-  // remove common stack-items
-  while (preStack.length > 0 && preStack[0] === afterStack[0]) {
-    preStack.shift()
-    afterStack.shift()
-  }
-  const topLevelMove = preStack.length > 0 ? preStack[0].moved : (afterStack.length > 0 ? afterStack[0].moved : null)
-
-  // remove stack-items that are useless for our computation (that wouldn't produce meaningful ranges)
-  // @todo
-
-  while (preStack.length > 0) {
-    const move = /** @type {Item} */ (preStack.pop())
-    ranges.push({
-      start,
-      end: /** @type {ContentMove} */ (move.content).end,
-      move
-    })
-    start = createRelativePosition(walker.type, createID(move.id.client, move.id.clock), -1)
-  }
-
-  const middleMove = { start, end, move: topLevelMove }
-  ranges.push(middleMove)
-
-  while (afterStack.length > 0) {
-    const move = /** @type {Item} */ (afterStack.pop())
-    ranges.push({
-      start: /** @type {ContentMove} */ (move.content).start,
-      end,
-      move
-    })
-    end = createRelativePosition(walker.type, createID(move.id.client, move.id.clock), 0)
-  }
-
-  // Update end of the center move operation
-  // Move ranges must be applied in order
-  middleMove.end = end
-
-  const normalizedRanges = array.flatten(ranges.map(range => {
-    // A subset of a range could be moved by another move with a higher priority.
-    // If that is the case, we need to ignore those moved items.
-    const { start, end } = getMovedCoords(range, tr, false)
-    const move = range.move
-    const ranges = []
-    /**
-     * @type {RelativePosition | null}
-     */
-    let rangeStart = range.start
-    /**
-     * @type {Item}
-     */
-    let item = start
-    while (item !== end) {
-      if (item.moved !== move && rangeStart != null) {
-        ranges.push({ start: rangeStart, end: createRelativePosition(walker.type, createID(item.id.client, item.id.clock), 0) })
-        rangeStart = null
-      }
-      if (item.moved === move && rangeStart === null) {
-        // @todo It might be better to set this to item.left, with assoc -1
-        rangeStart = createRelativePosition(walker.type, createID(item.id.client, item.id.clock), 0)
-      }
-      item = /** @type {Item} */ (item.right)
-    }
-    if (rangeStart != null) {
-      ranges.push({
-        start: rangeStart,
-        end: range.end
-      })
-    }
-    return ranges
-  }))
-
-  // filter out unnecessary ranges
-  return normalizedRanges.filter(range => !compareRelativePositions(range.start, range.end))
-}

src/utils/PermanentUserData.js

@@ -71,7 +71,7 @@ export class PermanentUserData {
    * @param {Doc} doc
    * @param {number} clientid
    * @param {string} userDescription
-   * @param {Object} [conf]
+   * @param {Object} conf
    * @param {function(Transaction, DeleteSet):boolean} [conf.filter]
    */
   setUserMapping (doc, clientid, userDescription, { filter = () => true } = {}) {
@@ -84,7 +84,7 @@ export class PermanentUserData {
       users.set(userDescription, user)
     }
     user.get('ids').push([clientid])
-    users.observe(event => {
+    users.observe(_event => {
       setTimeout(() => {
         const userOverwrite = users.get(userDescription)
         if (userOverwrite !== user) {

src/utils/RelativePosition.js

@@ -9,8 +9,6 @@ import {
   createID,
   ContentType,
   followRedone,
-  transact,
-  useSearchMarker,
   ID, Doc, AbstractType // eslint-disable-line
 } from '../internals.js'
 
@@ -75,10 +73,6 @@ export class RelativePosition {
      */
     this.assoc = assoc
   }
-
-  clone () {
-    return new RelativePosition(this.type, this.tname, this.item, this.assoc)
-  }
 }
 
 /**
@@ -167,6 +161,7 @@ export const createRelativePosition = (type, item, assoc) => {
  * @function
  */
 export const createRelativePositionFromTypeIndex = (type, index, assoc = 0) => {
+  let t = type._start
   if (assoc < 0) {
     // associated to the left character or the beginning of a type, increment index if possible.
     if (index === 0) {
@@ -174,17 +169,21 @@ export const createRelativePositionFromTypeIndex = (type, index, assoc = 0) => {
     }
     index--
   }
-  return transact(/** @type {Doc} */ (type.doc), tr =>
-    useSearchMarker(tr, type, index, walker => {
-      if (walker.reachedEnd) {
-        const item = assoc < 0 ? /** @type {Item} */ (walker.nextItem).lastId : null
-        return createRelativePosition(type, item, assoc)
-      } else {
-        const id = /** @type {Item} */ (walker.nextItem).id
-        return createRelativePosition(type, createID(id.client, id.clock + walker.rel), assoc)
+  while (t !== null) {
+    if (!t.deleted && t.countable) {
+      if (t.length > index) {
+        // case 1: found position somewhere in the linked list
+        return createRelativePosition(type, createID(t.id.client, t.id.clock + index), assoc)
       }
-    })
-  )
+      index -= t.length
+    }
+    if (t.right === null && assoc < 0) {
+      // left-associated position, return last available id
+      return createRelativePosition(type, t.lastId, assoc)
+    }
+    t = t.right
+  }
+  return createRelativePosition(type, null, assoc)
 }
 
 /**
@@ -196,7 +195,7 @@ export const createRelativePositionFromTypeIndex = (type, index, assoc = 0) => {
 export const writeRelativePosition = (encoder, rpos) => {
   const { type, tname, item, assoc } = rpos
   if (item !== null) {
-    encoding.writeUint8(encoder, 0)
+    encoding.writeVarUint(encoder, 0)
     writeID(encoder, item)
   } else if (tname !== null) {
     // case 2: found position at the end of the list and type is stored in y.share
@@ -233,7 +232,7 @@ export const readRelativePosition = decoder => {
   let type = null
   let tname = null
   let itemID = null
-  switch (decoding.readUint8(decoder)) {
+  switch (decoding.readVarUint(decoder)) {
     case 0:
       // case 1: found position somewhere in the linked list
       itemID = readID(decoder)

src/utils/StructStore.js

@@ -199,18 +199,19 @@ export const getItemCleanStart = (transaction, id) => {
  * Expects that id is actually in store. This function throws or is an infinite loop otherwise.
  *
  * @param {Transaction} transaction
+ * @param {StructStore} store
  * @param {ID} id
  * @return {Item}
  *
  * @private
  * @function
  */
-export const getItemCleanEnd = (transaction, id) => {
+export const getItemCleanEnd = (transaction, store, id) => {
   /**
    * @type {Array<Item>}
    */
   // @ts-ignore
-  const structs = transaction.doc.store.clients.get(id.client)
+  const structs = store.clients.get(id.client)
   const index = findIndexSS(structs, id.clock)
   const struct = structs[index]
   if (id.clock !== struct.id.clock + struct.length - 1 && struct.constructor !== GC) {

src/utils/Transaction.js

@@ -14,7 +14,6 @@ import {
   UpdateEncoderV1, UpdateEncoderV2, GC, StructStore, AbstractType, AbstractStruct, YEvent, Doc // eslint-disable-line
 } from '../internals.js'
 
-import * as encoding from 'lib0/encoding'
 import * as map from 'lib0/map'
 import * as math from 'lib0/math'
 import * as set from 'lib0/set'
@@ -62,13 +61,6 @@ export class Transaction {
      * @type {DeleteSet}
      */
     this.deleteSet = new DeleteSet()
-    /**
-     * These deletes were used to cleanup the document and
-     * should be broadcasted again using a different transaction-origin.
-     *
-     * @type {DeleteSet}
-     */
-    this.cleanupDeletions = new DeleteSet()
     /**
      * Holds the state before the transaction started.
      * @type {Map<Number,Number>}
@@ -83,13 +75,13 @@ export class Transaction {
      * All types that were directly modified (property added or child
      * inserted/deleted). New types are not included in this Set.
      * Maps from type to parentSubs (`item.parentSub = null` for YArray)
-     * @type {Map<AbstractType<YEvent>,Set<String|null>>}
+     * @type {Map<AbstractType<YEvent<any>>,Set<String|null>>}
      */
     this.changed = new Map()
     /**
      * Stores the events for the types that observe also child elements.
      * It is mainly used by `observeDeep`.
-     * @type {Map<AbstractType<YEvent>,Array<YEvent>>}
+     * @type {Map<AbstractType<YEvent<any>>,Array<YEvent<any>>>}
      */
     this.changedParentTypes = new Map()
     /**
@@ -122,14 +114,6 @@ export class Transaction {
      * @type {Set<Doc>}
      */
     this.subdocsLoaded = new Set()
-    /**
-     * We store the reference that last moved an item.
-     * This is needed to compute the delta when multiple ContentMove move
-     * the same item.
-     *
-     * @type {Map<Item, Item>}
-     */
-    this.prevMoved = new Map()
   }
 }
 
@@ -148,18 +132,6 @@ export const writeUpdateMessageFromTransaction = (encoder, transaction) => {
   return true
 }
 
-/**
- * @param {UpdateEncoderV1 | UpdateEncoderV2} encoder
- * @param {Transaction} transaction
- */
-export const writeCleanupMessageFromTransaction = (encoder, transaction) => {
-  const ds = transaction.cleanupDeletions
-  sortAndMergeDeleteSet(ds)
-  // write structs: 0 structs were created
-  encoding.writeVarUint(encoder.restEncoder, 0)
-  writeDeleteSet(encoder, ds)
-}
-
 /**
  * @param {Transaction} transaction
  *
@@ -176,7 +148,7 @@ export const nextID = transaction => {
  * did not change, it was just added and we should not fire events for `type`.
  *
  * @param {Transaction} transaction
- * @param {AbstractType<YEvent>} type
+ * @param {AbstractType<YEvent<any>>} type
  * @param {string|null} parentSub
  */
 export const addChangedTypeToTransaction = (transaction, type, parentSub) => {
@@ -279,7 +251,6 @@ const cleanupTransactions = (transactionCleanups, i) => {
     try {
       sortAndMergeDeleteSet(ds)
       transaction.afterState = getStateVector(transaction.doc.store)
-      doc._transaction = null
       doc.emit('beforeObserverCalls', [transaction, doc])
       /**
        * An array of event callbacks.
@@ -364,17 +335,11 @@ const cleanupTransactions = (transactionCleanups, i) => {
       }
       // @todo Merge all the transactions into one and provide send the data as a single update message
       doc.emit('afterTransactionCleanup', [transaction, doc])
-      const needsCleanupEvent = transaction.cleanupDeletions.clients.size > 0
       if (doc._observers.has('update')) {
         const encoder = new UpdateEncoderV1()
         const hasContent = writeUpdateMessageFromTransaction(encoder, transaction)
         if (hasContent) {
           doc.emit('update', [encoder.toUint8Array(), transaction.origin, doc, transaction])
-          if (needsCleanupEvent) {
-            const encoder = new UpdateEncoderV1()
-            writeCleanupMessageFromTransaction(encoder, transaction)
-            doc.emit('update', [encoder.toUint8Array(), 'cleanup', doc, transaction])
-          }
         }
       }
       if (doc._observers.has('updateV2')) {
@@ -382,11 +347,6 @@ const cleanupTransactions = (transactionCleanups, i) => {
         const hasContent = writeUpdateMessageFromTransaction(encoder, transaction)
         if (hasContent) {
           doc.emit('updateV2', [encoder.toUint8Array(), transaction.origin, doc, transaction])
-          if (needsCleanupEvent) {
-            const encoder = new UpdateEncoderV2()
-            writeCleanupMessageFromTransaction(encoder, transaction)
-            doc.emit('updateV2', [encoder.toUint8Array(), 'cleanup', doc, transaction])
-          }
         }
       }
       const { subdocsAdded, subdocsLoaded, subdocsRemoved } = transaction
@@ -416,12 +376,9 @@ const cleanupTransactions = (transactionCleanups, i) => {
 /**
  * Implements the functionality of `y.transact(()=>{..})`
  *
- * @template T
- *
  * @param {Doc} doc
- * @param {function(Transaction):T} f
+ * @param {function(Transaction):void} f
  * @param {any} [origin=true]
- * @return {T}
  *
  * @function
  */
@@ -437,28 +394,23 @@ export const transact = (doc, f, origin = null, local = true) => {
     }
     doc.emit('beforeTransaction', [doc._transaction, doc])
   }
-  let res
   try {
-    res = f(doc._transaction)
+    f(doc._transaction)
   } finally {
-    if (initialCall && transactionCleanups[0] === doc._transaction) {
-      // The first transaction ended, now process observer calls.
-      // Observer call may create new transactions for which we need to call the observers and do cleanup.
-      // We don't want to nest these calls, so we execute these calls one after
-      // another.
-      // Also we need to ensure that all cleanups are called, even if the
-      // observes throw errors.
-      // This file is full of hacky try {} finally {} blocks to ensure that an
-      // event can throw errors and also that the cleanup is called.
-      cleanupTransactions(transactionCleanups, 0)
+    if (initialCall) {
+      const finishCleanup = doc._transaction === transactionCleanups[0]
+      doc._transaction = null
+      if (finishCleanup) {
+        // The first transaction ended, now process observer calls.
+        // Observer call may create new transactions for which we need to call the observers and do cleanup.
+        // We don't want to nest these calls, so we execute these calls one after
+        // another.
+        // Also we need to ensure that all cleanups are called, even if the
+        // observes throw errors.
+        // This file is full of hacky try {} finally {} blocks to ensure that an
+        // event can throw errors and also that the cleanup is called.
+        cleanupTransactions(transactionCleanups, 0)
+      }
     }
   }
-  return res
 }
-
-/**
- * @param {Transaction} tr
- * @param {AbstractStruct} struct
- */
-export const addsStruct = (tr, struct) =>
-  struct.id.clock >= (tr.beforeState.get(struct.id.client) || 0)

src/utils/UndoManager.js

@@ -14,6 +14,7 @@ import {
 } from '../internals.js'
 
 import * as time from 'lib0/time'
+import * as array from 'lib0/array'
 import { Observable } from 'lib0/observable'
 
 class StackItem {
@@ -30,6 +31,18 @@ class StackItem {
     this.meta = new Map()
   }
 }
+/**
+ * @param {Transaction} tr
+ * @param {UndoManager} um
+ * @param {StackItem} stackItem
+ */
+const clearUndoManagerStackItem = (tr, um, stackItem) => {
+  iterateDeletedStructs(tr, stackItem.deletions, item => {
+    if (item instanceof Item && um.scope.some(type => isParentOf(type, item))) {
+      keepItem(item, false)
+    }
+  })
+}
 
 /**
  * @param {UndoManager} undoManager
@@ -88,7 +101,7 @@ const popStackItem = (undoManager, stack, eventType) => {
         }
       })
       itemsToRedo.forEach(struct => {
-        performedChange = redoItem(transaction, struct, itemsToRedo, itemsToDelete) !== null || performedChange
+        performedChange = redoItem(transaction, struct, itemsToRedo, stackItem.insertions, undoManager.ignoreRemoteMapChanges) !== null || performedChange
       })
       // We want to delete in reverse order so that children are deleted before
       // parents, so we have more information available when items are filtered.
@@ -119,11 +132,14 @@ const popStackItem = (undoManager, stack, eventType) => {
 /**
  * @typedef {Object} UndoManagerOptions
  * @property {number} [UndoManagerOptions.captureTimeout=500]
+ * @property {function(Transaction):boolean} [UndoManagerOptions.captureTransaction] Do not capture changes of a Transaction if result false.
  * @property {function(Item):boolean} [UndoManagerOptions.deleteFilter=()=>true] Sometimes
- * it is necessary to filter whan an Undo/Redo operation can delete. If this
+ * it is necessary to filter what an Undo/Redo operation can delete. If this
  * filter returns false, the type/item won't be deleted even it is in the
  * undo/redo scope.
  * @property {Set<any>} [UndoManagerOptions.trackedOrigins=new Set([null])]
+ * @property {boolean} [ignoreRemoteMapChanges] Experimental. By default, the UndoManager will never overwrite remote changes. Enable this property to enable overwriting remote changes on key-value changes (Y.Map, properties on Y.Xml, etc..).
+ * @property {Doc} [doc] The document that this UndoManager operates on. Only needed if typeScope is empty.
  */
 
 /**
@@ -133,19 +149,31 @@ const popStackItem = (undoManager, stack, eventType) => {
  * Fires 'stack-item-popped' event when a stack item was popped from either the
  * undo- or the redo-stack. You may restore the saved stack information from `event.stackItem.meta`.
  *
- * @extends {Observable<'stack-item-added'|'stack-item-popped'>}
+ * @extends {Observable<'stack-item-added'|'stack-item-popped'|'stack-cleared'|'stack-item-updated'>}
  */
 export class UndoManager extends Observable {
   /**
    * @param {AbstractType<any>|Array<AbstractType<any>>} typeScope Accepts either a single type, or an array of types
    * @param {UndoManagerOptions} options
    */
-  constructor (typeScope, { captureTimeout = 500, deleteFilter = () => true, trackedOrigins = new Set([null]) } = {}) {
+  constructor (typeScope, {
+    captureTimeout = 500,
+    captureTransaction = tr => true,
+    deleteFilter = () => true,
+    trackedOrigins = new Set([null]),
+    ignoreRemoteMapChanges = false,
+    doc = /** @type {Doc} */ (array.isArray(typeScope) ? typeScope[0].doc : typeScope.doc)
+  } = {}) {
     super()
-    this.scope = typeScope instanceof Array ? typeScope : [typeScope]
+    /**
+     * @type {Array<AbstractType<any>>}
+     */
+    this.scope = []
+    this.addToScope(typeScope)
     this.deleteFilter = deleteFilter
     trackedOrigins.add(this)
     this.trackedOrigins = trackedOrigins
+    this.captureTransaction = captureTransaction
     /**
      * @type {Array<StackItem>}
      */
@@ -161,11 +189,20 @@ export class UndoManager extends Observable {
      */
     this.undoing = false
     this.redoing = false
-    this.doc = /** @type {Doc} */ (this.scope[0].doc)
+    this.doc = doc
     this.lastChange = 0
-    this.doc.on('afterTransaction', /** @param {Transaction} transaction */ transaction => {
+    this.ignoreRemoteMapChanges = ignoreRemoteMapChanges
+    this.captureTimeout = captureTimeout
+    /**
+     * @param {Transaction} transaction
+     */
+    this.afterTransactionHandler = transaction => {
       // Only track certain transactions
-      if (!this.scope.some(type => transaction.changedParentTypes.has(type)) || (!this.trackedOrigins.has(transaction.origin) && (!transaction.origin || !this.trackedOrigins.has(transaction.origin.constructor)))) {
+      if (
+        !this.captureTransaction(transaction) ||
+        !this.scope.some(type => transaction.changedParentTypes.has(type)) ||
+        (!this.trackedOrigins.has(transaction.origin) && (!transaction.origin || !this.trackedOrigins.has(transaction.origin.constructor)))
+      ) {
         return
       }
       const undoing = this.undoing
@@ -175,7 +212,7 @@ export class UndoManager extends Observable {
         this.stopCapturing() // next undo should not be appended to last stack item
       } else if (!redoing) {
         // neither undoing nor redoing: delete redoStack
-        this.redoStack = []
+        this.clear(false, true)
       }
       const insertions = new DeleteSet()
       transaction.afterState.forEach((endClock, client) => {
@@ -186,7 +223,8 @@ export class UndoManager extends Observable {
         }
       })
       const now = time.getUnixTime()
-      if (now - this.lastChange < captureTimeout && stack.length > 0 && !undoing && !redoing) {
+      let didAdd = false
+      if (this.lastChange > 0 && now - this.lastChange < this.captureTimeout && stack.length > 0 && !undoing && !redoing) {
         // append change to last stack op
         const lastOp = stack[stack.length - 1]
         lastOp.deletions = mergeDeleteSets([lastOp.deletions, transaction.deleteSet])
@@ -194,6 +232,7 @@ export class UndoManager extends Observable {
       } else {
         // create a new stack op
         stack.push(new StackItem(transaction.deleteSet, insertions))
+        didAdd = true
       }
       if (!undoing && !redoing) {
         this.lastChange = now
@@ -204,27 +243,59 @@ export class UndoManager extends Observable {
           keepItem(item, true)
         }
       })
-      this.emit('stack-item-added', [{ stackItem: stack[stack.length - 1], origin: transaction.origin, type: undoing ? 'redo' : 'undo', changedParentTypes: transaction.changedParentTypes }, this])
+      const changeEvent = [{ stackItem: stack[stack.length - 1], origin: transaction.origin, type: undoing ? 'redo' : 'undo', changedParentTypes: transaction.changedParentTypes }, this]
+      if (didAdd) {
+        this.emit('stack-item-added', changeEvent)
+      } else {
+        this.emit('stack-item-updated', changeEvent)
+      }
+    }
+    this.doc.on('afterTransaction', this.afterTransactionHandler)
+    this.doc.on('destroy', () => {
+      this.destroy()
     })
   }
 
-  clear () {
-    this.doc.transact(transaction => {
-      /**
-       * @param {StackItem} stackItem
-       */
-      const clearItem = stackItem => {
-        iterateDeletedStructs(transaction, stackItem.deletions, item => {
-          if (item instanceof Item && this.scope.some(type => isParentOf(type, item))) {
-            keepItem(item, false)
-          }
-        })
+  /**
+   * @param {Array<AbstractType<any>> | AbstractType<any>} ytypes
+   */
+  addToScope (ytypes) {
+    ytypes = array.isArray(ytypes) ? ytypes : [ytypes]
+    ytypes.forEach(ytype => {
+      if (this.scope.every(yt => yt !== ytype)) {
+        this.scope.push(ytype)
       }
-      this.undoStack.forEach(clearItem)
-      this.redoStack.forEach(clearItem)
     })
-    this.undoStack = []
-    this.redoStack = []
+  }
+
+  /**
+   * @param {any} origin
+   */
+  addTrackedOrigin (origin) {
+    this.trackedOrigins.add(origin)
+  }
+
+  /**
+   * @param {any} origin
+   */
+  removeTrackedOrigin (origin) {
+    this.trackedOrigins.delete(origin)
+  }
+
+  clear (clearUndoStack = true, clearRedoStack = true) {
+    if ((clearUndoStack && this.canUndo()) || (clearRedoStack && this.canRedo())) {
+      this.doc.transact(tr => {
+        if (clearUndoStack) {
+          this.undoStack.forEach(item => clearUndoManagerStackItem(tr, this, item))
+          this.undoStack = []
+        }
+        if (clearRedoStack) {
+          this.redoStack.forEach(item => clearUndoManagerStackItem(tr, this, item))
+          this.redoStack = []
+        }
+        this.emit('stack-cleared', [{ undoStackCleared: clearUndoStack, redoStackCleared: clearRedoStack }])
+      })
+    }
   }
 
   /**
@@ -282,4 +353,28 @@ export class UndoManager extends Observable {
     }
     return res
   }
+
+  /**
+   * Are undo steps available?
+   *
+   * @return {boolean} `true` if undo is possible
+   */
+  canUndo () {
+    return this.undoStack.length > 0
+  }
+
+  /**
+   * Are redo steps available?
+   *
+   * @return {boolean} `true` if redo is possible
+   */
+  canRedo () {
+    return this.redoStack.length > 0
+  }
+
+  destroy () {
+    this.trackedOrigins.delete(this)
+    this.doc.off('afterTransaction', this.afterTransactionHandler)
+    super.destroy()
+  }
 }

src/utils/UpdateEncoder.js

@@ -298,10 +298,24 @@ export class UpdateEncoderV2 extends DSEncoderV2 {
   writeKey (key) {
     const clock = this.keyMap.get(key)
     if (clock === undefined) {
+      /**
+       * @todo uncomment to introduce this feature finally
+       *
+       * Background. The ContentFormat object was always encoded using writeKey, but the decoder used to use readString.
+       * Furthermore, I forgot to set the keyclock. So everything was working fine.
+       *
+       * However, this feature here is basically useless as it is not being used (it actually only consumes extra memory).
+       *
+       * I don't know yet how to reintroduce this feature..
+       *
+       * Older clients won't be able to read updates when we reintroduce this feature. So this should probably be done using a flag.
+       *
+       */
+      // this.keyMap.set(key, this.keyClock)
       this.keyClockEncoder.write(this.keyClock++)
       this.stringEncoder.write(key)
     } else {
-      this.keyClockEncoder.write(this.keyClock++)
+      this.keyClockEncoder.write(clock)
     }
   }
 }

src/utils/YEvent.js

@@ -1,27 +1,25 @@
 
 import {
   isDeleted,
-  getMovedCoords,
-  ContentMove, Item, AbstractType, Transaction, AbstractStruct // eslint-disable-line
+  Item, AbstractType, Transaction, AbstractStruct // eslint-disable-line
 } from '../internals.js'
 
 import * as set from 'lib0/set'
 import * as array from 'lib0/array'
-import { addsStruct } from './Transaction.js'
-import { ListCursor } from './ListCursor.js'
 
 /**
+ * @template {AbstractType<any>} T
  * YEvent describes the changes on a YType.
  */
 export class YEvent {
   /**
-   * @param {AbstractType<any>} target The changed type.
+   * @param {T} target The changed type.
    * @param {Transaction} transaction
    */
   constructor (target, transaction) {
     /**
      * The type on which this event was created on.
-     * @type {AbstractType<any>}
+     * @type {T}
      */
     this.target = target
     /**
@@ -63,7 +61,7 @@ export class YEvent {
    */
   get path () {
     // @ts-ignore _item is defined because target is integrated
-    return getPathTo(this.currentTarget, this.target, this.transaction)
+    return getPathTo(this.currentTarget, this.target)
   }
 
   /**
@@ -143,13 +141,11 @@ export class YEvent {
    *
    * In contrast to change.deleted, this method also returns true if the struct was added and then deleted.
    *
-   * @todo this can be removed in the next release (prefer function)
-   *
    * @param {AbstractStruct} struct
    * @return {boolean}
    */
   adds (struct) {
-    return addsStruct(this.transaction, struct)
+    return struct.id.clock >= (this.transaction.beforeState.get(struct.id.client) || 0)
   }
 
   /**
@@ -158,129 +154,62 @@ export class YEvent {
   get changes () {
     let changes = this._changes
     if (changes === null) {
-      this.transaction.doc.transact(tr => {
-        const target = this.target
-        const added = set.create()
-        const deleted = set.create()
+      const target = this.target
+      const added = set.create()
+      const deleted = set.create()
+      /**
+       * @type {Array<{insert:Array<any>}|{delete:number}|{retain:number}>}
+       */
+      const delta = []
+      changes = {
+        added,
+        deleted,
+        delta,
+        keys: this.keys
+      }
+      const changed = /** @type Set<string|null> */ (this.transaction.changed.get(target))
+      if (changed.has(null)) {
         /**
-         * @type {Array<{insert:Array<any>}|{delete:number}|{retain:number}>}
+         * @type {any}
          */
-        const delta = []
-        changes = {
-          added,
-          deleted,
-          delta,
-          keys: this.keys
+        let lastOp = null
+        const packOp = () => {
+          if (lastOp) {
+            delta.push(lastOp)
+          }
         }
-        const changed = /** @type Set<string|null> */ (this.transaction.changed.get(target))
-        if (changed.has(null)) {
-          /**
-           * @type {Array<{ end: Item | null, move: Item | null, isNew: boolean, isDeleted: boolean }>}
-           */
-          const movedStack = []
-          /**
-           * @type {Item | null}
-           */
-          let currMove = null
-          /**
-           * @type {boolean}
-           */
-          let currMoveIsNew = false
-          /**
-           * @type {boolean}
-           */
-          let currMoveIsDeleted = false
-          /**
-           * @type {Item | null}
-           */
-          let currMoveEnd = null
-          /**
-           * @type {any}
-           */
-          let lastOp = null
-          const packOp = () => {
-            if (lastOp) {
-              delta.push(lastOp)
-            }
-          }
-          /**
-           * @param {Item} item
-           */
-          const isMovedByNew = item => {
-            let moved = item.moved
-            while (moved != null) {
-              if (this.adds(moved)) {
-                return true
+        for (let item = target._start; item !== null; item = item.right) {
+          if (item.deleted) {
+            if (this.deletes(item) && !this.adds(item)) {
+              if (lastOp === null || lastOp.delete === undefined) {
+                packOp()
+                lastOp = { delete: 0 }
               }
-              moved = moved.moved
-            }
-            return false
-          }
-          for (let item = target._start; ;) {
-            if (item === currMoveEnd && currMove) {
-              item = currMove
-              const { end, move, isNew, isDeleted } = movedStack.pop() || { end: null, move: null, isNew: false, isDeleted: false }
-              currMoveIsNew = isNew
-              currMoveIsDeleted = isDeleted
-              currMoveEnd = end
-              currMove = move
-            } else if (item === null) {
-              break
-            } else if (item.content.constructor === ContentMove) {
-              if (item.moved === currMove && (!item.deleted || (this.deletes(item) && !this.adds(item)))) {
-                movedStack.push({ end: currMoveEnd, move: currMove, isNew: currMoveIsNew, isDeleted: currMoveIsDeleted })
-                const { start, end } = getMovedCoords(item.content, tr, true) // We must split items for move-ranges, for single moves no splitting suffices
-                currMove = item
-                currMoveEnd = end
-                currMoveIsNew = this.adds(item) || currMoveIsNew
-                currMoveIsDeleted = item.deleted || currMoveIsDeleted
-                item = start
-                continue // do not move to item.right
-              }
-            } else if (item.moved !== currMove) {
-              if (!currMoveIsNew && item.countable && (!item.deleted || this.deletes(item)) && !this.adds(item) && (item.moved === null || isMovedByNew(item) || currMoveIsDeleted) && (this.transaction.prevMoved.get(item) || null) === currMove) {
-                if (lastOp === null || lastOp.delete === undefined) {
-                  packOp()
-                  lastOp = { delete: 0 }
-                }
-                lastOp.delete += item.length
-              }
-            } else if (item.deleted) {
-              if (!currMoveIsNew && this.deletes(item) && !this.adds(item) && !this.transaction.prevMoved.has(item)) {
-                if (lastOp === null || lastOp.delete === undefined) {
-                  packOp()
-                  lastOp = { delete: 0 }
-                }
-                lastOp.delete += item.length
-                deleted.add(item)
+              lastOp.delete += item.length
+              deleted.add(item)
+            } // else nop
+          } else {
+            if (this.adds(item)) {
+              if (lastOp === null || lastOp.insert === undefined) {
+                packOp()
+                lastOp = { insert: [] }
               }
+              lastOp.insert = lastOp.insert.concat(item.content.getContent())
+              added.add(item)
             } else {
-              if (currMoveIsNew || this.adds(item) || this.transaction.prevMoved.has(item)) {
-                if (lastOp === null || lastOp.insert === undefined) {
-                  packOp()
-                  lastOp = { insert: [] }
-                }
-                // @todo push items instead (or splice..)
-                lastOp.insert = lastOp.insert.concat(item.content.getContent())
-                if (!currMoveIsNew) {
-                  added.add(item)
-                }
-              } else {
-                if (lastOp === null || lastOp.retain === undefined) {
-                  packOp()
-                  lastOp = { retain: 0 }
-                }
-                lastOp.retain += item.length
+              if (lastOp === null || lastOp.retain === undefined) {
+                packOp()
+                lastOp = { retain: 0 }
               }
+              lastOp.retain += item.length
             }
-            item = /** @type {Item} */ (item).right
-          }
-          if (lastOp !== null && lastOp.retain == null) {
-            packOp()
           }
         }
-        this._changes = changes
-      })
+        if (lastOp !== null && lastOp.retain === undefined) {
+          packOp()
+        }
+      }
+      this._changes = changes
     }
     return /** @type {any} */ (changes)
   }
@@ -298,13 +227,12 @@ export class YEvent {
  *
  * @param {AbstractType<any>} parent
  * @param {AbstractType<any>} child target
- * @param {Transaction} tr
  * @return {Array<string|number>} Path to the target
  *
  * @private
  * @function
  */
-const getPathTo = (parent, child, tr) => {
+const getPathTo = (parent, child) => {
   const path = []
   while (child._item !== null && child !== parent) {
     if (child._item.parentSub !== null) {
@@ -312,11 +240,15 @@ const getPathTo = (parent, child, tr) => {
       path.unshift(child._item.parentSub)
     } else {
       // parent is array-ish
-      const c = new ListCursor(/** @type {AbstractType<any>} */ (child._item.parent))
-      while (c.nextItem != null && !c.reachedEnd && c.nextItem !== child._item) {
-        c.forward(tr, (c.nextItem.countable && !c.nextItem.deleted) ? c.nextItem.length : 0, true)
+      let i = 0
+      let c = /** @type {AbstractType<any>} */ (child._item.parent)._start
+      while (c !== child._item && c !== null) {
+        if (!c.deleted) {
+          i++
+        }
+        c = c.right
       }
-      path.unshift(c.index)
+      path.unshift(i)
     }
     child = /** @type {AbstractType<any>} */ (child._item.parent)
   }

src/utils/encoding.js

@@ -193,6 +193,7 @@ export const readClientsStructRefs = (decoder, doc) => {
         }
       }
     }
+    // console.log('time to read: ', performance.now() - start) // @todo remove
   }
   return clientRefs
 }
@@ -388,6 +389,10 @@ export const readUpdateV2 = (decoder, ydoc, transactionOrigin, structDecoder = n
     const store = doc.store
     // let start = performance.now()
     const ss = readClientsStructRefs(structDecoder, doc)
+    // console.log('time to read structs: ', performance.now() - start) // @todo remove
+    // start = performance.now()
+    // console.log('time to merge: ', performance.now() - start) // @todo remove
+    // start = performance.now()
     const restStructs = integrateStructs(transaction, store, ss)
     const pending = store.pendingStructs
     if (pending) {
@@ -411,6 +416,8 @@ export const readUpdateV2 = (decoder, ydoc, transactionOrigin, structDecoder = n
     } else {
       store.pendingStructs = restStructs
     }
+    // console.log('time to integrate: ', performance.now() - start) // @todo remove
+    // start = performance.now()
     const dsRest = readAndApplyDeleteSet(structDecoder, transaction, store)
     if (store.pendingDs) {
       // @todo we could make a lower-bound state-vector check as we do above
@@ -430,6 +437,11 @@ export const readUpdateV2 = (decoder, ydoc, transactionOrigin, structDecoder = n
       // Either dsRest == null && pendingDs == null OR dsRest != null
       store.pendingDs = dsRest
     }
+    // console.log('time to cleanup: ', performance.now() - start) // @todo remove
+    // start = performance.now()
+
+    // console.log('time to resume delete readers: ', performance.now() - start) // @todo remove
+    // start = performance.now()
     if (retry) {
       const update = /** @type {{update: Uint8Array}} */ (store.pendingStructs).update
       store.pendingStructs = null

src/utils/updates.js

@@ -112,6 +112,30 @@ export const logUpdateV2 = (update, YDecoder = UpdateDecoderV2) => {
   logging.print('DeleteSet: ', ds)
 }
 
+/**
+ * @param {Uint8Array} update
+ *
+ */
+export const decodeUpdate = (update) => decodeUpdateV2(update, UpdateDecoderV1)
+
+/**
+ * @param {Uint8Array} update
+ * @param {typeof UpdateDecoderV2 | typeof UpdateDecoderV1} [YDecoder]
+ *
+ */
+export const decodeUpdateV2 = (update, YDecoder = UpdateDecoderV2) => {
+  const structs = []
+  const updateDecoder = new YDecoder(decoding.createDecoder(update))
+  const lazyDecoder = new LazyStructReader(updateDecoder, false)
+  for (let curr = lazyDecoder.curr; curr !== null; curr = lazyDecoder.next()) {
+    structs.push(curr)
+  }
+  return {
+    structs,
+    ds: readDeleteSet(updateDecoder)
+  }
+}
+
 export class LazyStructWriter {
   /**
    * @param {UpdateEncoderV1 | UpdateEncoderV2} encoder
@@ -308,7 +332,6 @@ export const mergeUpdatesV2 = (updates, YDecoder = UpdateDecoderV2, YEncoder = U
   // Note: Should handle that some operations cannot be applied yet ()
 
   while (true) {
-    // @todo this incurs an exponential overhead. We could instead only sort the item that changed.
     // Write higher clients first ⇒ sort by clientID & clock and remove decoders without content
     lazyStructDecoders = lazyStructDecoders.filter(dec => dec.curr !== null)
     lazyStructDecoders.sort(

tests/doc.tests.js

@@ -2,12 +2,37 @@
 import * as Y from '../src/index.js'
 import * as t from 'lib0/testing'
 
+/**
+ * @param {t.TestCase} _tc
+ */
+export const testOriginInTransaction = _tc => {
+  const doc = new Y.Doc()
+  const ytext = doc.getText()
+  /**
+   * @type {Array<string>}
+   */
+  const origins = []
+  doc.on('afterTransaction', (tr) => {
+    origins.push(tr.origin)
+    if (origins.length <= 1) {
+      ytext.toDelta()
+      doc.transact(() => {
+        ytext.insert(0, 'a')
+      }, 'nested')
+    }
+  })
+  doc.transact(() => {
+    ytext.insert(0, '0')
+  }, 'first')
+  t.compareArrays(origins, ['first', 'cleanup', 'nested'])
+}
+
 /**
  * Client id should be changed when an instance receives updates from another client using the same client id.
  *
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testClientIdDuplicateChange = tc => {
+export const testClientIdDuplicateChange = _tc => {
   const doc1 = new Y.Doc()
   doc1.clientID = 0
   const doc2 = new Y.Doc()
@@ -19,9 +44,9 @@ export const testClientIdDuplicateChange = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testGetTypeEmptyId = tc => {
+export const testGetTypeEmptyId = _tc => {
   const doc1 = new Y.Doc()
   doc1.getText('').insert(0, 'h')
   doc1.getText().insert(1, 'i')
@@ -32,15 +57,14 @@ export const testGetTypeEmptyId = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testToJSON = tc => {
+export const testToJSON = _tc => {
   const doc = new Y.Doc()
   t.compare(doc.toJSON(), {}, 'doc.toJSON yields empty object')
 
   const arr = doc.getArray('array')
   arr.push(['test1'])
-  t.compare(arr.toJSON(), ['test1'])
 
   const map = doc.getMap('map')
   map.set('k1', 'v1')
@@ -60,9 +84,9 @@ export const testToJSON = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testSubdoc = tc => {
+export const testSubdoc = _tc => {
   const doc = new Y.Doc()
   doc.load() // doesn't do anything
   {
@@ -127,9 +151,9 @@ export const testSubdoc = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testSubdocLoadEdgeCases = tc => {
+export const testSubdocLoadEdgeCases = _tc => {
   const ydoc = new Y.Doc()
   const yarray = ydoc.getArray()
   const subdoc1 = new Y.Doc()
@@ -174,9 +198,9 @@ export const testSubdocLoadEdgeCases = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testSubdocLoadEdgeCasesAutoload = tc => {
+export const testSubdocLoadEdgeCasesAutoload = _tc => {
   const ydoc = new Y.Doc()
   const yarray = ydoc.getArray()
   const subdoc1 = new Y.Doc({ autoLoad: true })
@@ -216,9 +240,9 @@ export const testSubdocLoadEdgeCasesAutoload = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testSubdocsUndo = tc => {
+export const testSubdocsUndo = _tc => {
   const ydoc = new Y.Doc()
   const elems = ydoc.getXmlFragment()
   const undoManager = new Y.UndoManager(elems)
@@ -231,9 +255,9 @@ export const testSubdocsUndo = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testLoadDocs = async tc => {
+export const testLoadDocsEvent = async _tc => {
   const ydoc = new Y.Doc()
   t.assert(ydoc.isLoaded === false)
   let loadedEvent = false
@@ -245,3 +269,44 @@ export const testLoadDocs = async tc => {
   t.assert(loadedEvent)
   t.assert(ydoc.isLoaded)
 }
+
+/**
+ * @param {t.TestCase} _tc
+ */
+export const testSyncDocsEvent = async _tc => {
+  const ydoc = new Y.Doc()
+  t.assert(ydoc.isLoaded === false)
+  t.assert(ydoc.isSynced === false)
+  let loadedEvent = false
+  ydoc.once('load', () => {
+    loadedEvent = true
+  })
+  let syncedEvent = false
+  ydoc.once('sync', /** @param {any} isSynced */ (isSynced) => {
+    syncedEvent = true
+    t.assert(isSynced)
+  })
+  ydoc.emit('sync', [true, ydoc])
+  await ydoc.whenLoaded
+  const oldWhenSynced = ydoc.whenSynced
+  await ydoc.whenSynced
+  t.assert(loadedEvent)
+  t.assert(syncedEvent)
+  t.assert(ydoc.isLoaded)
+  t.assert(ydoc.isSynced)
+  let loadedEvent2 = false
+  ydoc.on('load', () => {
+    loadedEvent2 = true
+  })
+  let syncedEvent2 = false
+  ydoc.on('sync', (isSynced) => {
+    syncedEvent2 = true
+    t.assert(isSynced === false)
+  })
+  ydoc.emit('sync', [false, ydoc])
+  t.assert(!loadedEvent2)
+  t.assert(syncedEvent2)
+  t.assert(ydoc.isLoaded)
+  t.assert(!ydoc.isSynced)
+  t.assert(ydoc.whenSynced !== oldWhenSynced)
+}

tests/encoding.tests.js

@@ -12,7 +12,6 @@ import {
   readContentFormat,
   readContentAny,
   readContentDoc,
-  readContentMove,
   Doc,
   PermanentUserData,
   encodeStateAsUpdate,
@@ -25,8 +24,7 @@ import * as Y from '../src/index.js'
  * @param {t.TestCase} tc
  */
 export const testStructReferences = tc => {
-  t.assert(contentRefs.length === 12)
-  // contentRefs[0] is reserved for GC
+  t.assert(contentRefs.length === 11)
   t.assert(contentRefs[1] === readContentDeleted)
   t.assert(contentRefs[2] === readContentJSON) // TODO: deprecate content json?
   t.assert(contentRefs[3] === readContentBinary)
@@ -37,7 +35,6 @@ export const testStructReferences = tc => {
   t.assert(contentRefs[8] === readContentAny)
   t.assert(contentRefs[9] === readContentDoc)
   // contentRefs[10] is reserved for Skip structs
-  t.assert(contentRefs[11] === readContentMove)
 }
 
 /**
@@ -75,9 +72,9 @@ export const testPermanentUserData = async tc => {
 export const testDiffStateVectorOfUpdateIsEmpty = tc => {
   const ydoc = new Y.Doc()
   /**
-   * @type {null | Uint8Array}
+   * @type {any}
    */
-  let sv = /** @type {any} */ (null)
+  let sv = null
   ydoc.getText().insert(0, 'a')
   ydoc.on('update', update => {
     sv = Y.encodeStateVectorFromUpdate(update)

tests/index.js

@@ -10,7 +10,6 @@ import * as doc from './doc.tests.js'
 import * as snapshot from './snapshot.tests.js'
 import * as updates from './updates.tests.js'
 import * as relativePositions from './relativePositions.tests.js'
-import * as Y from './testHelper.js'
 
 import { runTests } from 'lib0/testing'
 import { isBrowser, isNode } from 'lib0/environment'
@@ -18,8 +17,6 @@ import * as log from 'lib0/logging'
 
 if (isBrowser) {
   log.createVConsole(document.body)
-  // @ts-ignore
-  window.Y = Y
 }
 runTests({
   doc, map, array, text, xml, encoding, undoredo, compatibility, snapshot, updates, relativePositions

tests/testHelper.js

@@ -5,6 +5,7 @@ import * as encoding from 'lib0/encoding'
 import * as decoding from 'lib0/decoding'
 import * as syncProtocol from 'y-protocols/sync'
 import * as object from 'lib0/object'
+import * as map from 'lib0/map'
 import * as Y from '../src/index.js'
 export * from '../src/index.js'
 
@@ -89,8 +90,8 @@ export class TestYInstance extends Y.Doc {
         const encoder = encoding.createEncoder()
         syncProtocol.writeUpdate(encoder, update)
         broadcastMessage(this, encoding.toUint8Array(encoder))
-        this.updates.push(update)
       }
+      this.updates.push(update)
     })
     this.connect()
   }
@@ -133,12 +134,7 @@ export class TestYInstance extends Y.Doc {
    * @param {TestYInstance} remoteClient
    */
   _receive (message, remoteClient) {
-    let messages = this.receiving.get(remoteClient)
-    if (messages === undefined) {
-      messages = []
-      this.receiving.set(remoteClient, messages)
-    }
-    messages.push(message)
+    map.setIfUndefined(this.receiving, remoteClient, () => []).push(message)
   }
 }
 
@@ -202,17 +198,6 @@ export class TestConnector {
         // send reply message
         sender._receive(encoding.toUint8Array(encoder), receiver)
       }
-      {
-        // If update message, add the received message to the list of received messages
-        const decoder = decoding.createDecoder(m)
-        const messageType = decoding.readVarUint(decoder)
-        switch (messageType) {
-          case syncProtocol.messageYjsUpdate:
-          case syncProtocol.messageYjsSyncStep2:
-            receiver.updates.push(decoding.readVarUint8Array(decoder))
-            break
-        }
-      }
       return true
     }
     return false

tests/undo-redo.tests.js

@@ -1,8 +1,21 @@
-import { init, compare, applyRandomTests, Doc, UndoManager } from './testHelper.js' // eslint-disable-line
+import { init, compare, applyRandomTests, Doc } from './testHelper.js' // eslint-disable-line
 
 import * as Y from '../src/index.js'
 import * as t from 'lib0/testing'
 
+/**
+ * @param {t.TestCase} tc
+ */
+export const testInfiniteCaptureTimeout = tc => {
+  const { array0 } = init(tc, { users: 3 })
+  const undoManager = new Y.UndoManager(array0, { captureTimeout: Number.MAX_VALUE })
+  array0.push([1, 2, 3])
+  undoManager.stopCapturing()
+  array0.push([4, 5, 6])
+  undoManager.undo()
+  t.compare(array0.toArray(), [1, 2, 3])
+}
+
 /**
  * @param {t.TestCase} tc
  */
@@ -49,6 +62,20 @@ export const testUndoText = tc => {
   t.compare(text0.toDelta(), [{ insert: 'b' }, { insert: 'cxy', attributes: { bold: true } }, { insert: 'z' }])
 }
 
+/**
+ * Test case to fix #241
+ * @param {t.TestCase} tc
+ */
+export const testEmptyTypeScope = tc => {
+  const ydoc = new Y.Doc()
+  const um = new Y.UndoManager([], { doc: ydoc })
+  const yarray = ydoc.getArray()
+  um.addToScope(yarray)
+  yarray.insert(0, [1])
+  um.undo()
+  t.assert(yarray.length === 0)
+}
+
 /**
  * Test case to fix #241
  * @param {t.TestCase} tc
@@ -372,3 +399,248 @@ export const testUndoNestedUndoIssue = tc => {
   undoManager.redo()
   t.compare(design.toJSON(), { text: { blocks: { text: 'Something Else' } } })
 }
+
+/**
+ * This issue has been reported in https://github.com/yjs/yjs/issues/355
+ *
+ * @param {t.TestCase} tc
+ */
+export const testConsecutiveRedoBug = tc => {
+  const doc = new Y.Doc()
+  const yRoot = doc.getMap()
+  const undoMgr = new Y.UndoManager(yRoot)
+
+  let yPoint = new Y.Map()
+  yPoint.set('x', 0)
+  yPoint.set('y', 0)
+  yRoot.set('a', yPoint)
+  undoMgr.stopCapturing()
+
+  yPoint.set('x', 100)
+  yPoint.set('y', 100)
+  undoMgr.stopCapturing()
+
+  yPoint.set('x', 200)
+  yPoint.set('y', 200)
+  undoMgr.stopCapturing()
+
+  yPoint.set('x', 300)
+  yPoint.set('y', 300)
+  undoMgr.stopCapturing()
+
+  t.compare(yPoint.toJSON(), { x: 300, y: 300 })
+
+  undoMgr.undo() // x=200, y=200
+  t.compare(yPoint.toJSON(), { x: 200, y: 200 })
+  undoMgr.undo() // x=100, y=100
+  t.compare(yPoint.toJSON(), { x: 100, y: 100 })
+  undoMgr.undo() // x=0, y=0
+  t.compare(yPoint.toJSON(), { x: 0, y: 0 })
+  undoMgr.undo() // nil
+  t.compare(yRoot.get('a'), undefined)
+
+  undoMgr.redo() // x=0, y=0
+  yPoint = yRoot.get('a')
+
+  t.compare(yPoint.toJSON(), { x: 0, y: 0 })
+  undoMgr.redo() // x=100, y=100
+  t.compare(yPoint.toJSON(), { x: 100, y: 100 })
+  undoMgr.redo() // x=200, y=200
+  t.compare(yPoint.toJSON(), { x: 200, y: 200 })
+  undoMgr.redo() // expected x=300, y=300, actually nil
+  t.compare(yPoint.toJSON(), { x: 300, y: 300 })
+}
+
+/**
+ * This issue has been reported in https://github.com/yjs/yjs/issues/304
+ *
+ * @param {t.TestCase} tc
+ */
+export const testUndoXmlBug = tc => {
+  const origin = 'origin'
+  const doc = new Y.Doc()
+  const fragment = doc.getXmlFragment('t')
+  const undoManager = new Y.UndoManager(fragment, {
+    captureTimeout: 0,
+    trackedOrigins: new Set([origin])
+  })
+
+  // create element
+  doc.transact(() => {
+    const e = new Y.XmlElement('test-node')
+    e.setAttribute('a', '100')
+    e.setAttribute('b', '0')
+    fragment.insert(fragment.length, [e])
+  }, origin)
+
+  // change one attribute
+  doc.transact(() => {
+    const e = fragment.get(0)
+    e.setAttribute('a', '200')
+  }, origin)
+
+  // change both attributes
+  doc.transact(() => {
+    const e = fragment.get(0)
+    e.setAttribute('a', '180')
+    e.setAttribute('b', '50')
+  }, origin)
+
+  undoManager.undo()
+  undoManager.undo()
+  undoManager.undo()
+
+  undoManager.redo()
+  undoManager.redo()
+  undoManager.redo()
+  t.compare(fragment.toString(), '<test-node a="180" b="50"></test-node>')
+}
+
+/**
+ * This issue has been reported in https://github.com/yjs/yjs/issues/343
+ *
+ * @param {t.TestCase} tc
+ */
+export const testUndoBlockBug = tc => {
+  const doc = new Y.Doc({ gc: false })
+  const design = doc.getMap()
+
+  const undoManager = new Y.UndoManager(design, { captureTimeout: 0 })
+
+  const text = new Y.Map()
+
+  const blocks1 = new Y.Array()
+  const blocks1block = new Y.Map()
+  doc.transact(() => {
+    blocks1block.set('text', '1')
+    blocks1.push([blocks1block])
+
+    text.set('blocks', blocks1block)
+    design.set('text', text)
+  })
+
+  const blocks2 = new Y.Array()
+  const blocks2block = new Y.Map()
+  doc.transact(() => {
+    blocks2block.set('text', '2')
+    blocks2.push([blocks2block])
+    text.set('blocks', blocks2block)
+  })
+
+  const blocks3 = new Y.Array()
+  const blocks3block = new Y.Map()
+  doc.transact(() => {
+    blocks3block.set('text', '3')
+    blocks3.push([blocks3block])
+    text.set('blocks', blocks3block)
+  })
+
+  const blocks4 = new Y.Array()
+  const blocks4block = new Y.Map()
+  doc.transact(() => {
+    blocks4block.set('text', '4')
+    blocks4.push([blocks4block])
+    text.set('blocks', blocks4block)
+  })
+
+  // {"text":{"blocks":{"text":"4"}}}
+  undoManager.undo() // {"text":{"blocks":{"3"}}}
+  undoManager.undo() // {"text":{"blocks":{"text":"2"}}}
+  undoManager.undo() // {"text":{"blocks":{"text":"1"}}}
+  undoManager.undo() // {}
+  undoManager.redo() // {"text":{"blocks":{"text":"1"}}}
+  undoManager.redo() // {"text":{"blocks":{"text":"2"}}}
+  undoManager.redo() // {"text":{"blocks":{"text":"3"}}}
+  undoManager.redo() // {"text":{}}
+  t.compare(design.toJSON(), { text: { blocks: { text: '4' } } })
+}
+
+/**
+ * Undo text formatting delete should not corrupt peer state.
+ *
+ * @see https://github.com/yjs/yjs/issues/392
+ * @param {t.TestCase} tc
+ */
+export const testUndoDeleteTextFormat = tc => {
+  const doc = new Y.Doc()
+  const text = doc.getText()
+  text.insert(0, 'Attack ships on fire off the shoulder of Orion.')
+  const doc2 = new Y.Doc()
+  const text2 = doc2.getText()
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+  const undoManager = new Y.UndoManager(text)
+
+  text.format(13, 7, { bold: true })
+  undoManager.stopCapturing()
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+
+  text.format(16, 4, { bold: null })
+  undoManager.stopCapturing()
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+
+  undoManager.undo()
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+
+  const expect = [
+    { insert: 'Attack ships ' },
+    {
+      insert: 'on fire',
+      attributes: { bold: true }
+    },
+    { insert: ' off the shoulder of Orion.' }
+  ]
+  t.compare(text.toDelta(), expect)
+  t.compare(text2.toDelta(), expect)
+}
+
+/**
+ * Undo text formatting delete should not corrupt peer state.
+ *
+ * @see https://github.com/yjs/yjs/issues/392
+ * @param {t.TestCase} tc
+ */
+export const testBehaviorOfIgnoreremotemapchangesProperty = tc => {
+  const doc = new Y.Doc()
+  const doc2 = new Y.Doc()
+  doc.on('update', update => Y.applyUpdate(doc2, update, doc))
+  doc2.on('update', update => Y.applyUpdate(doc, update, doc2))
+  const map1 = doc.getMap()
+  const map2 = doc2.getMap()
+  const um1 = new Y.UndoManager(map1, { ignoreRemoteMapChanges: true })
+  map1.set('x', 1)
+  map2.set('x', 2)
+  map1.set('x', 3)
+  map2.set('x', 4)
+  um1.undo()
+  t.assert(map1.get('x') === 2)
+  t.assert(map2.get('x') === 2)
+}
+
+/**
+ * Special deletion case.
+ *
+ * @see https://github.com/yjs/yjs/issues/447
+ * @param {t.TestCase} tc
+ */
+export const testSpecialDeletionCase = tc => {
+  const origin = 'undoable'
+  const doc = new Y.Doc()
+  const fragment = doc.getXmlFragment()
+  const undoManager = new Y.UndoManager(fragment, { trackedOrigins: new Set([origin]) })
+  doc.transact(() => {
+    const e = new Y.XmlElement('test')
+    e.setAttribute('a', '1')
+    e.setAttribute('b', '2')
+    fragment.insert(0, [e])
+  })
+  t.compareStrings(fragment.toString(), '<test a="1" b="2"></test>')
+  doc.transact(() => {
+    // change attribute "b" and delete test-node
+    const e = fragment.get(0)
+    e.setAttribute('b', '3')
+    fragment.delete(0)
+  }, origin)
+  t.compareStrings(fragment.toString(), '')
+  undoManager.undo()
+  t.compareStrings(fragment.toString(), '<test a="1" b="2"></test>')
+}

tests/updates.tests.js

@@ -112,6 +112,24 @@ export const testMergeUpdates = tc => {
   })
 }
 
+/**
+ * @param {t.TestCase} tc
+ */
+export const testKeyEncoding = tc => {
+  const { users, text0, text1 } = init(tc, { users: 2 })
+
+  text0.insert(0, 'a', { italic: true })
+  text0.insert(0, 'b')
+  text0.insert(0, 'c', { italic: true })
+
+  const update = Y.encodeStateAsUpdateV2(users[0])
+  Y.applyUpdateV2(users[1], update)
+
+  t.compare(text1.toDelta(), [{ insert: 'c', attributes: { italic: true } }, { insert: 'b' }, { insert: 'a', attributes: { italic: true } }])
+
+  compare(users)
+}
+
 /**
  * @param {Y.Doc} ydoc
  * @param {Array<Uint8Array>} updates - expecting at least 4 updates

tests/y-array.tests.js

@@ -1,71 +1,10 @@
-import { init, compare, applyRandomTests, Doc, Item } from './testHelper.js' // eslint-disable-line
+import { init, compare, applyRandomTests, Doc } from './testHelper.js' // eslint-disable-line
 
 import * as Y from '../src/index.js'
 import * as t from 'lib0/testing'
 import * as prng from 'lib0/prng'
 import * as math from 'lib0/math'
 
-/**
- * path should be correct when moving item - see yjs#481
- *
- * @param {t.TestCase} tc
- */
-export const testArrayMovePathIssue481 = tc => {
-  const { users, testConnector, array0, array1 } = init(tc, { users: 2 })
-  array0.observeDeep(events => {
-    events.forEach(event => {
-      if (event.path.length > 0) {
-        /**
-         * @type {any}
-         */
-        let target = event.currentTarget
-        event.path.forEach(p => {
-          target = target.get(p)
-        })
-        t.assert(target === event.target)
-      }
-    })
-  })
-  array0.push([
-    ['a', '1.1'],
-    ['b', '2.2'],
-    ['c', '3.1'],
-    ['d', '4.1'],
-    ['e', '5.1']
-  ].map(e => Y.Array.from(e)))
-  testConnector.flushAllMessages()
-  users[1].transact(() => {
-    array1.get(1).insert(0, ['0'])
-    array1.move(1, 0)
-  })
-  testConnector.flushAllMessages()
-  users[1].transact(() => {
-    array1.get(3).insert(0, ['1'])
-    array1.move(3, 4)
-  })
-  testConnector.flushAllMessages()
-  users[1].transact(() => {
-    array1.get(2).insert(0, ['2'])
-    array1.move(2, array1.length)
-  })
-  testConnector.flushAllMessages()
-}
-
-/**
- * foreach has correct index - see yjs#485
- *
- * @param {t.TestCase} tc
- */
-export const testArrayIndexIssue485 = tc => {
-  const doc = new Y.Doc()
-  const yarr = doc.getArray()
-  yarr.push([1, 2])
-  yarr.forEach((el, index) => {
-    t.info('index: ' + index)
-    t.assert(yarr.get(index) === el)
-  })
-}
-
 /**
  * @param {t.TestCase} tc
  */
@@ -93,6 +32,17 @@ export const testSlice = tc => {
   t.compareArrays(arr.slice(0, 2), [0, 1])
 }
 
+/**
+ * @param {t.TestCase} tc
+ */
+export const testArrayFrom = tc => {
+  const doc1 = new Y.Doc()
+  const db1 = doc1.getMap('root')
+  const nestedArray1 = Y.Array.from([0, 1, 2])
+  db1.set('array', nestedArray1)
+  t.compare(nestedArray1.toArray(), [0, 1, 2])
+}
+
 /**
  * Debugging yjs#297 - a critical bug connected to the search-marker approach
  *
@@ -493,192 +443,6 @@ export const testEventTargetIsSetCorrectlyOnRemote = tc => {
   compare(users)
 }
 
-/**
- * @param {t.TestCase} tc
- */
-export const testMove = tc => {
-  {
-    // move in uninitialized type
-    const yarr = new Y.Array()
-    yarr.insert(0, [1, 2, 3])
-    yarr.move(1, 0)
-    // @ts-ignore
-    t.compare(yarr._prelimContent, [2, 1, 3])
-  }
-  const { array0, array1, users } = init(tc, { users: 3 })
-  /**
-   * @type {any}
-   */
-  let event0 = null
-  /**
-   * @type {any}
-   */
-  let event1 = null
-  array0.observe(event => {
-    event0 = event
-  })
-  array1.observe(event => {
-    event1 = event
-  })
-  array0.insert(0, [1, 2, 3])
-  array0.move(1, 0)
-  t.compare(array0.toArray(), [2, 1, 3])
-  t.compare(event0.delta, [{ insert: [2] }, { retain: 1 }, { delete: 1 }])
-  Y.applyUpdate(users[1], Y.encodeStateAsUpdate(users[0]))
-  t.compare(array1.toArray(), [2, 1, 3])
-  t.compare(event1.delta, [{ insert: [2, 1, 3] }])
-  array0.move(0, 2)
-  t.compare(array0.toArray(), [1, 2, 3])
-  t.compare(event0.delta, [{ delete: 1 }, { retain: 1 }, { insert: [2] }])
-  compare(users)
-}
-
-/**
- * @param {t.TestCase} tc
- */
-export const testMove2 = tc => {
-  {
-    // move in uninitialized type
-    const yarr = new Y.Array()
-    yarr.insert(0, [1, 2])
-    yarr.move(1, 0)
-    // @ts-ignore
-    t.compare(yarr._prelimContent, [2, 1])
-  }
-  const { array0, array1, users } = init(tc, { users: 3 })
-  /**
-   * @type {any}
-   */
-  let event0 = null
-  /**
-   * @type {any}
-   */
-  let event1 = null
-  array0.observe(event => {
-    event0 = event
-  })
-  array1.observe(event => {
-    event1 = event
-  })
-  array0.insert(0, [1, 2])
-  array0.move(1, 0)
-  t.compare(array0.toArray(), [2, 1])
-  t.compare(event0.delta, [{ insert: [2] }, { retain: 1 }, { delete: 1 }])
-  Y.applyUpdate(users[1], Y.encodeStateAsUpdate(users[0]))
-  t.compare(array1.toArray(), [2, 1])
-  t.compare(event1.delta, [{ insert: [2, 1] }])
-  array0.move(0, 2)
-  t.compare(array0.toArray(), [1, 2])
-  t.compare(event0.delta, [{ delete: 1 }, { retain: 1 }, { insert: [2] }])
-  compare(users)
-}
-
-/**
- * @param {t.TestCase} tc
- */
-export const testMoveSingleItemRemovesPrev = tc => {
-  const ydoc = new Y.Doc()
-  const yarray = ydoc.getArray()
-  yarray.insert(0, [1, 2, 3])
-  yarray.move(0, 3)
-  t.compareArrays(yarray.toArray(), [2, 3, 1])
-  yarray.move(2, 0)
-  t.compareArrays(yarray.toArray(), [1, 2, 3])
-  let item = yarray._start
-  const items = []
-  while (item) {
-    items.push(item)
-    item = item.right
-  }
-  t.assert(items.length === 4)
-  t.assert(items.filter(item => !item.deleted).length === 3)
-}
-
-/**
- * Check that the searchMarker is reused correctly.
- *
- * @param {t.TestCase} tc
- */
-export const testListWalkerReusesSearchMarker = tc => {
-  const ydoc = new Y.Doc()
-  const yarray = ydoc.getArray()
-  const iterations = 100
-  for (let i = 0; i < iterations; i++) {
-    yarray.insert(0, [i])
-  }
-  /**
-   * @type {any}
-   */
-  let prevSm = null
-  for (let i = 0; i < iterations; i++) {
-    const v = yarray.get(i)
-    t.assert(v === iterations - i - 1)
-    t.assert(yarray._searchMarker.length <= 1)
-    const sm = yarray._searchMarker[0]
-    t.assert(prevSm == null || sm === prevSm)
-    prevSm = sm
-  }
-}
-
-/**
- * @param {t.TestCase} tc
- */
-export const testMoveDeletions = tc => {
-  const ydoc = new Y.Doc()
-  const yarray = ydoc.getArray()
-  const array = yarray.toArray()
-  /**
-   * @type {any}
-   */
-  let lastDelta = []
-  yarray.observe(event => {
-    lastDelta = event.delta
-    let pos = 0
-    for (let i = 0; i < lastDelta.length; i++) {
-      const d = lastDelta[i]
-      if (d.retain != null) {
-        pos += d.retain
-      } else if (d.insert instanceof Array) {
-        array.splice(pos, 0, ...d.insert)
-        pos += d.insert.length
-      } else if (d.delete != null) {
-        array.splice(pos, d.delete)
-      }
-    }
-  })
-  yarray.insert(0, [1, 2, 3])
-  // @todo should be old-position to new-position. so that below move matches
-  yarray.move(2, 0)
-  t.compare(lastDelta, [{ insert: [3] }, { retain: 2 }, { delete: 1 }])
-  t.compareArrays(yarray.toArray(), [3, 1, 2])
-  t.compareArrays(yarray.toArray(), array)
-  ydoc.transact(tr => {
-    /** @type {Item} */ (yarray._start).delete(tr)
-  })
-  t.compare(lastDelta, [{ delete: 1 }, { retain: 2 }, { insert: [3] }])
-  t.compareArrays(yarray.toArray(), [1, 2, 3])
-  t.compareArrays(yarray.toArray(), array)
-}
-
-/**
- * @todo
- * @param {t.TestCase} tc
- *
-export const testMoveCircles = tc => {
-  const { testConnector, array0, array1 } = init(tc, { users: 3 })
-  array0.insert(0, [1, 2, 3, 4])
-  testConnector.flushAllMessages()
-  array0.moveRange(0, 1, 3)
-  t.compare(array0.toArray(), [3, 1, 2, 4])
-  array1.moveRange(2, 3, 1)
-  t.compare(array1.toArray(), [1, 3, 4, 2])
-  testConnector.flushAllMessages()
-  t.assert(array0.length === 4)
-  t.assert(array0.length === array0.toArray().length)
-  t.compareArrays(array0.toArray(), array1.toArray())
-}
-*/
-
 /**
  * @param {t.TestCase} tc
  */
@@ -705,23 +469,6 @@ const getUniqueNumber = () => _uniqueNumber++
  * @type {Array<function(Doc,prng.PRNG,any):void>}
  */
 const arrayTransactions = [
-  function move (user, gen) {
-    const yarray = user.getArray('array')
-    if (yarray.length === 0) {
-      return
-    }
-    const pos = prng.int32(gen, 0, yarray.length - 1)
-    const len = 1 // prng.int32(gen, 1, math.min(3, yarray.length - pos))
-    const _newPosAdj = prng.int32(gen, 0, yarray.length - len)
-    // make sure that we don't insert in-between the moved range
-    const newPos = _newPosAdj + (_newPosAdj > pos ? len : 0)
-    const oldContent = yarray.toArray()
-    // yarray.moveRange(pos, pos + len - 1, newPos)
-    yarray.move(pos, newPos)
-    const movedValues = oldContent.splice(pos, len)
-    oldContent.splice(pos < newPos ? newPos - len : newPos, 0, ...movedValues)
-    t.compareArrays(yarray.toArray(), oldContent) // we want to make sure that fastSearch markers insert at the correct position
-  },
   function insert (user, gen) {
     const yarray = user.getArray('array')
     const uniqueNumber = getUniqueNumber()
@@ -780,156 +527,95 @@ const arrayTransactions = [
   }
 ]
 
-/**
- * @param {Y.Doc} user
- */
-const monitorArrayTestObject = user => {
-  /**
-   * @type {Array<any>}
-   */
-  const arr = []
-  const yarr = user.getArray('array')
-  yarr.observe(event => {
-    let currpos = 0
-    const delta = event.delta
-    for (let i = 0; i < delta.length; i++) {
-      const d = delta[i]
-      if (d.insert != null) {
-        arr.splice(currpos, 0, ...(/** @type {Array<any>} */ (d.insert)))
-        currpos += /** @type {Array<any>} */ (d.insert).length
-      } else if (d.retain != null) {
-        currpos += d.retain
-      } else {
-        arr.splice(currpos, d.delete)
-      }
-    }
-    t.compare(arr, yarr.toArray())
-  })
-  return arr
-}
-
-/**
- * @param {{ testObjects: Array<Array<any>>, users: Array<Y.Doc> }} cmp
- */
-const compareTestobjects = cmp => {
-  const arrs = cmp.testObjects
-  for (let i = 0; i < arrs.length; i++) {
-    const type = cmp.users[i].getArray('array')
-    t.compareArrays(arrs[i], type.toArray())
-    t.compareArrays(arrs[i], Array.from(type))
-  }
-}
-
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests6 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 6, monitorArrayTestObject))
-}
-
-/**
- * @param {t.TestCase} tc
- */
-export const testRepeatGeneratingYarrayTests10 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 10, monitorArrayTestObject))
-}
-
-/**
- * @param {t.TestCase} tc
- */
-export const testRepeatGeneratingYarrayTests30 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 30, monitorArrayTestObject))
-}
-
-/**
- * @param {t.TestCase} tc
- */
-export const testRepeatGeneratingYarrayTests35 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 35, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 6)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests40 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 40, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 40)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests42 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 42, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 42)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests43 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 43, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 43)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests44 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 44, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 44)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests45 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 45, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 45)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests46 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 46, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 46)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests300 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 300, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 300)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests400 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 400, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 400)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests500 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 500, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 500)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests600 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 600, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 600)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests1000 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 1000, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 1000)
 }
 
 /**
  * @param {t.TestCase} tc
  */
 export const testRepeatGeneratingYarrayTests1800 = tc => {
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 1800, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 1800)
 }
 
 /**
@@ -937,7 +623,7 @@ export const testRepeatGeneratingYarrayTests1800 = tc => {
  */
 export const testRepeatGeneratingYarrayTests3000 = tc => {
   t.skip(!t.production)
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 3000, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 3000)
 }
 
 /**
@@ -945,7 +631,7 @@ export const testRepeatGeneratingYarrayTests3000 = tc => {
  */
 export const testRepeatGeneratingYarrayTests5000 = tc => {
   t.skip(!t.production)
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 5000, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 5000)
 }
 
 /**
@@ -953,5 +639,5 @@ export const testRepeatGeneratingYarrayTests5000 = tc => {
  */
 export const testRepeatGeneratingYarrayTests30000 = tc => {
   t.skip(!t.production)
-  compareTestobjects(applyRandomTests(tc, arrayTransactions, 30000, monitorArrayTestObject))
+  applyRandomTests(tc, arrayTransactions, 30000)
 }

tests/y-map.tests.js

@@ -455,9 +455,9 @@ export const testChangeEvent = tc => {
 }
 
 /**
- * @param {t.TestCase} tc
+ * @param {t.TestCase} _tc
  */
-export const testYmapEventExceptionsShouldCompleteTransaction = tc => {
+export const testYmapEventExceptionsShouldCompleteTransaction = _tc => {
   const doc = new Y.Doc()
   const map = doc.getMap('map')
 

tests/y-text.tests.js

@@ -6,6 +6,417 @@ import * as math from 'lib0/math'
 const { init, compare } = Y
 
 /**
+ * https://github.com/yjs/yjs/issues/474
+ * @todo Remove debug: 127.0.0.1:8080/test.html?filter=\[88/
+ * @param {t.TestCase} _tc
+ */
+export const testDeltaBug = _tc => {
+  const initialDelta = [{
+    attributes: {
+      'block-id': 'block-28eea923-9cbb-4b6f-a950-cf7fd82bc087'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'table-col': {
+        width: '150'
+      }
+    },
+    insert: '\n\n\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-9144be72-e528-4f91-b0b2-82d20408e9ea',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-6kv2ls',
+        cell: 'cell-apba4k'
+      },
+      row: 'row-6kv2ls',
+      cell: 'cell-apba4k',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-639adacb-1516-43ed-b272-937c55669a1c',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-6kv2ls',
+        cell: 'cell-a8qf0r'
+      },
+      row: 'row-6kv2ls',
+      cell: 'cell-a8qf0r',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-6302ca4a-73a3-4c25-8c1e-b542f048f1c6',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-6kv2ls',
+        cell: 'cell-oi9ikb'
+      },
+      row: 'row-6kv2ls',
+      cell: 'cell-oi9ikb',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-ceeddd05-330e-4f86-8017-4a3a060c4627',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-d1sv2g',
+        cell: 'cell-dt6ks2'
+      },
+      row: 'row-d1sv2g',
+      cell: 'cell-dt6ks2',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-37b19322-cb57-4e6f-8fad-0d1401cae53f',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-d1sv2g',
+        cell: 'cell-qah2ay'
+      },
+      row: 'row-d1sv2g',
+      cell: 'cell-qah2ay',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-468a69b5-9332-450b-9107-381d593de249',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-d1sv2g',
+        cell: 'cell-fpcz5a'
+      },
+      row: 'row-d1sv2g',
+      cell: 'cell-fpcz5a',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-26b1d252-9b2e-4808-9b29-04e76696aa3c',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-pflz90',
+        cell: 'cell-zrhylp'
+      },
+      row: 'row-pflz90',
+      cell: 'cell-zrhylp',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-6af97ba7-8cf9-497a-9365-7075b938837b',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-pflz90',
+        cell: 'cell-s1q9nt'
+      },
+      row: 'row-pflz90',
+      cell: 'cell-s1q9nt',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-107e273e-86bc-44fd-b0d7-41ab55aca484',
+      'table-cell-line': {
+        rowspan: '1',
+        colspan: '1',
+        row: 'row-pflz90',
+        cell: 'cell-20b0j9'
+      },
+      row: 'row-pflz90',
+      cell: 'cell-20b0j9',
+      rowspan: '1',
+      colspan: '1'
+    },
+    insert: '\n'
+  },
+  {
+    attributes: {
+      'block-id': 'block-38161f9c-6f6d-44c5-b086-54cc6490f1e3'
+    },
+    insert: '\n'
+  },
+  {
+    insert: 'Content after table'
+  },
+  {
+    attributes: {
+      'block-id': 'block-15630542-ef45-412d-9415-88f0052238ce'
+    },
+    insert: '\n'
+  }
+  ]
+  const ydoc1 = new Y.Doc()
+  const ytext = ydoc1.getText()
+  ytext.applyDelta(initialDelta)
+  const addingDash = [
+    {
+      retain: 12
+    },
+    {
+      insert: '-'
+    }
+  ]
+  ytext.applyDelta(addingDash)
+  const addingSpace = [
+    {
+      retain: 13
+    },
+    {
+      insert: ' '
+    }
+  ]
+  ytext.applyDelta(addingSpace)
+  const addingList = [
+    {
+      retain: 12
+    },
+    {
+      delete: 2
+    },
+    {
+      retain: 1,
+      attributes: {
+        // Clear table line attribute
+        'table-cell-line': null,
+        // Add list attribute in place of table-cell-line
+        list: {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-pflz90',
+          cell: 'cell-20b0j9',
+          list: 'bullet'
+        }
+      }
+    }
+  ]
+  ytext.applyDelta(addingList)
+  const result = ytext.toDelta()
+  const expectedResult = [
+    {
+      attributes: {
+        'block-id': 'block-28eea923-9cbb-4b6f-a950-cf7fd82bc087'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'table-col': {
+          width: '150'
+        }
+      },
+      insert: '\n\n\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-9144be72-e528-4f91-b0b2-82d20408e9ea',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-6kv2ls',
+          cell: 'cell-apba4k'
+        },
+        row: 'row-6kv2ls',
+        cell: 'cell-apba4k',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-639adacb-1516-43ed-b272-937c55669a1c',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-6kv2ls',
+          cell: 'cell-a8qf0r'
+        },
+        row: 'row-6kv2ls',
+        cell: 'cell-a8qf0r',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-6302ca4a-73a3-4c25-8c1e-b542f048f1c6',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-6kv2ls',
+          cell: 'cell-oi9ikb'
+        },
+        row: 'row-6kv2ls',
+        cell: 'cell-oi9ikb',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-ceeddd05-330e-4f86-8017-4a3a060c4627',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-d1sv2g',
+          cell: 'cell-dt6ks2'
+        },
+        row: 'row-d1sv2g',
+        cell: 'cell-dt6ks2',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-37b19322-cb57-4e6f-8fad-0d1401cae53f',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-d1sv2g',
+          cell: 'cell-qah2ay'
+        },
+        row: 'row-d1sv2g',
+        cell: 'cell-qah2ay',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-468a69b5-9332-450b-9107-381d593de249',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-d1sv2g',
+          cell: 'cell-fpcz5a'
+        },
+        row: 'row-d1sv2g',
+        cell: 'cell-fpcz5a',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-26b1d252-9b2e-4808-9b29-04e76696aa3c',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-pflz90',
+          cell: 'cell-zrhylp'
+        },
+        row: 'row-pflz90',
+        cell: 'cell-zrhylp',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      attributes: {
+        'block-id': 'block-6af97ba7-8cf9-497a-9365-7075b938837b',
+        'table-cell-line': {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-pflz90',
+          cell: 'cell-s1q9nt'
+        },
+        row: 'row-pflz90',
+        cell: 'cell-s1q9nt',
+        rowspan: '1',
+        colspan: '1'
+      },
+      insert: '\n'
+    },
+    {
+      insert: '\n',
+      // This attibutes has only list and no table-cell-line
+      attributes: {
+        list: {
+          rowspan: '1',
+          colspan: '1',
+          row: 'row-pflz90',
+          cell: 'cell-20b0j9',
+          list: 'bullet'
+        },
+        'block-id': 'block-107e273e-86bc-44fd-b0d7-41ab55aca484',
+        row: 'row-pflz90',
+        cell: 'cell-20b0j9',
+        rowspan: '1',
+        colspan: '1'
+      }
+    },
+    // No table-cell-line below here
+    {
+      attributes: {
+        'block-id': 'block-38161f9c-6f6d-44c5-b086-54cc6490f1e3'
+      },
+      insert: '\n'
+    },
+    {
+      insert: 'Content after table'
+    },
+    {
+      attributes: {
+        'block-id': 'block-15630542-ef45-412d-9415-88f0052238ce'
+      },
+      insert: '\n'
+    }
+  ]
+  t.compare(result, expectedResult)
+}
+
+/**
+ * In this test we are mainly interested in the cleanup behavior and whether the resulting delta makes sense.
+ * It is fine if the resulting delta is not minimal. But applying the delta to a rich-text editor should result in a
+ * synced document.
+ *
  * @param {t.TestCase} tc
  */
 export const testDeltaAfterConcurrentFormatting = tc => {
@@ -14,12 +425,17 @@ export const testDeltaAfterConcurrentFormatting = tc => {
   testConnector.flushAllMessages()
   text0.format(0, 3, { bold: true })
   text1.format(2, 2, { bold: true })
-  let delta = null
+  /**
+   * @type {any}
+   */
+  const deltas = []
   text1.observe(event => {
-    delta = event.delta
+    if (event.delta.length > 0) {
+      deltas.push(event.delta)
+    }
   })
   testConnector.flushAllMessages()
-  t.compare(delta, [])
+  t.compare(deltas, [[{ retain: 3, attributes: { bold: true } }, { retain: 2, attributes: { bold: null } }]])
 }
 
 /**
@@ -138,6 +554,28 @@ export const testNotMergeEmptyLinesFormat = tc => {
   ])
 }
 
+/**
+ * @param {t.TestCase} tc
+ */
+export const testPreserveAttributesThroughDelete = tc => {
+  const ydoc = new Y.Doc()
+  const testText = ydoc.getText('test')
+  testText.applyDelta([
+    { insert: 'Text' },
+    { insert: '\n', attributes: { title: true } },
+    { insert: '\n' }
+  ])
+  testText.applyDelta([
+    { retain: 4 },
+    { delete: 1 },
+    { retain: 1, attributes: { title: true } }
+  ])
+  t.compare(testText.toDelta(), [
+    { insert: 'Text' },
+    { insert: '\n', attributes: { title: true } }
+  ])
+}
+
 /**
  * @param {t.TestCase} tc
  */
@@ -327,7 +765,6 @@ export const testFormattingDeltaUnnecessaryAttributeChange = tc => {
  * @param {t.TestCase} tc
  */
 export const testInsertAndDeleteAtRandomPositions = tc => {
-  // @todo optimize to run at least as fast as previous marker approach
   const N = 100000
   const { text0 } = init(tc, { users: 1 })
   const gen = tc.prng
@@ -553,6 +990,8 @@ export const testSearchMarkerBug1 = tc => {
 }
 
 /**
+ * Reported in https://github.com/yjs/yjs/pull/32
+ *
  * @param {t.TestCase} tc
  */
 export const testFormattingBug = async tc => {
@@ -562,6 +1001,7 @@ export const testFormattingBug = async tc => {
   text1.insert(0, '\n\n\n')
   text1.format(0, 3, { url: 'http://example.com' })
   ydoc1.getText().format(1, 1, { url: 'http://docs.yjs.dev' })
+  ydoc2.getText().format(1, 1, { url: 'http://docs.yjs.dev' })
   Y.applyUpdate(ydoc2, Y.encodeStateAsUpdate(ydoc1))
   const text2 = ydoc2.getText()
   const expectedResult = [
@@ -574,6 +1014,35 @@ export const testFormattingBug = async tc => {
   console.log(text1.toDelta())
 }
 
+/**
+ * Delete formatting should not leave redundant formatting items.
+ *
+ * @param {t.TestCase} tc
+ */
+export const testDeleteFormatting = tc => {
+  const doc = new Y.Doc()
+  const text = doc.getText()
+  text.insert(0, 'Attack ships on fire off the shoulder of Orion.')
+
+  const doc2 = new Y.Doc()
+  const text2 = doc2.getText()
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+
+  text.format(13, 7, { bold: true })
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+
+  text.format(16, 4, { bold: null })
+  Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+
+  const expected = [
+    { insert: 'Attack ships ' },
+    { insert: 'on ', attributes: { bold: true } },
+    { insert: 'fire off the shoulder of Orion.' }
+  ]
+  t.compare(text.toDelta(), expected)
+  t.compare(text2.toDelta(), expected)
+}
+
 // RANDOM TESTS
 
 let charCounter = 0