13.4.6

add Y.Array.from

INTERNALS.md

@@ -0,0 +1,179 @@
+# Yjs Internals
+
+This document roughly explains how Yjs works internally. There is a complete
+walkthrough of the Yjs codebase available as a recording:
+https://youtu.be/0l5XgnQ6rB4
+
+The Yjs CRDT algorithm is described in the [YATA
+paper](https://www.researchgate.net/publication/310212186_Near_Real-Time_Peer-to-Peer_Shared_Editing_on_Extensible_Data_Types)
+from 2016. For an algorithmic view of how it works, the paper is a reasonable
+place to start. There are a handful of small improvements implemented in Yjs
+which aren't described in the paper. The most notable is that items have an
+`originRight` as well as an `origin` property, which improves performance when
+many concurrent inserts happen after the same character.
+
+At its heart, Yjs is a list CRDT. Everything is squeezed into a list in order to
+reuse the CRDT resolution algorithm:
+
+- Arrays are easy - they're lists of arbitrary items.
+- Text is a list of characters, optionally punctuated by formatting markers and
+  embeds for rich text support. Several characters can be wrapped in a single
+linked list `Item` (this is also known as the compound representation of
+CRDTs). More information about this in [this blog
+article](https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/).
+- Maps are lists of entries. The last inserted entry for each key is used, and
+  all other duplicates for each key are flagged as deleted.
+
+Each client is assigned a unique *clientID* property on first insert. This is a
+random 53-bit integer (53 bits because that fits in the javascript safe integer
+range).
+
+## List items
+
+Each item in a Yjs list is made up of two objects:
+
+- An `Item` (*src/structs/Item.js*). This is used to relate the item to other
+  adjacent items.
+- An object in the `AbstractType` heirachy (subclasses of
+  *src/types/AbstractType.js* - eg `YText`). This stores the actual content in
+the Yjs document.
+
+The item and type object pair have a 1-1 mapping. The item's `content` field
+references the AbstractType object and the AbstractType object's `_item` field
+references the item.
+
+Everything inserted in a Yjs document is given a unique ID, formed from a
+*ID(clientID, clock)* pair (also known as a [Lamport
+Timestamp](https://en.wikipedia.org/wiki/Lamport_timestamp)). The clock counts
+up from 0 with the first inserted character or item a client makes. This is
+similar to automerge's operation IDs, but note that the clock is only
+incremented by inserts. Deletes are handled in a very different way (see
+below).
+
+If a run of characters is inserted into a document (eg `"abc"`), the clock will
+be incremented for each character (eg 3 times here). But Yjs will only add a
+single `Item` into the list. This has no effect on the core CRDT algorithm, but
+the optimization dramatically decreases the number of javascript objects
+created during normal text editing. This optimization only applies if the
+characters share the same clientID, they're inserted in order, and all
+characters have either been deleted or all characters are not deleted. The item
+will be split if the run is interrupted for any reason (eg a character in the
+middle of the run is deleted).
+
+When an item is created, it stores a reference to the IDs of the preceeding and
+succeeding item. These are stored in the item's `origin` and `originRight`
+fields, respectively. These are used when peers concurrently insert at the same
+location in a document. Though quite rare in practice, Yjs needs to make sure
+the list items always resolve to the same order on all peers. The actual logic
+is relatively simple - its only a couple dozen lines of code and it lives in
+the `Item#integrate()` method. The YATA paper has much more detail on the this
+algorithm.
+
+### Item Storage
+
+The items themselves are stored in two data structures and a cache:
+
+- The items are stored in a tree of doubly-linked lists in *document order*.
+  Each item has `left` and `right` properties linking to its siblings in the
+document. Items also have a `parent` property to reference their parent in the
+document tree (null at the root). (And you can access an item's children, if
+any, through `item.content`).
+- All items are referenced in *insertion order* inside the struct store
+  (*src/utils/StructStore.js*). This references the list of items inserted by
+for each client, in chronological order. This is used to find an item in the
+tree with a given ID (using a binary search). It is also used to efficiently
+gather the operations a peer is missing during sync (more on this below).
+
+When a local insert happens, Yjs needs to map the insert position in the
+document (eg position 1000) to an ID. With just the linked list, this would
+require a slow O(n) linear scan of the list. But when editing a document, most
+inserts are either at the same position as the last insert, or nearby. To
+improve performance, Yjs stores a cache of the 10 most recently looked up
+insert positions in the document. This is consulted and updated when a position
+is looked up to improve performance in the average case. The cache is updated
+using a heuristic that is still changing (currently, it is updated when a new
+position significantly diverges from existing markers in the cache). Internally
+this is referred to as the skip list / fast search marker.
+
+### Deletions
+
+Deletions in Yjs are treated very differently from insertions. Insertions are
+implemented as a sequential operation based CRDT, but deletions are treated as
+a simpler state based CRDT.
+
+When an item has been deleted by any peer, at any point in history, it is
+flagged as deleted on the item. (Internally Yjs uses the `info` bitfield.) Yjs
+does not record metadata about a deletion:
+
+- No data is kept on *when* an item was deleted, or which user deleted it.
+- The struct store does not contain deletion records
+- The clientID's clock is not incremented
+
+If garbage collection is enabled in Yjs, when an object is deleted its content
+is discarded. If a deleted object contains children (eg a field is deleted in
+an object), the content is replaced with a `GC` object (*src/structs/GC.js*).
+This is a very lightweight structure - it only stores the length of the removed
+content.
+
+Yjs has some special logic to share which content in a document has been
+deleted:
+
+- When a delete happens, as well as marking the item, the deleted IDs are
+  listed locally within the transaction. (See below for more information about
+transactions.) When a transaction has been committed locally, the set of
+deleted items is appended to a transaction's update message.
+- A snapshot (a marked point in time in the Yjs history) is specified using
+  both the set of (clientID, clock) pairs *and* the set of all deleted item
+IDs. The deleted set is O(n), but because deletions usually happen in runs,
+this data set is usually tiny in practice. (The real world editing trace from
+the B4 benchmark document contains 182k inserts and 77k deleted characters. The
+deleted set size in a snapshot is only 4.5Kb).
+
+## Transactions
+
+All updates in Yjs happen within a *transaction*. (Defined in
+*src/utils/Transaction.js*.)
+
+The transaction collects a set of updates to the Yjs document to be applied on
+remote peers atomically. Once a transaction has been committed locally, it
+generates a compressed *update message* which is broadcast to synchronized
+remote peers to notify them of the local change. The update message contains:
+
+- The set of newly inserted items
+- The set of items deleted within the transaction.
+
+## Network protocol
+
+The network protocol is not really a part of Yjs. There are a few relevant
+concepts that can be used to create a custom network protocol:
+
+* `update`: The Yjs document can be encoded to an *update* object that can be
+  parsed to reconstruct the document. Also every change on the document fires
+an incremental document updates that allows clients to sync with each other.
+The update object is an Uint8Array that efficiently encodes `Item` objects and
+the delete set.
+* `state vector`: A state vector defines the know state of each user (a set of
+  tubles `(client, clock)`). This object is also efficiently encoded as a
+Uint8Array.
+
+The client can ask a remote client for missing document updates by sending
+their state vector (often referred to as *sync step 1*). The remote peer can
+compute the missing `Item` objects using the `clocks` of the respective clients
+and compute a minimal update message that reflects all missing updates (sync
+step 2).
+
+An implementation of the syncing process is in
+[y-protocols](https://github.com/yjs/y-protocols).
+
+## Snapshots
+
+A snapshot can be used to restore an old document state. It is a `state vector`
++ `delete set`. I client can restore an old document state by iterating through
+the sequence CRDT and ignoring all Items that have an `id.clock >
+stateVector[id.client].clock`. Instead of using `item.deleted` the client will
+use the delete set to find out if an item was deleted or not.
+
+It is not recommended to restore an old document state using snapshots,
+although that would certainly be possible. Instead, the old state should be
+computed by iterating through the newest state and using the additional
+information from the state vector.

README.md

@@ -31,29 +31,29 @@ I'm currently looking for sponsors that allow me to be less dependent on
 contracting work. These awesome backers already fund further development of
 Yjs:
 
-[![Vincent Waller](https://github.com/vwall.png?size=60)](https://github.com/vwall)
-[<img src="https://user-images.githubusercontent.com/5553757/83337333-a7bcb380-a2ba-11ea-837b-e404eb35d318.png"
-height="60px" />](https://input.com/)
-[![Duane Johnson](https://github.com/canadaduane.png?size=60)](https://github.com/canadaduane)
-[![Joe Reeve](https://github.com/ISNIT0.png?size=60)](https://github.com/ISNIT0)
-[<img src="https://room.sh/img/icons/android-chrome-192x192.png" height="60px" />](https://room.sh/)
+[![Ifiok Jr.](https://github.com/ifiokjr.png?size=60)](https://github.com/ifiokjr)
+[![Burke Libbey](https://github.com/burke.png?size=60)](https://github.com/burke)
+[![Beni Cherniavsky-Paskin](https://github.com/cben.png?size=60)](https://github.com/cben)
+[![Tom Moor](https://github.com/tommoor.png?size=60)](https://github.com/tommoor)
+[![Michael Meyers](https://github.com/michaelemeyers.png?size=60)](https://github.com/michaelemeyers)
+[![Cristiano Benjamin](https://github.com/csbenjamin.png?size=60)](https://github.com/csbenjamin)
+[![Braden](https://github.com/AdventureBeard.png?size=60)](https://github.com/AdventureBeard)
+[![nimbuswebinc](https://nimbusweb.me/new-style-img/note-icon.svg)](https://github.com/nimbuswebinc)
 [![JourneyApps](https://github.com/journeyapps.png?size=60)](https://github.com/journeyapps)
 [![Adam Brunnmeier](https://github.com/adabru.png?size=60)](https://github.com/adabru)
 [![Nathanael Anderson](https://github.com/NathanaelA.png?size=60)](https://github.com/NathanaelA)
-[![Gremloon](https://github.com/gremloon.png?size=60)](https://github.com/gremloon)
-[![ifiokjr](https://github.com/ifiokjr.png?size=60)](https://github.com/ifiokjr)
-[![mrfambo](https://github.com/mrfambo.png?size=60)](https://github.com/mrfambo)
+[<img src="https://room.sh/img/icons/android-chrome-192x192.png" height="60px" />](https://room.sh/)
 
 Sponsorship also comes with special perks! [![Become a Sponsor](https://img.shields.io/static/v1?label=Become%20a%20Sponsor&message=%E2%9D%A4&logo=GitHub&style=flat&color=d42f2d)](https://github.com/sponsors/dmonad)
 
 ## Who is using Yjs
 
-* [Relm](http://www.relm.us/) A collaborative gameworld for teamwork and
+* [Relm](https://www.relm.us/) A collaborative gameworld for teamwork and
   community. :star2:
 * [Input](https://input.com/) A collaborative note taking app. :star2:
 * [Room.sh](https://room.sh/) A meeting application with integrated
   collaborative drawing, editing, and coding tools. :star:
-* [http://coronavirustechhandbook.com/](https://coronavirustechhandbook.com/)
+* [https://coronavirustechhandbook.com/](https://coronavirustechhandbook.com/)
   A collaborative wiki that is edited by thousands of different people to work
   on a rapid and sophisticated response to the coronavirus outbreak and
   subsequent impacts. :star:
@@ -90,10 +90,10 @@ are implemented in separate modules.
 
 | Name | Cursors | Binding |  Demo |
 |---|:-:|---|---|
-| [ProseMirror](https://prosemirror.net/) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | ✔ | [y-prosemirror](http://github.com/yjs/y-prosemirror) | [demo](https://demos.yjs.dev/prosemirror/prosemirror.html) |
-| [Quill](https://quilljs.com/) | ✔ | [y-quill](http://github.com/yjs/y-quill) | [demo](https://demos.yjs.dev/quill/quill.html) |
-| [CodeMirror](https://codemirror.net/) | ✔ | [y-codemirror](http://github.com/yjs/y-codemirror) | [demo](https://demos.yjs.dev/codemirror/codemirror.html) |
-| [Monaco](https://microsoft.github.io/monaco-editor/) | ✔ | [y-monaco](http://github.com/yjs/y-monaco) | [demo](https://demos.yjs.dev/monaco/monaco.html) |
+| [ProseMirror](https://prosemirror.net/) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | ✔ | [y-prosemirror](https://github.com/yjs/y-prosemirror) | [demo](https://demos.yjs.dev/prosemirror/prosemirror.html) |
+| [Quill](https://quilljs.com/) | ✔ | [y-quill](https://github.com/yjs/y-quill) | [demo](https://demos.yjs.dev/quill/quill.html) |
+| [CodeMirror](https://codemirror.net/) | ✔ | [y-codemirror](https://github.com/yjs/y-codemirror) | [demo](https://demos.yjs.dev/codemirror/codemirror.html) |
+| [Monaco](https://microsoft.github.io/monaco-editor/) | ✔ | [y-monaco](https://github.com/yjs/y-monaco) | [demo](https://demos.yjs.dev/monaco/monaco.html) |
 
 ### Providers
 
@@ -103,7 +103,7 @@ manage all that for you and are the perfect starting point for your
 collaborative app.
 
 <dl>
-  <dt><a href="http://github.com/yjs/y-webrtc">y-webrtc</a></dt>
+  <dt><a href="https://github.com/yjs/y-webrtc">y-webrtc</a></dt>
   <dd>
 Propagates document updates peer-to-peer using WebRTC. The peers exchange
 signaling data over signaling servers. Publically available signaling servers
@@ -111,19 +111,19 @@ are available. Communication over the signaling servers can be encrypted by
 providing a shared secret, keeping the connection information and the shared
 document private.
   </dd>
-  <dt><a href="http://github.com/yjs/y-websocket">y-websocket</a></dt>
+  <dt><a href="https://github.com/yjs/y-websocket">y-websocket</a></dt>
   <dd>
 A module that contains a simple websocket backend and a websocket client that
 connects to that backend. The backend can be extended to persist updates in a
 leveldb database.
   </dd>
-  <dt><a href="http://github.com/yjs/y-indexeddb">y-indexeddb</a></dt>
+  <dt><a href="https://github.com/yjs/y-indexeddb">y-indexeddb</a></dt>
   <dd>
 Efficiently persists document updates to the browsers indexeddb database.
 The document is immediately available and only diffs need to be synced through the
 network provider.
   </dd>
-  <dt><a href="http://github.com/yjs/y-dat">y-dat</a></dt>
+  <dt><a href="https://github.com/yjs/y-dat">y-dat</a></dt>
   <dd>
 [WIP] Write document updates effinciently to the dat network using
 <a href="https://github.com/kappa-db/multifeed">multifeed</a>. Each client has
@@ -198,7 +198,7 @@ const ydoc = new Y.Doc()
 
 // this allows you to instantly get the (cached) documents data
 const indexeddbProvider = new IndexeddbPersistence('count-demo', ydoc)
-idbP.whenSynced.then(() => {
+indexeddbProvider.whenSynced.then(() => {
   console.log('loaded data from indexed db')
 })
 
@@ -255,6 +255,8 @@ position 0.
     <dd></dd>
     <b><code>get(index:number)</code></b>
     <dd></dd>
+    <b><code>slice(start:number, end:number):Array&lt;Object|boolean|Array|string|number|Uint8Array|Y.Type&gt;</code></b>
+    <dd>Retrieve a range of content</dd>
     <b><code>length:number</code></b>
     <dd></dd>
     <b>
@@ -320,6 +322,8 @@ or any of its children.
     <dd></dd>
     <b><code>get(index:number)</code></b>
     <dd></dd>
+    <b><code>clone():Y.Map</code></b>
+    <dd>Clone this type into a fresh Yjs type.</dd>
     <b><code>toJSON():Object&lt;string, Object|boolean|Array|string|number|Uint8Array&gt;</code></b>
     <dd>
 Copies the <code>[key,value]</code> pairs of this YMap to a new Object.It
@@ -451,8 +455,12 @@ or any of its children.
     <dd></dd>
     <b><code>get(index:number)</code></b>
     <dd></dd>
+    <b><code>slice(start:number, end:number):Array&lt;Y.XmlElement|Y.XmlText&gt;</code></b>
+    <dd>Retrieve a range of content</dd>
     <b><code>length:number</code></b>
     <dd></dd>
+    <b><code>clone():Y.XmlFragment</code></b>
+    <dd>Clone this type into a fresh Yjs type.</dd>
     <b><code>toArray():Array&lt;Y.XmlElement|Y.XmlText&gt;</code></b>
     <dd>Copies the children to a new Array.</dd>
     <b><code>toDOM():DocumentFragment</code></b>
@@ -512,6 +520,12 @@ content and be actually XML compliant.
     <dd></dd>
     <b><code>getAttributes(attributeName:string):Object&lt;string,string&gt;</code></b>
     <dd></dd>
+    <b><code>get(i:number):Y.XmlElement|Y.XmlText</code></b>
+    <dd>Retrieve the i-th element.</dd>
+    <b><code>slice(start:number, end:number):Array&lt;Y.XmlElement|Y.XmlText&gt;</code></b>
+    <dd>Retrieve a range of content</dd>
+    <b><code>clone():Y.XmlElement</code></b>
+    <dd>Clone this type into a fresh Yjs type.</dd>
     <b><code>toArray():Array&lt;Y.XmlElement|Y.XmlText&gt;</code></b>
     <dd>Copies the children to a new Array.</dd>
     <b><code>toDOM():Element</code></b>
@@ -571,7 +585,7 @@ triggers a single change event. <br>You can specify an optional <code>origin</co
 parameter that is stored on <code>transaction.origin</code> and
 <code>on('update', (update, origin) => ..)</code>.
   </dd>
-  <b><code>toJSON():any</code><b>
+  <b><code>toJSON():any</code></b>
   <dd>
 Converts the entire document into a js object, recursively traversing each yjs type.
   </dd>
@@ -892,11 +906,14 @@ do not require a central source of truth.
 
 Yjs implements a modified version of the algorithm described in [this
 paper](https://www.researchgate.net/publication/310212186_Near_Real-Time_Peer-to-Peer_Shared_Editing_on_Extensible_Data_Types).
-I will eventually publish a paper that describes why this approach works so well
-in practice. Note: Since operations make up the document structure, we prefer
-the term *struct* now.
+This [article](https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/)
+explains a simple optimization on the CRDT model and
+gives more insight about the performance characteristics in Yjs.
+More information about the specific implementation is available in
+[INTERNALS.md](./INTERNALS.md) and in
+[this walkthrough of the Yjs codebase](https://youtu.be/0l5XgnQ6rB4).
 
-CRDTs suitable for shared text editing suffer from the fact that they only grow
+CRDTs that suitable for shared text editing suffer from the fact that they only grow
 in size. There are CRDTs that do not grow in size, but they do not have the
 characteristics that are benificial for shared text editing (like intention
 preservation). Yjs implements many improvements to the original algorithm that

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "yjs",
-  "version": "13.3.2",
+  "version": "13.4.6",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {
@@ -31,9 +31,9 @@
       }
     },
     "@babel/parser": {
-      "version": "7.10.2",
-      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.10.2.tgz",
-      "integrity": "sha512-PApSXlNMJyB4JiGVhCOlzKIif+TKFTvu0aQAhnTvfP/z3vVSN6ZypH5bfUNwFXXjRQtUEBNFd2PtmCmG2Py3qQ==",
+      "version": "7.11.5",
+      "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.11.5.tgz",
+      "integrity": "sha512-X9rD8qqm695vgmeaQ4fvz/o3+Wk4ZzQvSHkDBgpYKxpD4qTAUm88ZKtHkVqIOsYFFbIQ6wQYhC6q7pjqVK0E0Q==",
       "dev": true
     },
     "@rollup/plugin-commonjs": {
@@ -1459,9 +1459,9 @@
       }
     },
     "jsdoc": {
-      "version": "3.6.4",
-      "resolved": "https://registry.npmjs.org/jsdoc/-/jsdoc-3.6.4.tgz",
-      "integrity": "sha512-3G9d37VHv7MFdheviDCjUfQoIjdv4TC5zTTf5G9VODLtOnVS6La1eoYBDlbWfsRT3/Xo+j2MIqki2EV12BZfwA==",
+      "version": "3.6.5",
+      "resolved": "https://registry.npmjs.org/jsdoc/-/jsdoc-3.6.5.tgz",
+      "integrity": "sha512-SbY+i9ONuxSK35cgVHaI8O9senTE4CDYAmGSDJ5l3+sfe62Ff4gy96osy6OW84t4K4A8iGnMrlRrsSItSNp3RQ==",
       "dev": true,
       "requires": {
         "@babel/parser": "^7.9.4",
@@ -1548,9 +1548,9 @@
       }
     },
     "lib0": {
-      "version": "0.2.32",
-      "resolved": "https://registry.npmjs.org/lib0/-/lib0-0.2.32.tgz",
-      "integrity": "sha512-cHHKhHTojtvFSsthTk+CKuD17jMHIxuZxYpTzXj9TeQLPNoGNDPl6ax+J6eFETVe3ZvPMh3V0nGfJgGo6QgSvA==",
+      "version": "0.2.33",
+      "resolved": "https://registry.npmjs.org/lib0/-/lib0-0.2.33.tgz",
+      "integrity": "sha512-Pnm8FzjUr+aTYkEu2A20c1EfVHla8GbVX+GXn6poxx0gcmEuCs+XszjLmtEbI9xYOoI/83xVi7VOIoyHgOO87w==",
       "requires": {
         "isomorphic.js": "^0.1.3"
       }
@@ -2786,9 +2786,9 @@
       "dev": true
     },
     "typescript": {
-      "version": "3.9.6",
-      "resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.6.tgz",
-      "integrity": "sha512-Pspx3oKAPJtjNwE92YS05HQoY7z2SFyOpHo9MqJor3BXAGNaPUs83CuVp9VISFkSjyRfiTpmKuAYGJB7S7hOxw==",
+      "version": "3.9.7",
+      "resolved": "https://registry.npmjs.org/typescript/-/typescript-3.9.7.tgz",
+      "integrity": "sha512-BLbiRkiBzAwsjut4x/dsibSTB6yWpwT5qWmC2OfuCg3GgVQCSgMs4vEctYPhsaGtd0AeuuHMkjZ2h2WG8MSzRw==",
       "dev": true
     },
     "uc.micro": {

package.json

@@ -1,6 +1,6 @@
 {
   "name": "yjs",
-  "version": "13.3.2",
+  "version": "13.4.6",
   "description": "Shared Editing Library",
   "main": "./dist/yjs.cjs",
   "module": "./dist/yjs.mjs",
@@ -61,20 +61,20 @@
   },
   "homepage": "https://yjs.dev",
   "dependencies": {
-    "lib0": "^0.2.32"
+    "lib0": "^0.2.33"
   },
   "devDependencies": {
     "@rollup/plugin-commonjs": "^11.1.0",
     "@rollup/plugin-node-resolve": "^7.1.3",
     "concurrently": "^3.6.1",
     "http-server": "^0.12.3",
-    "jsdoc": "^3.6.4",
+    "jsdoc": "^3.6.5",
     "markdownlint-cli": "^0.23.2",
     "rollup": "^1.32.1",
     "rollup-cli": "^1.0.9",
     "standard": "^14.3.4",
     "tui-jsdoc-template": "^1.2.2",
-    "typescript": "^3.9.6",
+    "typescript": "^3.9.7",
     "y-protocols": "^0.2.3"
   }
 }

src/index.js

@@ -45,8 +45,10 @@ export {
   snapshot,
   emptySnapshot,
   findRootTypeKey,
+  getItem,
   typeListToArraySnapshot,
   typeMapGetSnapshot,
+  createDocFromSnapshot,
   iterateDeletedStructs,
   applyUpdate,
   applyUpdateV2,

src/internals.js

@@ -31,6 +31,7 @@ export * from './structs/AbstractStruct.js'
 export * from './structs/GC.js'
 export * from './structs/ContentBinary.js'
 export * from './structs/ContentDeleted.js'
+export * from './structs/ContentDoc.js'
 export * from './structs/ContentEmbed.js'
 export * from './structs/ContentFormat.js'
 export * from './structs/ContentJSON.js'

src/structs/ContentDoc.js

@@ -0,0 +1,135 @@
+
+import {
+  Doc, AbstractUpdateDecoder, AbstractUpdateEncoder, StructStore, Transaction, Item // eslint-disable-line
+} from '../internals.js'
+
+import * as error from 'lib0/error.js'
+
+/**
+ * @private
+ */
+export class ContentDoc {
+  /**
+   * @param {Doc} doc
+   */
+  constructor (doc) {
+    if (doc._item) {
+      console.error('This document was already integrated as a sub-document. You should create a second instance instead with the same guid.')
+    }
+    /**
+     * @type {Doc}
+     */
+    this.doc = doc
+    /**
+     * @type {any}
+     */
+    const opts = {}
+    this.opts = opts
+    if (!doc.gc) {
+      opts.gc = false
+    }
+    if (doc.autoLoad) {
+      opts.autoLoad = true
+    }
+    if (doc.meta !== null) {
+      opts.meta = doc.meta
+    }
+  }
+
+  /**
+   * @return {number}
+   */
+  getLength () {
+    return 1
+  }
+
+  /**
+   * @return {Array<any>}
+   */
+  getContent () {
+    return [this.doc]
+  }
+
+  /**
+   * @return {boolean}
+   */
+  isCountable () {
+    return true
+  }
+
+  /**
+   * @return {ContentDoc}
+   */
+  copy () {
+    return new ContentDoc(this.doc)
+  }
+
+  /**
+   * @param {number} offset
+   * @return {ContentDoc}
+   */
+  splice (offset) {
+    throw error.methodUnimplemented()
+  }
+
+  /**
+   * @param {ContentDoc} right
+   * @return {boolean}
+   */
+  mergeWith (right) {
+    return false
+  }
+
+  /**
+   * @param {Transaction} transaction
+   * @param {Item} item
+   */
+  integrate (transaction, item) {
+    // this needs to be reflected in doc.destroy as well
+    this.doc._item = item
+    transaction.subdocsAdded.add(this.doc)
+    if (this.doc.shouldLoad) {
+      transaction.subdocsLoaded.add(this.doc)
+    }
+  }
+
+  /**
+   * @param {Transaction} transaction
+   */
+  delete (transaction) {
+    if (transaction.subdocsAdded.has(this.doc)) {
+      transaction.subdocsAdded.delete(this.doc)
+    } else {
+      transaction.subdocsRemoved.add(this.doc)
+    }
+  }
+
+  /**
+   * @param {StructStore} store
+   */
+  gc (store) { }
+
+  /**
+   * @param {AbstractUpdateEncoder} encoder
+   * @param {number} offset
+   */
+  write (encoder, offset) {
+    encoder.writeString(this.doc.guid)
+    encoder.writeAny(this.opts)
+  }
+
+  /**
+   * @return {number}
+   */
+  getRef () {
+    return 9
+  }
+}
+
+/**
+ * @private
+ *
+ * @param {AbstractUpdateDecoder} decoder
+ * @return {ContentDoc}
+ */
+export const readContentDoc = decoder => new ContentDoc(new Doc({ guid: decoder.readString(), ...decoder.readAny() }))

src/structs/ContentString.js

@@ -51,6 +51,17 @@ export class ContentString {
   splice (offset) {
     const right = new ContentString(this.str.slice(offset))
     this.str = this.str.slice(0, offset)
+
+    // Prevent encoding invalid documents because of splitting of surrogate pairs: https://github.com/yjs/yjs/issues/248
+    const firstCharCode = this.str.charCodeAt(offset - 1)
+    if (firstCharCode >= 0xD800 && firstCharCode <= 0xDBFF) {
+      // Last character of the left split is the start of a surrogate utf16/ucs2 pair.
+      // We don't support splitting of surrogate pairs because this may lead to invalid documents.
+      // Replace the invalid character with a unicode replacement character (<28> / U+FFFD)
+      this.str = this.str.slice(0, offset - 1) + '<27>'
+      // replace right as well
+      right.str = '<27>' + right.str.slice(1)
+    }
     return right
   }
 

src/structs/Item.js

@@ -17,6 +17,7 @@ import {
   readContentAny,
   readContentString,
   readContentEmbed,
+  readContentDoc,
   createID,
   readContentFormat,
   readContentType,
@@ -25,8 +26,6 @@ import {
 } from '../internals.js'
 
 import * as error from 'lib0/error.js'
-import * as maplib from 'lib0/map.js'
-import * as set from 'lib0/set.js'
 import * as binary from 'lib0/binary.js'
 
 /**
@@ -593,7 +592,7 @@ export class Item extends AbstractStruct {
       }
       this.markDeleted()
       addToDeleteSet(transaction.deleteSet, this.id.client, this.id.clock, this.length)
-      maplib.setIfUndefined(transaction.changed, parent, set.create).add(this.parentSub)
+      addChangedTypeToTransaction(transaction, parent, this.parentSub)
       this.content.delete(transaction)
     }
   }
@@ -672,14 +671,15 @@ export const readItemContent = (decoder, info) => contentRefs[info & binary.BITS
  */
 export const contentRefs = [
   () => { throw error.unexpectedCase() }, // GC is not ItemContent
-  readContentDeleted,
-  readContentJSON,
-  readContentBinary,
-  readContentString,
-  readContentEmbed,
-  readContentFormat,
-  readContentType,
-  readContentAny
+  readContentDeleted, // 1
+  readContentJSON, // 2
+  readContentBinary, // 3
+  readContentString, // 4
+  readContentEmbed, // 5
+  readContentFormat, // 6
+  readContentType, // 7
+  readContentAny, // 8
+  readContentDoc // 9
 ]
 
 /**

src/types/AbstractType.js

@@ -11,7 +11,7 @@ import {
   ContentAny,
   ContentBinary,
   getItemCleanStart,
-  YText, YArray, AbstractUpdateEncoder, Doc, Snapshot, Transaction, EventHandler, YEvent, Item, // eslint-disable-line
+  ContentDoc, YText, YArray, AbstractUpdateEncoder, Doc, Snapshot, Transaction, EventHandler, YEvent, Item, // eslint-disable-line
 } from '../internals.js'
 
 import * as map from 'lib0/map.js'
@@ -309,6 +309,13 @@ export class AbstractType {
     throw error.methodUnimplemented()
   }
 
+  /**
+   * @return {AbstractType<EventType>}
+   */
+  clone () {
+    throw error.methodUnimplemented()
+  }
+
   /**
    * @param {AbstractUpdateEncoder} encoder
    */
@@ -381,6 +388,43 @@ export class AbstractType {
   toJSON () {}
 }
 
+/**
+ * @param {AbstractType<any>} type
+ * @param {number} start
+ * @param {number} end
+ * @return {Array<any>}
+ *
+ * @private
+ * @function
+ */
+export const typeListSlice = (type, start, end) => {
+  if (start < 0) {
+    start = type._length + start
+  }
+  if (end < 0) {
+    end = type._length + end
+  }
+  let len = end - start
+  const cs = []
+  let n = type._start
+  while (n !== null && len > 0) {
+    if (n.countable && !n.deleted) {
+      const c = n.content.getContent()
+      if (c.length <= start) {
+        start -= c.length
+      } else {
+        for (let i = start; i < c.length && len > 0; i++) {
+          cs.push(c[i])
+          len--
+        }
+        start = 0
+      }
+    }
+    n = n.right
+  }
+  return cs
+}
+
 /**
  * @param {AbstractType<any>} type
  * @return {Array<any>}
@@ -611,6 +655,10 @@ export const typeListInsertGenericsAfter = (transaction, parent, referenceItem,
             left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentBinary(new Uint8Array(/** @type {Uint8Array} */ (c))))
             left.integrate(transaction, 0)
             break
+          case Doc:
+            left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentDoc(/** @type {Doc} */ (c)))
+            left.integrate(transaction, 0)
+            break
           default:
             if (c instanceof AbstractType) {
               left = new Item(createID(ownClientId, getState(store, ownClientId)), left, left && left.lastId, right, right && right.id, parent, null, new ContentType(c))
@@ -761,6 +809,9 @@ export const typeMapSet = (transaction, parent, key, value) => {
       case Uint8Array:
         content = new ContentBinary(/** @type {Uint8Array} */ (value))
         break
+      case Doc:
+        content = new ContentDoc(/** @type {Doc} */ (value))
+        break
       default:
         if (value instanceof AbstractType) {
           content = new ContentType(value)

src/types/YArray.js

@@ -17,6 +17,7 @@ import {
   transact,
   ArraySearchMarker, AbstractUpdateDecoder, AbstractUpdateEncoder, Doc, Transaction, Item // eslint-disable-line
 } from '../internals.js'
+import { typeListSlice } from './AbstractType.js'
 
 /**
  * Event that describes the changes on a YArray
@@ -53,6 +54,18 @@ export class YArray extends AbstractType {
     this._searchMarker = []
   }
 
+  /**
+   * Construct a new YArray containing the specified items.
+   * @template T
+   * @param {Array<T>} items
+   * @return {YArray<T>}
+   */
+  static from (items) {
+    const a = new YArray()
+    a.push(items)
+    return a
+  }
+
   /**
    * Integrate this type into the Yjs instance.
    *
@@ -73,6 +86,17 @@ export class YArray extends AbstractType {
     return new YArray()
   }
 
+  /**
+   * @return {YArray<T>}
+   */
+  clone () {
+    const arr = new YArray()
+    arr.insert(0, this.toArray().map(el =>
+      el instanceof AbstractType ? el.clone() : el
+    ))
+    return arr
+  }
+
   get length () {
     return this._prelimContent === null ? this._length : this._prelimContent.length
   }
@@ -167,6 +191,17 @@ export class YArray extends AbstractType {
     return typeListToArray(this)
   }
 
+  /**
+   * Transforms this YArray to a JavaScript Array.
+   *
+   * @param {number} [start]
+   * @param {number} [end]
+   * @return {Array<T>}
+   */
+  slice (start = 0, end = this.length) {
+    return typeListSlice(this, start, end)
+  }
+
   /**
    * Transforms this Shared Type to a JSON object.
    *

src/types/YMap.js

@@ -84,6 +84,17 @@ export class YMap extends AbstractType {
     return new YMap()
   }
 
+  /**
+   * @return {YMap<T>}
+   */
+  clone () {
+    const map = new YMap()
+    this.forEach((value, key) => {
+      map.set(key, value instanceof AbstractType ? value.clone() : value)
+    })
+    return map
+  }
+
   /**
    * Creates YMapEvent and calls observers.
    *
@@ -132,9 +143,9 @@ export class YMap extends AbstractType {
   }
 
   /**
-   * Returns the keys for each element in the YMap Type.
+   * Returns the values for each element in the YMap Type.
    *
-   * @return {IterableIterator<string>}
+   * @return {IterableIterator<any>}
    */
   values () {
     return iterator.iteratorMap(createMapIterator(this._map), /** @param {any} v */ v => v[1].content.getContent()[v[1].length - 1])

src/types/YText.js

@@ -501,7 +501,7 @@ const deleteText = (transaction, currPos, length) => {
  * @typedef {Object} DeltaItem
  * @property {number|undefined} DeltaItem.delete
  * @property {number|undefined} DeltaItem.retain
- * @property {string|undefined} DeltaItem.string
+ * @property {string|undefined} DeltaItem.insert
  * @property {Object<string,any>} DeltaItem.attributes
  */
 
@@ -762,6 +762,15 @@ export class YText extends AbstractType {
     return new YText()
   }
 
+  /**
+   * @return {YText}
+   */
+  clone () {
+    const text = new YText()
+    text.applyDelta(this.toDelta())
+    return text
+  }
+
   /**
    * Creates YTextEvent and calls observers.
    *

src/types/YXmlElement.js

@@ -8,7 +8,7 @@ import {
   typeMapGetAll,
   typeListForEach,
   YXmlElementRefID,
-  AbstractUpdateDecoder, AbstractUpdateEncoder, Snapshot, Doc, Item // eslint-disable-line
+  AbstractType, AbstractUpdateDecoder, AbstractUpdateEncoder, Snapshot, Doc, Item // eslint-disable-line
 } from '../internals.js'
 
 /**
@@ -55,6 +55,20 @@ export class YXmlElement extends YXmlFragment {
     return new YXmlElement(this.nodeName)
   }
 
+  /**
+   * @return {YXmlElement}
+   */
+  clone () {
+    const el = new YXmlElement(this.nodeName)
+    const attrs = this.getAttributes()
+    for (const key in attrs) {
+      el.setAttribute(key, attrs[key])
+    }
+    // @ts-ignore
+    el.insert(0, el.toArray().map(item => item instanceof AbstractType ? item.clone() : item))
+    return el
+  }
+
   /**
    * Returns the XML serialization of this YXmlElement.
    * The attributes are ordered by attribute-name, so you can easily use this

src/types/YXmlFragment.js

@@ -14,6 +14,8 @@ import {
   YXmlFragmentRefID,
   callTypeObservers,
   transact,
+  typeListGet,
+  typeListSlice,
   AbstractUpdateDecoder, AbstractUpdateEncoder, Doc, ContentType, Transaction, Item, YXmlText, YXmlHook, Snapshot // eslint-disable-line
 } from '../internals.js'
 
@@ -148,6 +150,16 @@ export class YXmlFragment extends AbstractType {
     return new YXmlFragment()
   }
 
+  /**
+   * @return {YXmlFragment}
+   */
+  clone () {
+    const el = new YXmlFragment()
+    // @ts-ignore
+    el.insert(0, el.toArray().map(item => item instanceof AbstractType ? item.clone() : item))
+    return el
+  }
+
   get length () {
     return this._prelimContent === null ? this._length : this._prelimContent.length
   }
@@ -316,6 +328,45 @@ export class YXmlFragment extends AbstractType {
     return typeListToArray(this)
   }
 
+  /**
+   * Appends content to this YArray.
+   *
+   * @param {Array<YXmlElement|YXmlText>} content Array of content to append.
+   */
+  push (content) {
+    this.insert(this.length, content)
+  }
+
+  /**
+   * Preppends content to this YArray.
+   *
+   * @param {Array<YXmlElement|YXmlText>} content Array of content to preppend.
+   */
+  unshift (content) {
+    this.insert(0, content)
+  }
+
+  /**
+   * Returns the i-th element from a YArray.
+   *
+   * @param {number} index The index of the element to return from the YArray
+   * @return {YXmlElement|YXmlText}
+   */
+  get (index) {
+    return typeListGet(this, index)
+  }
+
+  /**
+   * Transforms this YArray to a JavaScript Array.
+   *
+   * @param {number} [start]
+   * @param {number} [end]
+   * @return {Array<YXmlElement|YXmlText>}
+   */
+  slice (start = 0, end = this.length) {
+    return typeListSlice(this, start, end)
+  }
+
   /**
    * Transform the properties of this type to binary and write it to an
    * BinaryEncoder.

src/types/YXmlHook.js

@@ -29,6 +29,17 @@ export class YXmlHook extends YMap {
     return new YXmlHook(this.hookName)
   }
 
+  /**
+   * @return {YXmlHook}
+   */
+  clone () {
+    const el = new YXmlHook(this.hookName)
+    this.forEach((value, key) => {
+      el.set(key, value)
+    })
+    return el
+  }
+
   /**
    * Creates a Dom Element that mirrors this YXmlElement.
    *

src/types/YXmlText.js

@@ -14,6 +14,15 @@ export class YXmlText extends YText {
     return new YXmlText()
   }
 
+  /**
+   * @return {YXmlText}
+   */
+  clone () {
+    const text = new YXmlText()
+    text.applyDelta(this.toDelta())
+    return text
+  }
+
   /**
    * Creates a Dom Element that mirrors this YXmlText.
    *

src/utils/Doc.js

@@ -10,30 +10,39 @@ import {
   YMap,
   YXmlFragment,
   transact,
-  Item, Transaction, YEvent // eslint-disable-line
+  ContentDoc, Item, Transaction, YEvent // eslint-disable-line
 } from '../internals.js'
 
 import { Observable } from 'lib0/observable.js'
 import * as random from 'lib0/random.js'
 import * as map from 'lib0/map.js'
+import * as array from 'lib0/array.js'
 
 export const generateNewClientId = random.uint32
 
+/**
+ * @typedef {Object} DocOpts
+ * @property {boolean} [DocOpts.gc=true] Disable garbage collection (default: gc=true)
+ * @property {function(Item):boolean} [DocOpts.gcFilter] Will be called before an Item is garbage collected. Return false to keep the Item.
+ * @property {string} [DocOpts.guid] Define a globally unique identifier for this document
+ * @property {any} [DocOpts.meta] Any kind of meta information you want to associate with this document. If this is a subdocument, remote peers will store the meta information as well.
+ * @property {boolean} [DocOpts.autoLoad] If a subdocument, automatically load document. If this is a subdocument, remote peers will load the document as well automatically.
+ */
+
 /**
  * A Yjs instance handles the state of shared data.
  * @extends Observable<string>
  */
 export class Doc extends Observable {
   /**
-   * @param {Object} conf configuration
-   * @param {boolean} [conf.gc] Disable garbage collection (default: gc=true)
-   * @param {function(Item):boolean} [conf.gcFilter] Will be called before an Item is garbage collected. Return false to keep the Item.
+   * @param {DocOpts} [opts] configuration
    */
-  constructor ({ gc = true, gcFilter = () => true } = {}) {
+  constructor ({ guid = random.uuidv4(), gc = true, gcFilter = () => true, meta = null, autoLoad = false } = {}) {
     super()
     this.gc = gc
     this.gcFilter = gcFilter
     this.clientID = generateNewClientId()
+    this.guid = guid
     /**
      * @type {Map<string, AbstractType<YEvent>>}
      */
@@ -47,6 +56,43 @@ export class Doc extends Observable {
      * @type {Array<Transaction>}
      */
     this._transactionCleanups = []
+    /**
+     * @type {Set<Doc>}
+     */
+    this.subdocs = new Set()
+    /**
+     * If this document is a subdocument - a document integrated into another document - then _item is defined.
+     * @type {Item?}
+     */
+    this._item = null
+    this.shouldLoad = autoLoad
+    this.autoLoad = autoLoad
+    this.meta = meta
+  }
+
+  /**
+   * Notify the parent document that you request to load data into this subdocument (if it is a subdocument).
+   *
+   * `load()` might be used in the future to request any provider to load the most current data.
+   *
+   * It is safe to call `load()` multiple times.
+   */
+  load () {
+    const item = this._item
+    if (item !== null && !this.shouldLoad) {
+      transact(/** @type {any} */ (item.parent).doc, transaction => {
+        transaction.subdocsLoaded.add(this)
+      }, null, true)
+    }
+    this.shouldLoad = true
+  }
+
+  getSubdocs () {
+    return this.subdocs
+  }
+
+  getSubdocGuids () {
+    return new Set(Array.from(this.subdocs).map(doc => doc.guid))
   }
 
   /**
@@ -191,13 +237,33 @@ export class Doc extends Observable {
    * Emit `destroy` event and unregister all event handlers.
    */
   destroy () {
+    array.from(this.subdocs).forEach(subdoc => subdoc.destroy())
+    const item = this._item
+    if (item !== null) {
+      this._item = null
+      const content = /** @type {ContentDoc} */ (item.content)
+      if (item.deleted) {
+        // @ts-ignore
+        content.doc = null
+      } else {
+        content.doc = new Doc({ guid: this.guid, ...content.opts })
+        content.doc._item = item
+      }
+      transact(/** @type {any} */ (item).parent.doc, transaction => {
+        if (!item.deleted) {
+          transaction.subdocsAdded.add(content.doc)
+        }
+        transaction.subdocsRemoved.add(this)
+      }, null, true)
+    }
     this.emit('destroyed', [true])
+    this.emit('destroy', [this])
     super.destroy()
   }
 
   /**
    * @param {string} eventName
-   * @param {function} f
+   * @param {function(...any):any} f
    */
   on (eventName, f) {
     super.on(eventName, f)

src/utils/EventHandler.js

@@ -51,7 +51,12 @@ export const addEventHandlerListener = (eventHandler, f) =>
  * @function
  */
 export const removeEventHandlerListener = (eventHandler, f) => {
-  eventHandler.l = eventHandler.l.filter(g => f !== g)
+  const l = eventHandler.l
+  const len = l.length
+  eventHandler.l = l.filter(g => f !== g)
+  if (len === eventHandler.l.length) {
+    console.error('[yjs] Tried to remove event handler that doesn\'t exist.')
+  }
 }
 
 /**

src/utils/PermanentUserData.js

@@ -51,7 +51,7 @@ export class PermanentUserData {
           })
         })
       })
-      this.dss.set(userDescription, mergeDeleteSets(ds.map(encodedDs => readDeleteSet(new DSDecoderV1(encodedDs)))))
+      this.dss.set(userDescription, mergeDeleteSets(ds.map(encodedDs => readDeleteSet(new DSDecoderV1(decoding.createDecoder(encodedDs))))))
       ids.observe(/** @param {YArrayEvent<any>} event */ event =>
         event.changes.added.forEach(item => item.content.getContent().forEach(addClientId))
       )

src/utils/Snapshot.js

@@ -12,13 +12,17 @@ import {
   createDeleteSet,
   createID,
   getState,
-  AbstractDSDecoder, AbstractDSEncoder, DSEncoderV1, DSEncoderV2, DSDecoderV1, DSDecoderV2, Transaction, Doc, DeleteSet, Item // eslint-disable-line
+  findIndexSS,
+  UpdateEncoderV2,
+  DefaultDSEncoder,
+  applyUpdateV2,
+  AbstractDSDecoder, AbstractDSEncoder, DSEncoderV2, DSDecoderV1, DSDecoderV2, Transaction, Doc, DeleteSet, Item // eslint-disable-line
 } from '../internals.js'
 
 import * as map from 'lib0/map.js'
 import * as set from 'lib0/set.js'
 import * as decoding from 'lib0/decoding.js'
-import { DefaultDSEncoder } from './encoding.js'
+import * as encoding from 'lib0/encoding.js'
 
 export class Snapshot {
   /**
@@ -148,3 +152,51 @@ export const splitSnapshotAffectedStructs = (transaction, snapshot) => {
     meta.add(snapshot)
   }
 }
+
+/**
+ * @param {Doc} originDoc
+ * @param {Snapshot} snapshot
+ * @param {Doc} [newDoc] Optionally, you may define the Yjs document that receives the data from originDoc
+ * @return {Doc}
+ */
+export const createDocFromSnapshot = (originDoc, snapshot, newDoc = new Doc()) => {
+  if (originDoc.gc) {
+    // we should not try to restore a GC-ed document, because some of the restored items might have their content deleted
+    throw new Error('originDoc must not be garbage collected')
+  }
+  const { sv, ds } = snapshot
+
+  const encoder = new UpdateEncoderV2()
+  originDoc.transact(transaction => {
+    let size = 0
+    sv.forEach(clock => {
+      if (clock > 0) {
+        size++
+      }
+    })
+    encoding.writeVarUint(encoder.restEncoder, size)
+    // splitting the structs before writing them to the encoder
+    for (const [client, clock] of sv) {
+      if (clock === 0) {
+        continue
+      }
+      if (clock < getState(originDoc.store, client)) {
+        getItemCleanStart(transaction, createID(client, clock))
+      }
+      const structs = originDoc.store.clients.get(client) || []
+      const lastStructIndex = findIndexSS(structs, clock - 1)
+      // write # encoded structs
+      encoding.writeVarUint(encoder.restEncoder, lastStructIndex + 1)
+      encoder.writeClient(client)
+      // first clock written is 0
+      encoding.writeVarUint(encoder.restEncoder, 0)
+      for (let i = 0; i <= lastStructIndex; i++) {
+        structs[i].write(encoder, 0)
+      }
+    }
+    writeDeleteSet(encoder, ds)
+  })
+
+  applyUpdateV2(newDoc, encoder.toUint8Array(), 'snapshot')
+  return newDoc
+}

src/utils/Transaction.js

@@ -102,6 +102,18 @@ export class Transaction {
      * @type {boolean}
      */
     this.local = local
+    /**
+     * @type {Set<Doc>}
+     */
+    this.subdocsAdded = new Set()
+    /**
+     * @type {Set<Doc>}
+     */
+    this.subdocsRemoved = new Set()
+    /**
+     * @type {Set<Doc>}
+     */
+    this.subdocsLoaded = new Set()
   }
 }
 
@@ -272,6 +284,9 @@ const cleanupTransactions = (transactionCleanups, i) => {
                 .forEach(event => {
                   event.currentTarget = type
                 })
+              // sort events by path length so that top-level events are fired first.
+              events
+                .sort((event1, event2) => event1.path.length - event2.path.length)
               // We don't need to check for events.length
               // because we know it has at least one element
               callEventHandlerListeners(type._dEH, events, transaction)
@@ -335,6 +350,12 @@ const cleanupTransactions = (transactionCleanups, i) => {
           doc.emit('updateV2', [encoder.toUint8Array(), transaction.origin, doc])
         }
       }
+      transaction.subdocsAdded.forEach(subdoc => doc.subdocs.add(subdoc))
+      transaction.subdocsRemoved.forEach(subdoc => doc.subdocs.delete(subdoc))
+
+      doc.emit('subdocs', [{ loaded: transaction.subdocsLoaded, added: transaction.subdocsAdded, removed: transaction.subdocsRemoved }])
+      transaction.subdocsRemoved.forEach(subdoc => subdoc.destroy())
+
       if (transactionCleanups.length <= i + 1) {
         doc._transactionCleanups = []
         doc.emit('afterAllTransactions', [doc, transactionCleanups])

src/utils/UndoManager.js

@@ -123,6 +123,12 @@ const popStackItem = (undoManager, stack, eventType) => {
         undoManager.emit('stack-item-popped', [{ stackItem: result, type: eventType }, undoManager])
       }
     }
+    transaction.changed.forEach((subProps, type) => {
+      // destroy search marker if necessary
+      if (subProps.has(null) && type._searchMarker) {
+        type._searchMarker.length = 0
+      }
+    })
   }, undoManager)
   return result
 }
@@ -151,10 +157,7 @@ export class UndoManager extends Observable {
    * @param {AbstractType<any>|Array<AbstractType<any>>} typeScope Accepts either a single type, or an array of types
    * @param {UndoManagerOptions} options
    */
-  constructor (typeScope, { captureTimeout, deleteFilter = () => true, trackedOrigins = new Set([null]) } = {}) {
-    if (captureTimeout == null) {
-      captureTimeout = 500
-    }
+  constructor (typeScope, { captureTimeout = 500, deleteFilter = () => true, trackedOrigins = new Set([null]) } = {}) {
     super()
     this.scope = typeScope instanceof Array ? typeScope : [typeScope]
     this.deleteFilter = deleteFilter

src/utils/YEvent.js

@@ -78,7 +78,7 @@ export class YEvent {
   }
 
   /**
-   * @return {{added:Set<Item>,deleted:Set<Item>,delta:Array<{insert:Array<any>}|{delete:number}|{retain:number}>}}
+   * @return {{added:Set<Item>,deleted:Set<Item>,keys:Map<string,{action:'add'|'update'|'delete',oldValue:any}>,delta:Array<{insert:Array<any>}|{delete:number}|{retain:number}>}}
    */
   get changes () {
     let changes = this._changes

tests/doc.tests.js

@@ -57,3 +57,70 @@ export const testToJSON = tc => {
     }
   }, 'doc.toJSON has array and recursive map')
 }
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testSubdoc = tc => {
+  const doc = new Y.Doc()
+  doc.load() // doesn't do anything
+  {
+    /**
+     * @type {Array<any>|null}
+     */
+    let event = /** @type {any} */ (null)
+    doc.on('subdocs', subdocs => {
+      event = [Array.from(subdocs.added).map(x => x.guid), Array.from(subdocs.removed).map(x => x.guid), Array.from(subdocs.loaded).map(x => x.guid)]
+    })
+    const subdocs = doc.getMap('mysubdocs')
+    const docA = new Y.Doc({ guid: 'a' })
+    docA.load()
+    subdocs.set('a', docA)
+    t.compare(event, [['a'], [], ['a']])
+
+    event = null
+    subdocs.get('a').load()
+    t.assert(event === null)
+
+    event = null
+    subdocs.get('a').destroy()
+    t.compare(event, [['a'], ['a'], []])
+    subdocs.get('a').load()
+    t.compare(event, [[], [], ['a']])
+
+    subdocs.set('b', new Y.Doc({ guid: 'a' }))
+    t.compare(event, [['a'], [], []])
+    subdocs.get('b').load()
+    t.compare(event, [[], [], ['a']])
+
+    const docC = new Y.Doc({ guid: 'c' })
+    docC.load()
+    subdocs.set('c', docC)
+    t.compare(event, [['c'], [], ['c']])
+
+    t.compare(Array.from(doc.getSubdocGuids()), ['a', 'c'])
+  }
+
+  const doc2 = new Y.Doc()
+  {
+    t.compare(Array.from(doc2.getSubdocs()), [])
+    /**
+     * @type {Array<any>|null}
+     */
+    let event = /** @type {any} */ (null)
+    doc2.on('subdocs', subdocs => {
+      event = [Array.from(subdocs.added).map(d => d.guid), Array.from(subdocs.removed).map(d => d.guid), Array.from(subdocs.loaded).map(d => d.guid)]
+    })
+    Y.applyUpdate(doc2, Y.encodeStateAsUpdate(doc))
+    t.compare(event, [['a', 'a', 'c'], [], []])
+
+    doc2.getMap('mysubdocs').get('a').load()
+    t.compare(event, [[], [], ['a']])
+
+    t.compare(Array.from(doc2.getSubdocGuids()), ['a', 'c'])
+
+    doc2.getMap('mysubdocs').delete('a')
+    t.compare(event, [[], ['a'], []])
+    t.compare(Array.from(doc2.getSubdocGuids()), ['a', 'c'])
+  }
+}

tests/encoding.tests.js

@@ -1,4 +1,5 @@
 import * as t from 'lib0/testing.js'
+import * as promise from 'lib0/promise.js'
 
 import {
   contentRefs,
@@ -9,14 +10,19 @@ import {
   readContentEmbed,
   readContentType,
   readContentFormat,
-  readContentAny
+  readContentAny,
+  readContentDoc,
+  Doc,
+  PermanentUserData,
+  encodeStateAsUpdate,
+  applyUpdate
 } from '../src/internals.js'
 
 /**
  * @param {t.TestCase} tc
  */
 export const testStructReferences = tc => {
-  t.assert(contentRefs.length === 9)
+  t.assert(contentRefs.length === 10)
   t.assert(contentRefs[1] === readContentDeleted)
   t.assert(contentRefs[2] === readContentJSON) // TODO: deprecate content json?
   t.assert(contentRefs[3] === readContentBinary)
@@ -25,4 +31,33 @@ export const testStructReferences = tc => {
   t.assert(contentRefs[6] === readContentFormat)
   t.assert(contentRefs[7] === readContentType)
   t.assert(contentRefs[8] === readContentAny)
+  t.assert(contentRefs[9] === readContentDoc)
+}
+
+/**
+ * There is some custom encoding/decoding happening in PermanentUserData.
+ * This is why it landed here.
+ *
+ * @param {t.TestCase} tc
+ */
+export const testPermanentUserData = async tc => {
+  const ydoc1 = new Doc()
+  const ydoc2 = new Doc()
+  const pd1 = new PermanentUserData(ydoc1)
+  const pd2 = new PermanentUserData(ydoc2)
+  pd1.setUserMapping(ydoc1, ydoc1.clientID, 'user a')
+  pd2.setUserMapping(ydoc2, ydoc2.clientID, 'user b')
+  ydoc1.getText().insert(0, 'xhi')
+  ydoc1.getText().delete(0, 1)
+  ydoc2.getText().insert(0, 'hxxi')
+  ydoc2.getText().delete(1, 2)
+  await promise.wait(10)
+  applyUpdate(ydoc2, encodeStateAsUpdate(ydoc1))
+  applyUpdate(ydoc1, encodeStateAsUpdate(ydoc2))
+
+  // now sync a third doc with same name as doc1 and then create PermanentUserData
+  const ydoc3 = new Doc()
+  applyUpdate(ydoc3, encodeStateAsUpdate(ydoc1))
+  const pd3 = new PermanentUserData(ydoc3)
+  pd3.setUserMapping(ydoc3, ydoc3.clientID, 'user a')
 }

tests/index.js

@@ -7,6 +7,7 @@ import * as encoding from './encoding.tests.js'
 import * as undoredo from './undo-redo.tests.js'
 import * as compatibility from './compatibility.tests.js'
 import * as doc from './doc.tests.js'
+import * as snapshot from './snapshot.tests.js'
 
 import { runTests } from 'lib0/testing.js'
 import { isBrowser, isNode } from 'lib0/environment.js'
@@ -16,7 +17,7 @@ if (isBrowser) {
   log.createVConsole(document.body)
 }
 runTests({
-  doc, map, array, text, xml, encoding, undoredo, compatibility
+  doc, map, array, text, xml, encoding, undoredo, compatibility, snapshot
 }).then(success => {
   /* istanbul ignore next */
   if (isNode) {

tests/snapshot.tests.js

@@ -0,0 +1,171 @@
+import { createDocFromSnapshot, Doc, snapshot, YMap } from '../src/internals'
+import * as t from 'lib0/testing.js'
+import { init } from './testHelper'
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testBasicRestoreSnapshot = tc => {
+  const doc = new Doc({ gc: false })
+  doc.getArray('array').insert(0, ['hello'])
+  const snap = snapshot(doc)
+  doc.getArray('array').insert(1, ['world'])
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray('array').toArray(), ['hello'])
+  t.compare(doc.getArray('array').toArray(), ['hello', 'world'])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testEmptyRestoreSnapshot = tc => {
+  const doc = new Doc({ gc: false })
+  const snap = snapshot(doc)
+  snap.sv.set(9999, 0)
+  doc.getArray().insert(0, ['world'])
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray().toArray(), [])
+  t.compare(doc.getArray().toArray(), ['world'])
+
+  // now this snapshot reflects the latest state. It shoult still work.
+  const snap2 = snapshot(doc)
+  const docRestored2 = createDocFromSnapshot(doc, snap2)
+  t.compare(docRestored2.getArray().toArray(), ['world'])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testRestoreSnapshotWithSubType = tc => {
+  const doc = new Doc({ gc: false })
+  doc.getArray('array').insert(0, [new YMap()])
+  const subMap = doc.getArray('array').get(0)
+  subMap.set('key1', 'value1')
+
+  const snap = snapshot(doc)
+  subMap.set('key2', 'value2')
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray('array').toJSON(), [{
+    key1: 'value1'
+  }])
+  t.compare(doc.getArray('array').toJSON(), [{
+    key1: 'value1',
+    key2: 'value2'
+  }])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testRestoreDeletedItem1 = tc => {
+  const doc = new Doc({ gc: false })
+  doc.getArray('array').insert(0, ['item1', 'item2'])
+
+  const snap = snapshot(doc)
+  doc.getArray('array').delete(0)
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray('array').toArray(), ['item1', 'item2'])
+  t.compare(doc.getArray('array').toArray(), ['item2'])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testRestoreLeftItem = tc => {
+  const doc = new Doc({ gc: false })
+  doc.getArray('array').insert(0, ['item1'])
+  doc.getMap('map').set('test', 1)
+  doc.getArray('array').insert(0, ['item0'])
+
+  const snap = snapshot(doc)
+  doc.getArray('array').delete(1)
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray('array').toArray(), ['item0', 'item1'])
+  t.compare(doc.getArray('array').toArray(), ['item0'])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testDeletedItemsBase = tc => {
+  const doc = new Doc({ gc: false })
+  doc.getArray('array').insert(0, ['item1'])
+  doc.getArray('array').delete(0)
+  const snap = snapshot(doc)
+  doc.getArray('array').insert(0, ['item0'])
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray('array').toArray(), [])
+  t.compare(doc.getArray('array').toArray(), ['item0'])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testDeletedItems2 = tc => {
+  const doc = new Doc({ gc: false })
+  doc.getArray('array').insert(0, ['item1', 'item2', 'item3'])
+  doc.getArray('array').delete(1)
+  const snap = snapshot(doc)
+  doc.getArray('array').insert(0, ['item0'])
+
+  const docRestored = createDocFromSnapshot(doc, snap)
+
+  t.compare(docRestored.getArray('array').toArray(), ['item1', 'item3'])
+  t.compare(doc.getArray('array').toArray(), ['item0', 'item1', 'item3'])
+}
+
+/**
+ * @param {t.TestCase} tc
+ */
+export const testDependentChanges = tc => {
+  const { array0, array1, testConnector } = init(tc, { users: 2 })
+
+  if (!array0.doc) {
+    throw new Error('no document 0')
+  }
+  if (!array1.doc) {
+    throw new Error('no document 1')
+  }
+
+  /**
+   * @type Doc
+   */
+  const doc0 = array0.doc
+  /**
+   * @type Doc
+   */
+  const doc1 = array1.doc
+
+  doc0.gc = false
+  doc1.gc = false
+
+  array0.insert(0, ['user1item1'])
+  testConnector.syncAll()
+  array1.insert(1, ['user2item1'])
+  testConnector.syncAll()
+
+  const snap = snapshot(array0.doc)
+
+  array0.insert(2, ['user1item2'])
+  testConnector.syncAll()
+  array1.insert(3, ['user2item2'])
+  testConnector.syncAll()
+
+  const docRestored0 = createDocFromSnapshot(array0.doc, snap)
+  t.compare(docRestored0.getArray('array').toArray(), ['user1item1', 'user2item1'])
+
+  const docRestored1 = createDocFromSnapshot(array1.doc, snap)
+  t.compare(docRestored1.getArray('array').toArray(), ['user1item1', 'user2item1'])
+}

tests/undo-redo.tests.js

@@ -53,6 +53,28 @@ export const testUndoText = tc => {
   t.compare(text0.toDelta(), [{ insert: 'b' }, { insert: 'cxy', attributes: { bold: true } }, { insert: 'z' }])
 }
 
+/**
+ * Test case to fix #241
+ * @param {t.TestCase} tc
+ */
+export const testDoubleUndo = tc => {
+  const doc = new Y.Doc()
+  const text = doc.getText()
+  text.insert(0, '1221')
+
+  const manager = new Y.UndoManager(text)
+
+  text.insert(2, '3')
+  text.insert(3, '3')
+
+  manager.undo()
+  manager.undo()
+
+  text.insert(2, '3')
+
+  t.compareStrings(text.toString(), '12321')
+}
+
 /**
  * @param {t.TestCase} tc
  */

tests/y-array.tests.js

@@ -17,6 +17,21 @@ export const testBasicUpdate = tc => {
   t.compare(doc2.getArray('array').toArray(), ['hi'])
 }
 
+/**
+ * @param {t.TestCase} tc
+ */
+export const testSlice = tc => {
+  const doc1 = new Y.Doc()
+  const arr = doc1.getArray('array')
+  arr.insert(0, [1, 2, 3])
+  t.compareArrays(arr.slice(0), [1, 2, 3])
+  t.compareArrays(arr.slice(1), [2, 3])
+  t.compareArrays(arr.slice(0, -1), [1, 2])
+  arr.insert(0, [0])
+  t.compareArrays(arr.slice(0), [0, 1, 2, 3])
+  t.compareArrays(arr.slice(0, 2), [0, 1])
+}
+
 /**
  * @param {t.TestCase} tc
  */
@@ -204,6 +219,34 @@ export const testInsertAndDeleteEventsForTypes = tc => {
   compare(users)
 }
 
+/**
+ * This issue has been reported in https://discuss.yjs.dev/t/order-in-which-events-yielded-by-observedeep-should-be-applied/261/2
+ *
+ * Deep observers generate multiple events. When an array added at item at, say, position 0,
+ * and item 1 changed then the array-add event should fire first so that the change event
+ * path is correct. A array binding might lead to an inconsistent state otherwise.
+ *
+ * @param {t.TestCase} tc
+ */
+export const testObserveDeepEventOrder = tc => {
+  const { array0, users } = init(tc, { users: 2 })
+  /**
+   * @type {Array<any>}
+   */
+  let events = []
+  array0.observeDeep(e => {
+    events = e
+  })
+  array0.insert(0, [new Y.Map()])
+  users[0].transact(() => {
+    array0.get(0).set('a', 'a')
+    array0.insert(0, [0])
+  })
+  for (let i = 1; i < events.length; i++) {
+    t.assert(events[i - 1].path.length <= events[i].path.length, 'path size increases, fire top-level events first')
+  }
+}
+
 /**
  * @param {t.TestCase} tc
  */

tests/y-text.tests.js

@@ -249,6 +249,8 @@ export const testAppendChars = tc => {
   t.assert(text0.length === N)
 }
 
+const largeDocumentSize = 100000
+
 const id = Y.createID(0, 0)
 const c = new Y.ContentString('a')
 
@@ -256,7 +258,7 @@ const c = new Y.ContentString('a')
  * @param {t.TestCase} tc
  */
 export const testBestCase = tc => {
-  const N = 2000000
+  const N = largeDocumentSize
   const items = new Array(N)
   t.measureTime('time to create two million items in the best case', () => {
     const parent = /** @type {any} */ ({})
@@ -293,7 +295,7 @@ const tryGc = () => {
  * @param {t.TestCase} tc
  */
 export const testLargeFragmentedDocument = tc => {
-  const itemsToInsert = 1000000
+  const itemsToInsert = largeDocumentSize
   let update = /** @type {any} */ (null)
   ;(() => {
     const doc1 = new Y.Doc()
@@ -321,6 +323,40 @@ export const testLargeFragmentedDocument = tc => {
   })()
 }
 
+/**
+ * Splitting surrogates can lead to invalid encoded documents.
+ *
+ * https://github.com/yjs/yjs/issues/248
+ *
+ * @param {t.TestCase} tc
+ */
+export const testSplitSurrogateCharacter = tc => {
+  {
+    const { users, text0 } = init(tc, { users: 2 })
+    users[1].disconnect() // disconnecting forces the user to encode the split surrogate
+    text0.insert(0, '👾') // insert surrogate character
+    // split surrogate, which should not lead to an encoding error
+    text0.insert(1, 'hi!')
+    compare(users)
+  }
+  {
+    const { users, text0 } = init(tc, { users: 2 })
+    users[1].disconnect() // disconnecting forces the user to encode the split surrogate
+    text0.insert(0, '👾👾') // insert surrogate character
+    // partially delete surrogate
+    text0.delete(1, 2)
+    compare(users)
+  }
+  {
+    const { users, text0 } = init(tc, { users: 2 })
+    users[1].disconnect() // disconnecting forces the user to encode the split surrogate
+    text0.insert(0, '👾👾') // insert surrogate character
+    // formatting will also split surrogates
+    text0.format(1, 2, { bold: true })
+    compare(users)
+  }
+}
+
 // RANDOM TESTS
 
 let charCounter = 0