- Make it clear that manually generating a Device ID is bad behavior

- Make DeviceID no longer required for matrix-server-config

.npmrc

@@ -0,0 +1 @@
+@matrix-org:registry=https://gitlab.matrix.org/api/v4/packages/npm/

README.md

@@ -3,6 +3,8 @@ Matrix chat server client for [Node-RED](https://nodered.org/)
 
 ***Currently we are in beta. We ask that you open any issues you have on our repository to help us reach a stable well tested version. Things may change & break before our first release so check changelog before updating.***
 
+If you need help with this feel free to join our public matrix room at [#node-red-contrib-matrix-chat:skylar.tech](https://app.element.io/#/room/#node-red-contrib-matrix-chat:skylar.tech)
+
 ### Features
 
 The following is supported from this package:
@@ -30,9 +32,6 @@ The following is supported from this package:
 
 Therefore, you can easily build a bot, chat relay, or administrate your Matrix server from within [Node-RED](https://nodered.org/).
 
-### Usage
-We have examples! [Check them out](https://github.com/Skylar-Tech/node-red-contrib-matrix-chat/tree/master/examples#readme)
-
 ### Installing
 
 You can either install from within Node-RED by searching for `node-red-contrib-matrix-chat` or run this from within your Node-RED directory:
@@ -40,8 +39,16 @@ You can either install from within Node-RED by searching for `node-red-contrib-m
 npm install node-red-contrib-matrix-chat
 ```
 
+### Usage
+We have examples! [Check them out](https://github.com/Skylar-Tech/node-red-contrib-matrix-chat/tree/master/examples#readme)
+
+#### Extra functionality
+You are not limited by just the nodes we have created. If you turn on global access when setting up your Matrix Client you can access the client directly from any function node to write your own logic.
+
+View an example [here](https://github.com/Skylar-Tech/node-red-contrib-matrix-chat/tree/master/examples#use-function-node-to-run-any-command)
+
 ### End-to-End Encryption Notes
-Currently this module has no way of getting encryption keys from other devices on the same account. Therefore it is recommended you use the bot exclusively with Node-RED after it's creation. Failure to do so will lead to your bot being unable to receive messages from e2ee rooms it joined from another client. Shared secret registration makes this super easy since it returns a token and device ID.
+Currently, this module has no way of getting encryption keys from other devices on the same account. Therefore it is recommended you use the bot exclusively with Node-RED after it's creation. Failure to do so will lead to your bot being unable to receive messages from e2ee rooms it joined from another client. Shared secret registration makes this super easy since it returns a token and device ID.
 
 This module stores a folder in your Node-RED directory called `matrix-client-storage` and is it vital that you periodically back this up if you are using e2ee. This is where the client stores all the keys necessary to decrypt messages and if lost you will lose access to e2e rooms. If you move your client to another NR install make sure to migrate this folder as well (and do not let both the old and new client run at same time).
 
@@ -61,4 +68,4 @@ You will need a user to use this module. Luckily this module comes with a node t
 ### Contributing
 All contributions are welcome! If you do add a feature please do a pull request so that everyone benefits :)
 
-Sharing is caring!
+Sharing is caring!

examples/README.md

@@ -9,6 +9,7 @@ Build something cool with these nodes? Feel free to submit a pull request to sha
 
 - [Create User with Shared Secret Registration](#create-user-with-shared-secret-registration)
 - [Create/Edit Synapse User](#createedit-synapse-user)
+- [Use function node to run any command](#use-function-node-to-run-any-command)
 - [Respond to "ping" with "pong"](#respond-to-ping-with-pong)
 - [Respond to "html" with an HTML message](#respond-to-html-with-an-html-message)
 - [Respond to "image" with an uploaded image](#respond-to-image-with-an-uploaded-image)
@@ -39,9 +40,21 @@ Allows an administrator to create or modify a user account with a specified `msg
 ![img.png](add-user-with-admin-user.png)
 
 
+### Use function node to run any command
+
+[View JSON](custom-redact-function-node.json)
+
+If we do not have a node for something you want to do (such as redacting events/messages) you can do this manually with a function node.
+
+**Note:** You should make sure to catch any errors in your function node otherwise you could cause Node-RED to crash.
+
+To view what sort of functions you have access to check out the `client.ts` file from `matrix-js-sdk` [here](https://github.com/matrix-org/matrix-js-sdk/blob/master/src/client.ts).
+
+![img.png](custom-redact-function-node.png)
+
 ### Respond to "ping" with "pong"
 
-[View JSON](respond-to-html-with-html.json)
+[View JSON](respond-ping-pong.json)
 
 Use this flow to respond to anyone that says "ping" with "pong" into the same room.
 

examples/custom-redact-function-node.json

@@ -0,0 +1,113 @@
+[
+  {
+    "id": "89eed7ddf7a96070",
+    "type": "function",
+    "z": "f025a8b9fbd1b054",
+    "name": "Redact Message",
+    "func": "let matrixClient = global.get(\"matrixClient['@bot:example.com']\"),\n    matrixOnline = global.get(\"matrixClientOnline['@bot:example.com']\");\n\nif(!matrixOnline) {\n    msg.error = 'Matrix client not connected.';\n    return [null, msg];\n}\n    \nmatrixClient.redactEvent(msg.topic, msg.eventId, undefined, {\n    reason: 'Redacted, muhahaha!'\n})\n    .then(function(e) {\n        msg.eventId = e.eventId;\n        node.send([msg, null]);\n    })\n    .catch(function(e){\n        node.warn(\"Error sending message \" + e);\n        msg.error = e;\n        node.send([null, msg]);\n    });",
+    "outputs": 2,
+    "noerr": 0,
+    "initialize": "",
+    "finalize": "",
+    "libs": [],
+    "x": 730,
+    "y": 1580,
+    "wires": [
+      [
+        "5decd492e2da79d9"
+      ],
+      [
+        "fadcf8c32bdfd8be"
+      ]
+    ]
+  },
+  {
+    "id": "5decd492e2da79d9",
+    "type": "debug",
+    "z": "f025a8b9fbd1b054",
+    "name": "",
+    "active": true,
+    "tosidebar": true,
+    "console": false,
+    "tostatus": false,
+    "complete": "true",
+    "targetType": "full",
+    "statusVal": "",
+    "statusType": "auto",
+    "x": 890,
+    "y": 1560,
+    "wires": []
+  },
+  {
+    "id": "f3636f4221511a81",
+    "type": "comment",
+    "z": "f025a8b9fbd1b054",
+    "name": "If message is \"redact\" then redact the message.",
+    "info": "",
+    "x": 420,
+    "y": 1540,
+    "wires": []
+  },
+  {
+    "id": "9f777828840bb9a2",
+    "type": "matrix-receive",
+    "z": "f025a8b9fbd1b054",
+    "name": "",
+    "roomId": "",
+    "acceptText": true,
+    "acceptEmotes": false,
+    "acceptStickers": false,
+    "acceptReactions": false,
+    "acceptFiles": false,
+    "acceptImages": false,
+    "x": 320,
+    "y": 1580,
+    "wires": [
+      [
+        "21a00b09f8c85399"
+      ]
+    ]
+  },
+  {
+    "id": "21a00b09f8c85399",
+    "type": "switch",
+    "z": "f025a8b9fbd1b054",
+    "name": "filter on msg.payload",
+    "property": "payload",
+    "propertyType": "msg",
+    "rules": [
+      {
+        "t": "eq",
+        "v": "redact",
+        "vt": "str"
+      }
+    ],
+    "checkall": "true",
+    "repair": false,
+    "outputs": 1,
+    "x": 520,
+    "y": 1580,
+    "wires": [
+      [
+        "89eed7ddf7a96070"
+      ]
+    ]
+  },
+  {
+    "id": "fadcf8c32bdfd8be",
+    "type": "debug",
+    "z": "f025a8b9fbd1b054",
+    "name": "",
+    "active": true,
+    "tosidebar": true,
+    "console": false,
+    "tostatus": false,
+    "complete": "true",
+    "targetType": "full",
+    "statusVal": "",
+    "statusType": "auto",
+    "x": 890,
+    "y": 1600,
+    "wires": []
+  }
+]

package.json

@@ -1,14 +1,15 @@
 {
   "name": "node-red-contrib-matrix-chat",
-  "version": "0.2.0",
+  "version": "0.4.0",
   "description": "Matrix chat server client for Node-RED",
   "dependencies": {
+    "@matrix-org/olm": "^3.2.8",
     "fs-extra": "^9.1.0",
     "got": "^11.8.2",
     "isomorphic-webcrypto": "^2.3.8",
-    "matrix-js-sdk": "^12.2.0",
+    "matrix-js-sdk": "^15.3.0",
     "node-localstorage": "^2.2.1",
-    "olm": "https://packages.matrix.org/npm/olm/olm-3.1.4.tgz",
+    "process": "^0.11.10",
     "utf8": "^3.0.0"
   },
   "node-red": {

src/matrix-create-room.html

@@ -39,7 +39,7 @@
         <dt class="optional">msg.payload
             <span class="property-type">string | object | null</span>
         </dt>
-        <dd> a list of options to pass to the /createRoom API. <a href="https://matrix.org/docs/spec/client_server/r0.4.0#id266">Click here</a> for information about what can be passed. If this is a string it just sets the room name and ignores other options.</dd>
+        <dd> a list of options to pass to the /createRoom API. <a href="https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-createroom">Click here</a> for information about what can be passed. If this is a string it just sets the room name and ignores other options.</dd>
     </dl>
 
     <h3>Outputs</h3>

src/matrix-invite-room.html

@@ -8,7 +8,8 @@
         outputs: 2,
         defaults: {
             name: { value: null },
-            server: { value: "", type: "matrix-server-config" }
+            server: { value: "", type: "matrix-server-config" },
+            roomId: { value: null },
         },
         label: function() {
             return this.name || "Room Invite";
@@ -27,6 +28,11 @@
         <label for="node-input-server"><i class="fa fa-user"></i> Matrix Server Config</label>
         <input type="text" id="node-input-server">
     </div>
+
+    <div class="form-row">
+        <label for="node-input-roomId"><i class="fa fa-user"></i> Room ID</label>
+        <input type="text" id="node-input-roomId" placeholder="msg.topic">
+    </div>
 </script>
 
 <script type="text/html" data-help-name="matrix-invite-room">

src/matrix-invite-room.js

@@ -6,6 +6,7 @@ module.exports = function(RED) {
 
         this.name = n.name;
         this.server = RED.nodes.getNode(n.server);
+        this.roomId = n.roomId;
 
         if(!this.server) {
             node.error('Server must be configured on the node.');
@@ -47,7 +48,7 @@ module.exports = function(RED) {
 
             msg.topic = node.roomId || msg.topic;
             if(!msg.topic) {
-                node.error("msg.topic must be defined or configured on the node.");
+                node.error("room must be defined in either msg.topic or in node config");
                 return;
             }
 

src/matrix-join-room.js

@@ -38,11 +38,6 @@ module.exports = function(RED) {
                 return;
             }
 
-            if(!msg.userId) {
-                node.error("msg.userId was not set.");
-                return;
-            }
-
             node.server.matrixClient.joinRoom(msg.topic, msg.joinOpts || {})
                 .then(function(e) {
                     msg.payload = e;

src/matrix-react.js

@@ -53,16 +53,14 @@ module.exports = function(RED) {
 
             msg.type = 'm.reaction';
 
-            node.server.matrixClient.sendCompleteEvent(
+            node.server.matrixClient.sendEvent(
                 msg.topic,
+                'm.reaction',
                 {
-                    type: 'm.reaction',
-                    content: {
-                        "m.relates_to": {
-                            event_id: eventId,
-                            key: msg.payload,
-                            rel_type: "m.annotation"
-                        }
+                    "m.relates_to": {
+                        event_id: eventId,
+                        key: msg.payload,
+                        rel_type: "m.annotation"
                     }
                 }
             )

src/matrix-receive.js

@@ -22,7 +22,7 @@ module.exports = function(RED) {
             return;
         }
 
-        node.server.on("disconnected", function() {
+        node.server.on("disconnected", function(){
             node.status({ fill: "red", shape: "ring", text: "disconnected" });
         });
 
@@ -30,43 +30,12 @@ module.exports = function(RED) {
             node.status({ fill: "green", shape: "ring", text: "connected" });
         });
 
-        node.server.on("Room.timeline", async function(event, room, toStartOfTimeline, data) {
-            if (toStartOfTimeline) {
-                return; // ignore paginated results
-            }
-            if (!event.getSender() || event.getSender() === node.server.userId) {
-                return; // ignore our own messages
-            }
-            if (!event.getUnsigned() || event.getUnsigned().age > 1000) {
-                return; // ignore old messages
-            }
-
+        node.server.on("Room.timeline", async function(event, room, toStartOfTimeline, removed, data, msg) {
             // if node has a room ID set we only listen on that room
             if(node.roomIds.length && node.roomIds.indexOf(room.roomId) === -1) {
                 return;
             }
 
-            try {
-                await node.server.matrixClient.decryptEventIfNeeded(event);
-            } catch (error) {
-                node.error(error);
-                return;
-            }
-
-            let msg = {
-                encrypted : event.isEncrypted(),
-                redacted  : event.isRedacted(),
-                content   : event.getContent(),
-                type      : (event.getContent()['msgtype'] || event.getType()) || null,
-                payload   : (event.getContent()['body'] || event.getContent()) || null,
-                userId    : event.getSender(),
-                topic     : event.getRoomId(),
-                eventId   : event.getId(),
-                event     : event,
-            };
-
-            node.log("Received" + (msg.encrypted ? ' encrypted' : '') +" timeline event [" + msg.type + "]: (" + room.name + ") " + event.getSender() + " :: " + msg.content.body);
-
             switch(msg.type) {
                 case 'm.emote':
                     if(!node.acceptEmotes) return;

src/matrix-send-file.js

@@ -80,7 +80,7 @@ module.exports = function(RED) {
                         .sendMessage(msg.topic, content)
                             .then(function(e) {
                                 node.log("File message sent: " + e);
-                                msg.eventId = e.eventId;
+                                msg.eventId = e.event_id;
                                 node.send([msg, null]);
                             })
                             .catch(function(e){

src/matrix-send-image.html

@@ -34,7 +34,7 @@
     </div>
     <div class="form-row">
         <label for="node-input-contentType"><i class="fa fa-user"></i> Content-Type</label>
-        <input type="text" id="node-input-contentType" placeholder="application/png">
+        <input type="text" id="node-input-contentType" placeholder="msg.contentType">
     </div>
     <div class="form-tips">
         Must be a valid <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types" target="_blank">MIME Type</a> (ex: image/png) or left empty

src/matrix-send-message.js

@@ -89,6 +89,7 @@ module.exports = function(RED) {
             if(!node.server.isConnected()) {
                 node.error("Matrix server connection is currently closed");
                 node.send([null, msg]);
+                return;
             }
 
             msg.topic = node.roomId || msg.topic;
@@ -109,13 +110,16 @@ module.exports = function(RED) {
 
             if(msgFormat === 'html') {
                 content.format = "org.matrix.custom.html";
-                content.formatted_body = msg.formatted_payload || msg.payload;
+                content.formatted_body =
+                    (typeof msg.formatted_payload !== 'undefined' && msg.formatted_payload)
+                        ? msg.formatted_payload.toString()
+                        : msg.payload.toString();
             }
 
             node.server.matrixClient.sendMessage(msg.topic, content)
                 .then(function(e) {
                     node.log("Message sent: " + msg.payload);
-                    msg.eventId = e.eventId;
+                    msg.eventId = e.event_id;
                     node.send([msg, null]);
                 })
                 .catch(function(e){

src/matrix-server-config.html

@@ -1,17 +1,42 @@
+<style>
+    .matrix-loader {
+        border: 8px solid #f3f3f3;
+        border-radius: 50%;
+        border-top: 8px solid #3498db;
+        border-bottom: 8px solid #3498db;
+        width: 20px;
+        height: 20px;
+        -webkit-animation: matrix-spin 2s linear infinite;
+        animation: matrix-spin 2s linear infinite;
+    }
+
+    @-webkit-keyframes matrix-spin {
+        0% { -webkit-transform: rotate(0deg); }
+        100% { -webkit-transform: rotate(360deg); }
+    }
+
+    @keyframes matrix-spin {
+        0% { transform: rotate(0deg); }
+        100% { transform: rotate(360deg); }
+    }
+</style>
+
 <script type="text/javascript">
     RED.nodes.registerType('matrix-server-config',{
         category: 'config',
         color: '#00b7ca',
         credentials: {
             userId: { type: "text", required: true },
+            deviceLabel: { type: "text", required: false },
             accessToken: { type: "password", required: true },
-            deviceId: { type: "text", required: true },
+            deviceId: { type: "text", required: false },
             url: { type: "text", required: true }
         },
         defaults: {
             name: { value: null },
             autoAcceptRoomInvites: { value: true },
-            enableE2ee: { type: "checkbox", value: true }
+            enableE2ee: { type: "checkbox", value: true },
+            global: { type: "checkbox", value: true }
         },
         icon: "matrix.png",
         label: function() {
@@ -25,35 +50,58 @@
         <label for="node-config-input-name"><i class="fa fa-tag"></i> Name</label>
         <input type="text" id="node-config-input-name" placeholder="Name">
     </div>
+
     <div class="form-row">
-        <label for="node-config-input-userId"><i class="fa fa-server"></i> User ID</label>
+        <label for="node-config-input-deviceLabel"><i class="fa fa-tag"></i> Device Label</label>
+        <input type="text" placeholder="Node-RED" id="node-config-input-deviceLabel">
+    </div>
+
+    <div class="form-row">
+        <label for="node-config-input-url"><i class="fa fa-server"></i> Server URL</label>
+        <input type="text" placeholder="https://matrix.org" id="node-config-input-url">
+    </div>
+
+    <div class="form-row">
+        <label for="node-config-input-userId"><i class="fa fa-user"></i> User ID</label>
         <input type="text" placeholder="@example:matrix.org" id="node-config-input-userId">
     </div>
+
+    <div class="form-row">
+        <label for="node-config-input-password"><i class="fa fa-key"></i> Password</label>
+        <input type="password" placeholder="" id="node-config-input-password">
+    </div>
+    <div class="form-tips" style="margin-bottom: 12px;">
+        Password is never saved and is only used to fetch an access token using the button below.
+    </div>
+    <pre class="form-tips" id="matrix-chat-login-error" style="color: #721c24;background-color: #f8d7da;border-color: #f5c6cb;margin-bottom: 12px;display:none;"></pre>
+    <pre class="form-tips" id="matrix-chat-login-success" style="color: #155724;background-color: #d4edda;border-color: #c3e6cb;margin-bottom: 12px;display:none;"></pre>
+    <button class="ui-button ui-corner-all ui-widget primary" id="matrix-login-btn">Update Access Token</button>
+    <div class="matrix-loader" id="matrix-access-token-loader" style="display:none;"></div>
+
     <div class="form-row">
         <label for="node-config-input-accessToken"><i class="fa fa-key"></i> Access Token</label>
         <input type="text" id="node-config-input-accessToken">
     </div>
     <div class="form-tips" style="margin-bottom: 12px;">
-        View the <a href="javascript:$('#red-ui-tab-help-link-button').click();">node docs</a> to figure out how to generate an Access Token. You can also generate them using the Shared Secret Registration node.
+        You can either provide/generate an access token yourself or use the login button above to do it automatically. View the <a href="javascript:$('#red-ui-tab-help-link-button').click();">node docs</a> to figure out how to generate an Access Token manually. If you generated a user with shared secret registration you will already have an access token you can place here.
     </div>
+
     <div class="form-row">
-        <label for="node-config-input-deviceId"><i class="fa fa-key"></i> Device ID</label>
+        <label for="node-config-input-deviceId"><i class="fa fa-desktop"></i> Device ID</label>
         <input type="text" id="node-config-input-deviceId">
     </div>
     <div class="form-tips" style="margin-bottom: 12px;">
-        This can either be an existing Device ID attached to the above Access Token or you can enter a unique value to set a new one.
+        If you have a Device ID for this auth token you can place it here, or you can leave it blank to have one automatically generated (this field will always be blank if auto generated as it is saved internally). DO NOT manually set one, this must be generated by the server.<br />
+        WARNING: If you change this after the client has already initialized you will break encryption. Your Device ID is tied to your encryption keys.
     </div>
- 	 <div class="form-row">
-     	<label for="node-config-input-url"><i class="fa fa-globe"></i> Server URL</label>
-     	<input type="text" placeholder="https://matrix.org" id="node-config-input-url">
- 	</div>
+
     <div class="form-row">
         <input
             type="checkbox"
             id="node-config-input-autoAcceptRoomInvites"
             style="width: auto; margin-left: 125px; vertical-align: top"
         />
-        <label for="node-config-input-autoAcceptRoomInvites" style="width: auto">
+        <label for="node-config-input-autoAcceptRoomInvites" style="width: auto;max-width:50%;">
             Auto join invited rooms
         </label>
     </div>
@@ -63,17 +111,107 @@
                 id="node-config-input-enableE2ee"
                 style="width: auto; margin-left: 125px; vertical-align: top"
         />
-        <label for="node-config-input-enableE2ee" style="width: auto">
-            Enable end-to-end encryption (requires DeviceID)
+        <label for="node-config-input-enableE2ee" style="width: auto;max-width:50%;">
+            Enable end-to-end encryption (requires a Device ID to be set.)
         </label>
     </div>
 
-    <div class="form-tips" style="margin-bottom: 12px;">
-        E2ee requires a Device ID to be set.
+    <div class="form-row">
+        <input
+                type="checkbox"
+                id="node-config-input-global"
+                style="width: auto; margin-left: 125px; vertical-align: top"
+        />
+        <label for="node-config-input-global" style="width: auto">
+            Global access to Matrix Client
+        </label>
+        <div class="form-tips" style="margin-bottom: 12px;">
+            If global access is enabled you can access the client directly within a Function node. This way you can do <a href="https://github.com/Skylar-Tech/node-red-contrib-matrix-chat/tree/master/examples#use-function-node-to-run-any-command" target="_blank">whatever you want</a> with the client. Example:<br>
+            <code style="white-space: normal;">let client = global.get("matrixClient['@bot:example.com']");</code>
+        </div>
     </div>
+    <script type="text/javascript">
+        $("#matrix-login-btn").on("click", function() {
+            function prettyPrintJson(json) {
+                try{
+                    return typeof json === 'object' ? JSON.stringify(json, null, 2) : json;
+                }
+                catch (error){
+                    return json;
+                }
+            }
+
+            let userId = $("#node-config-input-userId").val(),
+                userPassword = $("#node-config-input-password").val(),
+                serverUrl = $("#node-config-input-url").val();
+
+            if(!userId) {
+                alert("User ID is required to fetch access token.");
+                return;
+            }
+            if(!userPassword) {
+                alert("Password is required to fetch access token.");
+                return;
+            }
+            if(!serverUrl) {
+                alert("Server URL is required to fetch access token.");
+                return;
+            }
+
+            $("#matrix-login-btn, #matrix-chat-login-error, #matrix-chat-login-success").hide();
+            $("#matrix-access-token-loader").show();
+            $.ajax({
+                type: 'POST',
+                url: '/matrix-chat/login',
+                dataType: 'json',
+                data: {
+                    'userId': userId,
+                    'password': userPassword,
+                    'baseUrl': serverUrl,
+                    'displayName': $("#node-config-input-deviceLabel").val(),
+                }
+            }).then(
+                function(data) {
+                    if(data.result && data.result === 'ok') {
+                        $("#matrix-chat-login-error").hide();
+                        $("#matrix-chat-login-success")
+                            .html("Login Successful! Auth Token and Device ID have been set below.")
+                            .show();
+
+                        $("#node-config-input-accessToken").val(data.token);
+                        $("#node-config-input-deviceId").val(data.device_id);
+                    } else if(data.result && data.result === 'error') {
+                        $("#matrix-chat-login-success").hide();
+                        $("#matrix-chat-login-error")
+                            .html(data.message ? ('Failed to login: <br />' + prettyPrintJson(data.message)) : 'Failed to login')
+                            .show();
+                    }
+                    $("#matrix-login-btn").show();
+                    $("#matrix-access-token-loader").hide();
+                }, function() {
+                    $("#matrix-chat-login-success").hide();
+                    $("#matrix-chat-login-error")
+                        .html("Failed to login due to server error communicating with Node-RED")
+                        .show();
+                    $("#matrix-login-btn").show();
+                    $("#matrix-access-token-loader").hide();
+                }
+            );
+        });
+    </script>
 </script>
 
 <script type="text/html" data-help-name="matrix-server-config">
     <h3>Details</h3>
-    <p>Matrix client configuration node</p>
+    <p>Matrix client connection configuration</p>
+
+    <h3>Setting up an account</h3>
+    <div>
+        <p>
+            You need an account for your client to use. If you are going to be using End-to-End Encryption you should generate the bot and only use it within Node-RED otherwise if you have other clients connected on the same user it could cause problems with e2ee (key sharing is currently not supported).
+        </p>
+        <p>If you have access to the server directly you can use Shared Secret Registration as described <a href="https://github.com/Skylar-Tech/node-red-contrib-matrix-chat/tree/master/examples#create-user-with-shared-secret-registration" target="_blank" style="text-decoration: underline;">here</a>.</p>
+        <p>If this is a server you do not administrate/have access to follow these instructions:</p>
+        <ol><li>In a private/incognito browser window, open Element.</li><li>Log in to the account you want to get the access token for, such as the bot's account. <strong>Do not setup key storage</strong>.</li><li>Click on the bot's name in the top left corner then "Settings".</li><li>(Optional) Set your bot's display name and avatar.</li><li>Click the "Help &amp; About" tab (left side of the dialog).</li><li>Scroll to the bottom and click the <code>&lt;click to reveal&gt;</code> part of <code>Access Token: &lt;click to reveal&gt;</code>.</li><li>Copy your access token to a safe place, like the bot's configuration file.</li><li><strong>Do not log out.</strong> Instead, just close the window. If you used a private browsing session, you should be able to still use Element for your own account. Logging out deletes the access token from the server, making the bot unable to use it.</li></ol>
+    </div>
 </script>

src/matrix-server-config.js

@@ -1,4 +1,4 @@
-global.Olm = require('olm');
+global.Olm = require('@matrix-org/olm');
 const fs = require("fs-extra");
 const sdk = require("matrix-js-sdk");
 const { LocalStorage } = require('node-localstorage');
@@ -12,9 +12,6 @@ module.exports = function(RED) {
     function MatrixServerNode(n) {
         let storageDir = './matrix-client-storage';
 
-        // we should add support for getting access token automatically from username/password
-        // ref: https://matrix.org/docs/guides/usage-of-the-matrix-js-sdk#login-with-an-access-token
-
         RED.nodes.createNode(this, n);
 
         let node = this;
@@ -26,14 +23,22 @@ module.exports = function(RED) {
 
         node.setMaxListeners(1000);
 
-        this.connected = false;
+        this.connected = null;
         this.name = n.name;
         this.userId = this.credentials.userId;
+        this.deviceLabel = this.credentials.deviceLabel || null;
         this.deviceId = this.credentials.deviceId || null;
         this.url = this.credentials.url;
         this.autoAcceptRoomInvites = n.autoAcceptRoomInvites;
         this.enableE2ee = n.enableE2ee || false;
         this.e2ee = (this.enableE2ee && this.deviceId);
+        this.globalAccess = n.global;
+        this.initializedAt = new Date();
+        let localStorageDir = storageDir + '/' + MatrixFolderNameFromUserId(this.userId),
+            localStorage = new LocalStorage(localStorageDir),
+            initialSetup = false;
+
+        let retryStartTimeout = null;
 
         if(!this.credentials.accessToken) {
             node.log("Matrix connection failed: missing access token.");
@@ -42,48 +47,142 @@ module.exports = function(RED) {
         } else if(!this.userId) {
             node.log("Matrix connection failed: missing user ID.");
         } else {
-            let localStorageDir = storageDir + '/' + MatrixFolderNameFromUserId(this.userId);
+            node.setConnected = function(connected, cb) {
+                if (node.connected !== connected) {
+                    node.connected = connected;
+                    if(typeof cb === 'function') {
+                        cb(connected);
+                    }
+
+                    if (connected) {
+                        node.log("Matrix server connection ready.");
+                        node.emit("connected");
+                        if(!initialSetup) {
+                            // store Device ID internally
+                            let stored_device_id = getStoredDeviceId(localStorage),
+                                device_id = this.matrixClient.getDeviceId();
+                            if(!stored_device_id || stored_device_id !== device_id) {
+                                node.log(`Saving Device ID (old:${stored_device_id} new:${device_id})`);
+                                storeDeviceId(localStorage, device_id);
+                            }
+
+                            // update device label
+                            if(node.deviceLabel) {
+                                node.matrixClient
+                                    .getDevice(device_id)
+                                    .then(
+                                        function(response) {
+                                            if(response.display_name !== node.deviceLabel) {
+                                                node.matrixClient.setDeviceDetails(device_id, {
+                                                    display_name: node.deviceLabel
+                                                }).then(
+                                                    function(response) {},
+                                                    function(error) {
+                                                        node.error("Failed to set device label: " + error);
+                                                    }
+                                                );
+                                            }
+                                        },
+                                        function(error) {
+                                            node.error("Failed to fetch device: " + error);
+                                        }
+                                    );
+                            }
+
+                            initialSetup = true;
+                        }
+                    } else {
+                        node.emit("disconnected");
+                    }
+
+                    if(this.globalAccess) {
+                        this.context().global.set('matrixClientOnline["'+this.userId+'"]', connected);
+                    }
+                }
+            };
+            node.setConnected(false);
 
             fs.ensureDirSync(storageDir); // create storage directory if it doesn't exist
             upgradeDirectoryIfNecessary(node, storageDir);
-            const localStorage = new LocalStorage(localStorageDir);
             node.matrixClient = sdk.createClient({
                 baseUrl: this.url,
                 accessToken: this.credentials.accessToken,
                 sessionStore: new sdk.WebStorageSessionStore(localStorage),
                 cryptoStore: new LocalStorageCryptoStore(localStorage),
                 userId: this.userId,
-                deviceId: this.deviceId || undefined,
+                deviceId: (this.deviceId || getStoredDeviceId(localStorage)) || undefined
             });
 
-            node.on('close', function(done) {
-                if(node.matrixClient) {
-                    node.matrixClient.close();
+            // set globally if configured to do so
+            if(this.globalAccess) {
+                this.context().global.set('matrixClient["'+this.userId+'"]', node.matrixClient);
+            }
+
+            function stopClient() {
+                if(node.matrixClient && node.matrixClient.clientRunning) {
                     node.matrixClient.stopClient();
                     node.setConnected(false);
                 }
 
+                if(retryStartTimeout) {
+                    clearTimeout(retryStartTimeout);
+                }
+            }
+
+            node.on('close', function(done) {
+                stopClient();
                 done();
             });
 
-            node.setConnected = function(connected) {
-                if (node.connected !== connected) {
-                    node.connected = connected;
-                    if (connected) {
-                        node.log("Matrix server connection ready.");
-                        node.emit("connected");
-                    } else {
-                        node.emit("disconnected");
-                    }
-                }
-            };
-
             node.isConnected = function() {
                 return node.connected;
             };
 
-            node.matrixClient.on("Room.timeline", async function(event, room, toStartOfTimeline, data) {
-                node.emit("Room.timeline", event, room, toStartOfTimeline, data);
+            node.matrixClient.on("Room.timeline", async function(event, room, toStartOfTimeline, removed, data) {
+                if (toStartOfTimeline) {
+                    return; // ignore paginated results
+                }
+                if (!event.getSender() || event.getSender() === node.userId) {
+                    return; // ignore our own messages
+                }
+                if (!data || !data.liveEvent) {
+                    return; // ignore old message (we only want live events)
+                }
+                if(node.initializedAt > event.getDate()) {
+                    return; // skip events that occurred before our client initialized
+                }
+
+                try {
+                    await node.matrixClient.decryptEventIfNeeded(event);
+                } catch (error) {
+                    node.error(error);
+                    return;
+                }
+
+                let msg = {
+                    encrypted : event.isEncrypted(),
+                    redacted  : event.isRedacted(),
+                    content   : event.getContent(),
+                    type      : (event.getContent()['msgtype'] || event.getType()) || null,
+                    payload   : (event.getContent()['body'] || event.getContent()) || null,
+                    userId    : event.getSender(),
+                    topic     : event.getRoomId(),
+                    eventId   : event.getId(),
+                    event     : event,
+                };
+
+                node.log("Received" + (msg.encrypted ? ' encrypted' : '') +" timeline event [" + msg.type + "]: (" + room.name + ") " + event.getSender() + " :: " + msg.content.body + (toStartOfTimeline ? ' [PAGINATED]' : ''));
+                node.emit("Room.timeline", event, room, toStartOfTimeline, removed, data, msg);
+            });
+
+            /**
+             * Fires when we want to suggest to the user that they restore their megolm keys
+             * from backup or by cross-signing the device.
+             *
+             * @event module:client~MatrixClient#"crypto.suggestKeyRestore"
+             */
+            node.matrixClient.on("crypto.suggestKeyRestore", function(){
+
             });
 
             // node.matrixClient.on("RoomMember.typing", async function(event, member) {
@@ -118,29 +217,70 @@ module.exports = function(RED) {
             });
 
             node.matrixClient.on("sync", async function(state, prevState, data) {
-                switch (state) {
-                    case "ERROR":
+                node.debug("SYNC [STATE=" + state + "] [PREVSTATE=" + prevState + "]");
+                if(prevState === null && state === "PREPARED" ) {
+                    // Occurs when the initial sync is completed first time.
+                    // This involves setting up filters and obtaining push rules.
+                    node.setConnected(true, function(){
+                        node.log("Matrix client connected");
+                    });
+                } else if(prevState === null && state === "ERROR") {
+                    // Occurs when the initial sync failed first time.
+                    node.setConnected(false, function(){
+                        node.error("Failed to connect to Matrix server");
+                    });
+                } else if(prevState === "ERROR" && state === "PREPARED") {
+                    // Occurs when the initial sync succeeds
+                    // after previously failing.
+                    node.setConnected(true, function(){
+                        node.log("Matrix client connected");
+                    });
+                } else if(prevState === "PREPARED" && state === "SYNCING") {
+                    // Occurs immediately after transitioning to PREPARED.
+                    // Starts listening for live updates rather than catching up.
+                    node.setConnected(true, function(){
+                        node.log("Matrix client connected");
+                    });
+                } else if(prevState === "SYNCING" && state === "RECONNECTING") {
+                    // Occurs when the live update fails.
+                    node.setConnected(false, function(){
                         node.error("Connection to Matrix server lost");
-                        node.setConnected(false);
-                        break;
-
-                    case "RECONNECTING":
-                    case "STOPPED":
-                        node.setConnected(false);
-                        break;
-
-                    case "SYNCING":
-                        break;
-
-                    case "PREPARED":
-                        node.setConnected(true);
-                        break;
-
-                    // case "PREPARED":
-                    //     // the client instance is ready to be queried.
-                    //     node.log("Matrix server connection ready.");
-                    //     node.setConnected(true);
-                    //     break;
+                    });
+                } else if(prevState === "RECONNECTING" && state === "RECONNECTING") {
+                    // Can occur if the update calls continue to fail,
+                    // but the keepalive calls (to /versions) succeed.
+                    node.setConnected(false, function(){
+                        node.error("Connection to Matrix server lost");
+                    });
+                } else if(prevState === "RECONNECTING" && state === "ERROR") {
+                    // Occurs when the keepalive call also fails
+                    node.setConnected(false, function(){
+                        node.error("Connection to Matrix server lost");
+                    });
+                } else if(prevState === "ERROR" && state === "SYNCING") {
+                    // Occurs when the client has performed a
+                    // live update after having previously failed.
+                    node.setConnected(true, function(){
+                        node.log("Matrix client connected");
+                    });
+                } else if(prevState === "ERROR" && state === "ERROR") {
+                    // Occurs when the client has failed to
+                    // keepalive for a second time or more.
+                    node.setConnected(false, function(){
+                        node.error("Connection to Matrix server lost");
+                    });
+                } else if(prevState === "SYNCING" && state === "SYNCING") {
+                    // Occurs when the client has performed a live update.
+                    // This is called <i>after</i> processing.
+                    node.setConnected(true, function(){
+                        node.log("Matrix client connected");
+                    });
+                } else if(state === "STOPPED") {
+                    // Occurs once the client has stopped syncing or
+                    // trying to sync after stopClient has been called.
+                    node.setConnected(false, function(){
+                        node.error("Connection to Matrix server lost");
+                    });
                 }
             });
 
@@ -156,7 +296,9 @@ module.exports = function(RED) {
                 //     httpStatus: 401
                 // }
 
-                node.error("[Session.logged_out] " + errorObj);
+                console.log("Authentication failure: ", errorObj);
+                node.error("Authentication failure: " + errorObj);
+                stopClient();
             });
 
             async function run() {
@@ -167,25 +309,88 @@ module.exports = function(RED) {
                         node.matrixClient.setGlobalErrorOnUnknownDevices(false);
                     }
                     node.log("Connecting to Matrix server...");
-                    await node.matrixClient.startClient({ initialSyncLimit: 8 });
-                } catch(error){
+                    await node.matrixClient.startClient({
+                        initialSyncLimit: 8
+                    });
+                } catch(error) {
                     node.error(error);
                 }
             }
 
-            run().catch((error) => node.error(error));
+            // do an authed request and only continue if we don't get an error
+            // this prevent the matrix client from crashing Node-RED on invalid auth token
+            (function checkAuthTokenThenStart() {
+                if(node.matrixClient.clientRunning) {
+                    return;
+                }
+
+                node.matrixClient.getAccountDataFromServer()
+                    .then(
+                        function() {
+                            run().catch((error) => node.error(error));
+                        },
+                        function(err) {
+                            // if the error isn't authentication related retry in a little bit
+                            if(err.code !== "M_UNKNOWN_TOKEN") {
+                                retryStartTimeout = setTimeout(checkAuthTokenThenStart, 15000);
+                                node.error("Auth check failed: " + err);
+                            }
+                        }
+                    )
+            })();
         }
     }
 
     RED.nodes.registerType("matrix-server-config", MatrixServerNode, {
         credentials: {
-            userId: { type:"text", required: true },
-            accessToken: { type:"text", required: true },
-            deviceId: { type: "text", required: true },
-            url: { type: "text", required: true },
+            deviceLabel: { type: "text", required: false },
+            userId: { type: "text", required: true },
+            accessToken: { type: "text", required: true },
+            deviceId: { type: "text", required: false },
+            url: { type: "text", required: true }
         }
     });
 
+    RED.httpAdmin.post(
+        "/matrix-chat/login",
+        RED.auth.needsPermission('flows.write'),
+        function(req, res) {
+            let userId = req.body.userId || undefined,
+                password = req.body.password || undefined,
+                baseUrl = req.body.baseUrl || undefined,
+                deviceId = req.body.deviceId || undefined,
+                displayName = req.body.displayName || undefined;
+
+            const matrixClient = sdk.createClient({
+                baseUrl: baseUrl,
+                deviceId: deviceId,
+                localTimeoutMs: '30000'
+            });
+
+            matrixClient.login(
+                'm.login.password', {
+                    user: userId,
+                    password: password,
+                    initial_device_display_name: displayName
+                })
+                .then(
+                    function(response) {
+                        res.json({
+                            'result': 'ok',
+                            'token': response.access_token,
+                            'device_id': response.device_id,
+                            'user_id': response.user_id,
+                        });
+                    },
+                    function(err) {
+                        res.json({
+                            'result': 'error',
+                            'message': err
+                        });
+                    }
+                );
+        });
+
     function upgradeDirectoryIfNecessary(node, storageDir) {
         let oldStorageDir = './matrix-local-storage';
 
@@ -211,4 +416,15 @@ module.exports = function(RED) {
             fs.renameSync(oldStorageDir, oldStorageDir + "-backup");
         }
     }
+
+    /**
+     * If a device ID is stored we will use that for the client
+     */
+    function getStoredDeviceId(localStorage) {
+        return localStorage.getItem('my_device_id');
+    }
+
+    function storeDeviceId(localStorage, deviceId) {
+        localStorage.setItem('my_device_id', deviceId);
+    }
 }

src/matrix-synapse-join-room.js

@@ -64,7 +64,7 @@ module.exports = function(RED) {
                     'POST',
                     node.encodeUri(
                         "/_synapse/admin/v1/join/$room_id_or_alias",
-                        { $room_id_or_alias: roomId },
+                        { $room_id_or_alias: msg.topic },
                     ),
                     undefined,
                     { "user_id": msg.userId },

src/matrix-synapse-register.html

@@ -78,7 +78,10 @@
         <dt class="optional">msg.payload.user_type
             <span class="property-type">string | null</span>
         </dt>
-        <dd> Set the user type. Leave this to null if you don't know what it is for. Check <a href="https://github.com/matrix-org/synapse/blob/master/synapse/api/constants.py">here</a> and look for <code>class UserTypes</code> to figure out what is valid.</dd>
+        <dd>
+            Set the user type. Leave this to null if you don't know what it is for. Check <a href="https://github.com/matrix-org/synapse/blob/master/synapse/api/constants.py">here</a> and look for <code>class UserTypes</code> to figure out what is valid.<br>
+            NOTE: Testing shows that "bot" and "support" do not work for this. Generate a normal user to use as a bot.
+        </dd>
     </dl>
 
     <h3>Outputs</h3>

src/matrix-synapse-register.js

@@ -62,7 +62,13 @@ module.exports = function(RED) {
                     .update(utf8.encode(msg.payload.password))
                     .update("\x00")
                     .update(msg.payload.admin ? "admin" : "notadmin")
-                    .digest('hex');
+
+                if(msg.payload.user_type || null) {
+                    hmac.update("\x00")
+                        .update(msg.payload.user_type);
+                }
+
+                hmac = hmac.digest('hex');
 
                 try {
                     response = await got.post(this.server + '/_synapse/admin/v1/register', {

src/matrix-synapse-users.js

@@ -39,7 +39,7 @@ module.exports = function(RED) {
             };
 
             if(msg.guests) {
-                queryParams['guests'] = msg.guests;
+                queryParams['guests'] = msg.guests ? 'true' : 'false';
             }
 
             if(msg.order_by) {