This is my contribution to Superalgos

Projects/Superalgos/Schemas/Docs-Topics/trading-farms-001-trading-farms-fundamental-concepts.json

@@ -8,17 +8,54 @@
     },
     "paragraphs": [
         {
-            "style": "Text",
-            "text": "Superalgos is designed for coordinated, flexible, and scalable trading operations.",
-            "updated": 1611343220386
+            "style": "Title",
+            "text": "Introduction",
+            "updated": 1614353180566
         },
         {
             "style": "Text",
-            "text": "In its current version, the system may be deployed in multiple machines—as many as desired—and each machine may constitute a node in a network of nodes. In fact, different networks of nodes may live within the same computer network."
+            "text": "Superalgos is designed for coordinated, flexible, and scalable trading operations."
+        },
+        {
+            "style": "Title",
+            "text": "Network",
+            "updated": 1614353250354
+        },
+        {
+            "style": "Text",
+            "text": "A Network, represented by the Network Hierarchy it is just a definition that allows the system to know where Tasks need to be run, and where data is hosted. At any workspace you mighd define one or even more than one Netwrok Hiererchy. For simplicity we will continue the explanation assuming only one Network is defined.",
+            "updated": 1614353405812
+        },
+        {
+            "style": "Text",
+            "text": "In its current version, the system may be deployed in multiple machines—as many as desired—and each machine may constitute a node in a network of nodes. In fact, different networks of nodes may live within the same computer network, because they are just an abstractions, not something physical.",
+            "updated": 1614353448225
+        },
+        {
+            "style": "Title",
+            "text": "Purpose"
+        },
+        {
+            "style": "Text",
+            "text": "The whole purpose of the Network definition is to let the system know where each task must run and where the data is localted. ",
+            "updated": 1614353600414
+        },
+        {
+            "style": "Success",
+            "text": "A Superalgos Client can become a node in a Network definition if you configure a Network Node with the IP and ports of a certain Superalgos Client."
+        },
+        {
+            "style": "Title",
+            "text": "Organization",
+            "updated": 1614353624048
+        },
+        {
+            "style": "Text",
+            "text": "There are many different way you might want to organize the Tasks across several Clients. It is really up to you. The system don't care or have any preferred way for you to do so. It just need to know where each Task runs, and where the Data is located. That's it."
         },
         {
             "style": "Callout",
-            "text": "Each node on a network may specialize in a single type of process or handle multiple processes of different types."
+            "text": "Each node on a Network may specialize in a single type of process or handle multiple processes of different types."
         },
         {
             "style": "Text",
@@ -36,18 +73,51 @@
             "style": "Callout",
             "text": "In other words, the system allows building any arrangement of networks of nodes responding to the organizational logic of your preference."
         },
+        {
+            "style": "Title",
+            "text": "Auto Tracking of Dependencies",
+            "updated": 1614353913256
+        },
         {
             "style": "Text",
-            "text": "What is unique about Superalgos is that the system keeps track of the network arrangements you create and makes sure that all components and data structures in the system can seamlessly track dependencies across the network. This means, for example, that trading sessions running on Node X may access data in Node Y and Node Z without requiring any sort of configuration."
+            "text": "When you define a Task, inside of it you are referencing a Process Definition, which has in turn definition regarding to which other processes it depends on (for example to start after some other process finished) and it also has Data Dependencies, which means that in order to run, it needs to read the data output from other processes."
+        },
+        {
+            "style": "Note",
+            "text": "All these definitions are built-in inside Data Mines, Trading Mines and Learning mines at a conceptual level. This means that a Data Mine does not really know where the data a process depends on actually is, or where a process that it depends on is actually running. That problem is solved at runtime once a Task is executed."
+        },
+        {
+            "style": "Text",
+            "text": "What is unique about Superalgos is that the system keeps track of the network arrangements you create and makes sure that all processes in the system can seamlessly track dependencies across the network. This means, for example, that trading sessions running on Node X may access data in Node Y and Node Z without requiring any sort of configuration.",
+            "updated": 1614354242360
+        },
+        {
+            "style": "Text",
+            "text": "This is achieved by sending to a newly executed Task a snapshot of the Network Map at the instant when the Task is being run. From the Mine definition the Task can know which processes and data it depends on, and by looking into the snapshot of the Network, it can guess which Client is hosting the Data or the Tasks that the process depends on, allowing it to communicate with those Client s and fetch the data from them or wait for the right events signaling that the dependen process has finished."
+        },
+        {
+            "style": "Title",
+            "text": "Peer to Peer",
+            "updated": 1614354292649
+        },
+        {
+            "style": "Text",
+            "text": "There is no preferred node where to manage the rest of the Client s.",
+            "updated": 1614354328656
         },
         {
             "style": "Callout",
-            "text": "On top of that, the operation may be managed entirely from any node on the network. In fact, it may be managed from a computer in the network not running Superalgos, by pointing the browser to any of the nodes in the network to run the UI."
+            "text": "The operation may be managed entirely from any node on the network. In fact, it may be managed from a computer in the network not running Superalgos, by pointing the browser to any of the nodes in the network to run the UI.",
+            "updated": 1614354336816
         },
         {
             "style": "Text",
             "text": "The following explanations assume that you are familiar with the basic workings of Superalgos, and in particular, of the Network hierarchy. If that is not the case, then please read the Network pages before diving into building a trading farm."
         },
+        {
+            "style": "Title",
+            "text": "Security"
+        },
         {
             "style": "Warning",
             "text": "At this stage, Superalgos does not implement any form of security measures, therefore, the system is to be used in the context of a restricted Local Area Network only.",

Projects/Superalgos/Schemas/Docs-Topics/trading-farms-002-trading-farms-set-up-a-network-of-nodes.json

@@ -9,17 +9,104 @@
     "paragraphs": [
         {
             "style": "Text",
-            "text": "This page does not go through the process of setting up a computer network. Instead, it assumes the network is already set up. So, do verify the local area network setup and make sure your computer network is functioning properly before starting.",
-            "updated": 1611343358129
+            "text": "There are essentially two approaches from here. ",
+            "updated": 1614354810094
+        },
+        {
+            "style": "List",
+            "text": "Install All in One Node: The first one and probably the easiest one is to first install the markets at one Network Node and later move the branches you want to move to other Network Nodes.",
+            "updated": 1614354841546
+        },
+        {
+            "style": "List",
+            "text": "Install All in All Nodes: In this approach, you first create all Network Nodes and the install the markets you wish to work with, the install procedure will install each market at all nodes, so then you will need to delete all the duplicate stuff by hand."
+        },
+        {
+            "style": "Title",
+            "text": "Install Market"
         },
         {
             "style": "Text",
-            "text": "In a network of nodes, each node has it’s own web server, web sockets server, and task server."
+            "text": "The procedure to setup a Network Node is by executing the Install Market menu option of a certain Market node. This is a complex procedure that will connect all the dots for you from A to Z. You can do by hand what this procedure does, but it will take you a lot of time and you will need to know all the details of how to connect everything together.",
+            "updated": 1614355139550
+        },
+        {
+            "style": "Important",
+            "text": "One of the most important things to undestand of this procedure is that it takes into account the things you have at the workspace at the moment you run this command. "
+        },
+        {
+            "style": "Text",
+            "text": "For exmaple: if you have 2 trading systems, it will automatically create backtesting and live trading tasks for both of them.",
+            "updated": 1614355238724
+        },
+        {
+            "style": "Text",
+            "text": "The procedure will also be applied to each Network Node found at the workspace assuming that you will later delete the Tasks or other branches you don't need. But in order to do so, you need to know what is being added as a result of this execution."
+        },
+        {
+            "style": "Subtitle",
+            "text": "At the Network Hierarchy"
+        },
+        {
+            "style": "Text",
+            "text": "Here this procedure will essentially add 2 thigs:"
+        },
+        {
+            "style": "List",
+            "text": "Tasks: This includes Backtesting, Live Trading, Learning or any other supported type of Task."
+        },
+        {
+            "style": "List",
+            "text": "Data Products: You can think of this as references to the physical data. This definitions are what tells the system in general where the data is physically located.",
+            "updated": 1614355507769
+        },
+        {
+            "style": "Subtitle",
+            "text": "At the Charting Space Hierarchy"
+        },
+        {
+            "style": "Text",
+            "text": "Here this procedure will add Time Machine s with Timeline Chart s inside of them. Deeper inside this brach, you will find Layer s node, which references the Data Products at the Network Hierarchy. That reference is crucial for the system to know when you turn on a Layer at the charts, where the data needs to be fetched from (which Client).",
+            "updated": 1614357169345
+        },
+        {
+            "style": "Success",
+            "text": "If you understand all the previous relationships, you can go ahead and install a market in all Network Node s at the same time, and then try to delete all the duplicated definitions. If you don't have all that clear, it is recommended that you install all your markets at one single Network Node and then move the braches you want from one Network Node to the other."
+        },
+        {
+            "style": "Title",
+            "text": "Moving Tasks"
+        },
+        {
+            "style": "Text",
+            "text": "Assuming you will go with the startegy of installing one Network Node first, this is what you need to know on how to move Task s to other Network Nodes.",
+            "updated": 1614357184981
+        },
+        {
+            "style": "Text",
+            "text": "There are different types of Tasks and each one needs to be moved together with their corresponding Data Products definitions in order for the system to being able to track dependencies. The Data Products dedinitions are the one under the Data Storage node of a Network Node. They are subdivided in different branches depending of the Type of Data, Project, Exchange, Market and many other stuff, but in the end you will find Data Product nodes which reference Product Definitions inside Mines. When you move Tasks from one Network Node to other you also need to move the related brach at Data Storage to the same Network Node.",
+            "updated": 1614356433556
+        },
+        {
+            "style": "Text",
+            "text": "Always remember that the system just needs you to define where you want the Tasks to run and the Data to be Hosted, but there is an implcit relationship between Tasks and Data because a Task produces certain Data, and those definition needs to be consistent and be found at the same Network Node."
+        },
+        {
+            "style": "Warning",
+            "text": "If you only move a Task and do not move the related Data Products definitions most likely things will stop working right there."
+        },
+        {
+            "style": "Success",
+            "text": "The refinitions at the Charting System hirierchy do not need to be changed at all. They will continue pointing to the Data Products they were pointint to no matter where you moved those Data Products definitions. If you changed them from one Network Node to another, the Pltotter will simply fetch the data from the Network Node where you left thos Data Product s definitions."
         },
         {
             "style": "Note",
-            "text": "The default web server and web sockets ports used by Superalgos are 34248 and 18041 respectively. If you wish to change these defaults, you must modify the.env file on the root folder, and also \\TaskServer\\.env.",
-            "updated": 1611343376974
+            "text": "Whey we say moving what we are saying is detaching a Node brach and re attaching that same brach somewhere else. Note that if you for example make a backup of a branch, restore that somehwhere else without deleting the original branch first, you will inadvertently be creating new ids for each Node at this branch and messing up with incomming references, which will produce things not to work anymoere.",
+            "updated": 1614357412126
+        },
+        {
+            "style": "Text",
+            "text": ""
         },
         {
             "style": "Title",
@@ -53,15 +140,6 @@
             "style": "Text",
             "text": "1. Set up Superalgos on each machine. Once Superalgos is installed, run the system for the first time using the node run noBrowser command. The noBrowser parameter runs the Client without firing up the UI."
         },
-        {
-            "style": "Success",
-            "text": "On computers with 8 GB of RAM or less, such as minicomputers or single-board computers like the Raspberry Pi, use node run noBrowser minMemo to run the app. minMemo sets up the system to run with minimal RAM requirements.",
-            "updated": 1611343440835
-        },
-        {
-            "style": "Text",
-            "text": "Leave the machine online, with the Client running."
-        },
         {
             "style": "Text",
             "text": "2. Run the UI to access the home node."

Projects/Superalgos/Schemas/Docs-Topics/trading-farms-003-trading-farms-operating-a-trading-farm.json

@@ -1,6 +1,6 @@
 {
     "topic": "Trading Farms",
-    "pageNumber": "4",
+    "pageNumber": "3",
     "type": "Trading Farms - Operating a Trading Farm",
     "definition": {
         "text": "Operate tasks in remote nodes pretty much as if they were running on the home node. ",

Projects/Superalgos/Schemas/Docs-Topics/trading-farms-003-trading-farms-set-up-each-node-on-the-network.json

@@ -1,25 +0,0 @@
-{
-    "topic": "Trading Farms",
-    "pageNumber": "3",
-    "type": "Trading Farms - Set Up Each Node on the Network",
-    "definition": {
-        "text": "Install the markets you wish to work with and remove the tasks where you don't need them. ",
-        "updated": 1614352898429
-    },
-    "paragraphs": [
-        {
-            "style": "Text",
-            "text": "When you install a market, several structures of nodes are created on each of the main sections of the network hierarchy for all nodes defined on the network: data storage, data mining, testing environment, and production environment, as well as the corresponding time machines in the charting space.",
-            "updated": 1611344492853
-        },
-        {
-            "style": "Text",
-            "text": "In other words, the process of installing a new market replicates all data structures in all nodes."
-        },
-        {
-            "style": "Success",
-            "text": "You will most likely want to delete the tasks that you don’t need on certain nodes, and keep only those that you wish each node to handle.",
-            "updated": 1611344498395
-        }
-    ]
-}

Projects/Superalgos/TS/Process-Modules/Dataset.js

@@ -190,7 +190,7 @@ exports.newSuperalgosProcessModulesDataset = function (processIndex) {
         A. We are interested in a particular Exchange and Market.
         B. We have the Network we received from the UI whe the user ran the Task.
         C. We have the Product Definition that is the parent node of the Data Set represented by this object.
-        D. This Product Definition is referenced by one or more Data Products nodes at some branch of the Network Hirierchy.
+        D. This Product Definition is referenced by one or more Data Products nodes at some branch of the Network Hierarchy.
         E. We need to find the right Data Product node, the one who is itself a descendent of the Market Data Products
         node that is referencing the market and exchange that we have as a parameter of this function.