Merge branch 'Superalgos:develop' into develop

2021-05-25 20:24:53 -05:00
commit 3f3b08e578
@@ -1,6 +1,13 @@
 {
    "type": "API Response Field Reference",
    "menuItems": [
+        {
+            "action": "Configure",
+            "actionFunction": "uiObject.configEditor.activate",
+            "label": "Configure",
+            "iconPathOn": "configuration",
+            "iconPathOff": "configuration"
+        },
        {
            "action": "Delete UI Object",
            "askConfirmation": true,
@@ -11,6 +18,12 @@
            "actionFunction": "payload.executeAction"
        }
    ],
+    "editors": {
+        "config": true
+    },
+    "initialValues": {
+        "config": "{\"nodePath\": \"\"}"
+    },
    "title": [
        "Use Reference Parent"
    ],
@@ -31,7 +31,7 @@
        "config": true
    },
    "initialValues": {
-        "config": "{ \n  \"codeName\": \"Multi-Period-Market|Daily\", \n  \"normalWaitTime\": 0, \n  \"retryWaitTime\": 10000, \n  \"framework\": { \n    \"name\": \"Multi-Period-Market|Daily\" \n  }\n}"
+        "config": "{ \n  \"codeName\": \"Multi-Period-Market|Multi-Period-Market|From-One-Min-To-Multi-Period-Daily|From-One-Min-To-Multi-Period-Market\", \n  \"normalWaitTime\": 0, \n  \"retryWaitTime\": 10000, \n  \"framework\": { \n    \"name\": \"Multi-Period-Market|Multi-Period-Market|From-One-Min-To-Multi-Period-Daily|From-One-Min-To-Multi-Period-Market\" \n  }\n}"
    },
    "level": 1,
    "attachingRules": {
@@ -7,8 +7,16 @@
    "paragraphs": [
        {
            "style": "Text",
-            "text": "An API Data Fetcher Bots is similar to a Sensor Bot, except that the second extracts data from exchanges, while the first one can do it from any reachable API, including exchanges.",
-            "updated": 1621022763112
+            "text": "An API Data Fetcher Bots is similar to a Sensor Bot, except that the second need dev users to code an algo for extracting the data, while the first one can do it without coding, just using an API Map.",
+            "updated": 1621969800744
+        },
+        {
+            "style": "Include",
+            "text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Main Workflow"
+        },
+        {
+            "style": "Title",
+            "text": "Configuration"
        }
    ]
 }
@@ -13,6 +13,37 @@
        {
            "style": "Important",
            "text": "All references originating at the same Record Definition must be to API Data Field s living at the same API Endpoint."
+        },
+        {
+            "style": "Title",
+            "text": "Record Property Configuration"
+        },
+        {
+            "style": "Subtitle",
+            "text": "Properties"
+        },
+        {
+            "style": "Success",
+            "text": "nodePath is a property that is required when the API Response Field referenced is not under the Data Root object within the response structure.",
+            "updated": 1621969132831
+        },
+        {
+            "style": "Text",
+            "text": "The response from the API Server might include some header type of data and the actual data requested. Since the response is a JSON object, then the actual data might be located somewhere within the structure of that object. The API Response Schema node has a property nodePath that allows users to declare the path to the actual data requested, so as to jump the header-like information. We call the API Response Field where the data is held, the Data Root. "
+        },
+        {
+            "style": "Text",
+            "text": "In order to extract individual values for each Record Property, the API Data Fetcher Bot needs to dynamically discover the nodePath for each Record Propety at the response data structure and can do it as long as the API Response Field referenced is under the Data Root. To autodiscover the nodePath it uses the API Response Schame nodePath to locate the Data Root, and then it follows the reference from the API Response Field Reference node to the API Response Field and climb upwards until it reach the API Response Schema. While climbing upwards it discovers the nodePath that it needs to grabb the value from the response data structure. That is how it works.",
+            "updated": 1621972472491
+        },
+        {
+            "style": "Text",
+            "text": "When the referenced API Response Field is outside the scope of the Data Root, the process is not able to auto dicover this path and need users to manually define the nodePath (from the API Response Schema to the API Response Field that holds the value) and set it in this proeprty. That happens for instance when the API Response Field is part of the header of the API Server response.",
+            "updated": 1621972505639
+        },
+        {
+            "style": "Text",
+            "text": ""
        }
    ]
 }
@@ -11,12 +11,33 @@
            "updated": 1621952655717
        },
        {
-            "style": "Subtitle",
+            "style": "Title",
            "text": "Configuration"
        },
        {
            "style": "List",
            "text": "nodePath: the response from the server might include some header type of data and the actual data requested. Since the response is a JSON object, then the actual data might be located somewhere within the structure of that object. This property allows you to declare the path to the actual data requested, so as to jump the header-like information."
+        },
+        {
+            "style": "Subtitle",
+            "text": "Why do we need to jump the header?"
+        },
+        {
+            "style": "Text",
+            "text": "In many cases it will be necesary to make several queries to the API Server in order to fetch all the data (think of pagination). Each page of data arrives at a certain location within the reponse data structure. That location is the one considered the Data Root. "
+        },
+        {
+            "style": "Text",
+            "text": "If you Endpoint does not support pagination, it is still a good practice to define which is the Data Root."
+        },
+        {
+            "style": "Important",
+            "text": "Most Record Properties will be referencing API Response Field nodes that are under the Data Root you defined, but there might be one or two that are not there, but under the response data structure in general, or at a section that could be considered a header. In that case, there is some special step you need to follow. At the API Response Field Reference node cofing, you will need to set up a nodePath property as explained below."
+        },
+        {
+            "style": "Text",
+            "text": "Check the API Response Field Reference node for details on how to setup this property.",
+            "updated": 1621971441459
        }
    ]
 }
@@ -27,8 +27,8 @@
        },
        {
            "style": "Title",
-            "text": "Record Property Configuration",
-            "updated": 1609788569963
+            "text": "Property Configuration",
+            "updated": 1621969011509
        },
        {
            "style": "Subtitle",
@@ -45,7 +45,17 @@
        },
        {
            "style": "List",
-            "text": " isCalculated determines if the field is stored in the dataset or if, instead, is calculated at a later stage. A value true* means that the record is not stored. In the case isCaculated is not defined in the configuration, it is assumed to be false."
+            "text": "isCalculated determines if the field is stored in the dataset or if, instead, is calculated at a later stage. A value true* means that the record is not stored. In the case isCaculated is not defined in the configuration, it is assumed to be false.",
+            "updated": 1621972714008
+        },
+        {
+            "style": "List",
+            "text": "aggregationMethod defines which is the method that is going to be used in order to aggregate information when converting one Dataset Type into another which requires data aggregation."
+        },
+        {
+            "style": "Include",
+            "text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Data Aggregation",
+            "updated": 1621974902589
        }
    ]
 }
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "1",
+    "pageNumber": 1,
    "type": "Data Mining - Introduction",
    "definition": {
        "text": "Data Mining in Superalgos means extracting data from external sources and procesing that data in order to have it ready for consumption by trading strategies.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "2",
+    "pageNumber": 2,
    "type": "Data Mining - Workflows",
    "definition": {
        "text": "Superalgos enable users to setup powerful data mining operations. At this page we will learn the workflows involved.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "3",
+    "pageNumber": 3,
    "type": "Data Mining - Dataset Types",
    "definition": {
        "text": "Superalgos organizes data in datasets. These datasets are standarized across the system in different types.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "4",
+    "pageNumber": 4,
    "type": "Data Mining - Datasets Type Conversions",
    "definition": {
        "text": "Processes fetching data from the Internet are optimized for accesing a wide range of APIs available out there. The Dataset Type they use to store raw data is usually not the best one for data consumption within the system. For that reason Dataset Type conversions needs to be done.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "5",
+    "pageNumber": 5,
    "type": "Data Mining - Indicator Frameworks",
    "definition": {
        "text": "Indicator Bots are ready to run custom code, or run code provided by a framework in order to do their job.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "6",
+    "pageNumber": 6,
    "type": "Data Mining - Data Mines",
    "definition": {
        "text": "Data Mines are user created packages of definitions. These definitions usually include Sensor Bot, Indicator Bot and Plotter definitions. The Data Mines shipped with Superalgos are packages contributed by users of the software.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "7",
+    "pageNumber": 7,
    "type": "Data Mining - Managing Execution",
    "definition": {
        "text": "Bots can execute at their own pace or they can wait for other bots to finish executing in order for them to start.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "8",
+    "pageNumber": 8,
    "type": "Data Mining - Data Dependencies",
    "definition": {
        "text": "Indicator Bots have data dependencies. That means that they have as their input a Dataset created by other Bot. ",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "9",
+    "pageNumber": 9,
    "type": "Data Mining - Status Reports",
    "definition": {
        "text": "Status Reports are files produced by each Bot's processes with some key information that the Bot needs to save in order to remember it the next time is run.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "10",
+    "pageNumber": 10,
    "type": "Data Mining - Status Dependencies",
    "definition": {
        "text": "Status Dependencies is a type of relationship between Bots. Sometimes bots don't depend on the Datasets produced by other bots, but they depend directly on the status of other bots.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "11",
+    "pageNumber": 11,
    "type": "Data Mining - Deployments",
    "definition": {
        "text": "Data Minign Bots can be deployed into one or many machines at the same time and they will automatically find the way too coordinate their actions.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "12",
+    "pageNumber": 12,
    "type": "Data Mining - Products",
    "definition": {
        "text": "All Datasets of any type of bot within the Superalgos system are considered products of the Bot that produced them. Why products? Because users will be able to sell hese datasets in the form of subscriptions once the Superalgos Network is implemented.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "13",
+    "pageNumber": 13,
    "type": "Data Mining - Plotting",
    "definition": {
        "text": "Each product might have one Plotter Module associated with it. This Plotter Module has all the definitions needed to interpret the Product's datasets and render the information at the charts.",
@@ -0,0 +1,256 @@
+{
+    "topic": "Data Mining",
+    "pageNumber": 14,
+    "type": "Data Mining - Fetching Data From APIs",
+    "definition": {
+        "text": "Superalgos allow it's users to fetch data from any API on the Internet without needing to write code to do it. Instead, users design an API Map, which is a set of definitions of how a Bot can access a certain API and how to interpret the API Server responses. With this API Maps, a API Data Fetcher Bot can be configured to extract data from any API Server and stored inside the system as Raw Data that later can processed and consumed by other Bots.",
+        "updated": 1621906349521
+    },
+    "paragraphs": [
+        {
+            "style": "Block",
+            "text": "Main Workflow",
+            "updated": 1621951016327
+        },
+        {
+            "style": "Title",
+            "text": "How to fetch data from an API?",
+            "updated": 1621951311144
+        },
+        {
+            "style": "Text",
+            "text": "The main workflow to pull this out is quite straight forward and it does not require coding at all. Follow this guide and you will be using data from any source at the Internet from your trading strategies in no time."
+        },
+        {
+            "style": "List",
+            "text": "1. Choose an API: Current version is limited to public APIs only. Secured APIs comming soon. "
+        },
+        {
+            "style": "List",
+            "text": "2. Design an API Map: Read the APIs online documentation and identify the Endpoints that returns the data you would like to fetch. Carefully map those Endpoints defining the parameters the API Server expects at each call, and also how the API Server response is formatted."
+        },
+        {
+            "style": "List",
+            "text": "3. Setup an API Data Fetcher Bot: Create a Data Mine for this API and create there an API Data Fetcher Bot. From that Bot's Record Properties you will need to reference the fields returned by the API Server defined at the API Map, defining in this way which fields from each response you are going to save to disk. You might also need to define query or path parameters here. "
+        },
+        {
+            "style": "List",
+            "text": "4. Setup an Indicator Bot for Dataset Type Conversion: The Raw Data saved to disk needs to be organized for consumption. You will need to setup an Indicator Bot that will do a Dataset Type Conversion from Raw Data to a standard Dataset Type. This process will do data aggregation in order to generate the elements with a fixed time range for every Time Frame possible. ",
+            "updated": 1621973482403
+        },
+        {
+            "style": "List",
+            "text": "5. Setup Indicator Bots: Once the data is converted, you can extract from it specific Datasets defining a set of indicators you can build from that data. These Indicators can be just a reorganization of the data, with some adjustments on the names of fields and splitting the data in different indicators, or it can also include some data processing / transformations if you wish (this would require some coding)."
+        },
+        {
+            "style": "List",
+            "text": "6. Setup Plotters: These last indicators can have their plotters if you wish."
+        },
+        {
+            "style": "List",
+            "text": "7. Consume Data from Trading Strategies: These indicators can be consumed from your trading strategies."
+        },
+        {
+            "style": "Success",
+            "text": "There is a Workspace called API Data Fetcher Bot Demo with a simple working example of the previous steps you can take as reference."
+        },
+        {
+            "style": "Block",
+            "text": "Data Aggregation"
+        },
+        {
+            "style": "Title",
+            "text": "How does data aggregation works?"
+        },
+        {
+            "style": "Text",
+            "text": "When Raw Data is fetched from the Internet, it needs to be converted into a Dataset Type either of Multi-Period-Daily and Multi-Period-Market types. Usually Raw Data is a timestamped dataset with a timesStamp field or it has a begin and end properties if they are fragmented in time slots of one minute.",
+            "updated": 1621973202254
+        },
+        {
+            "style": "Text",
+            "text": "From there the conversion process needs to generate elements with time slots of 2 min, 3 min, 4 min, 5 min, 10, min, ... up to 24hs. To achieve this, each field needs to be aggregated in a certain way."
+        },
+        {
+            "style": "Text",
+            "text": "At the Bot where this conversion happens, the Record Property nodes needs to have a config property agregattionMode that defines how this is going to be handled. Here is a list of the supported aggregated methods that can be used during this conversion procedure.",
+            "updated": 1621973294298
+        },
+        {
+            "style": "List",
+            "text": "First: This means that the fisrt value found within the time range being processed, will define the value of the Record Property."
+        },
+        {
+            "style": "List",
+            "text": "Last: This means that from all values found at the current time range, the last will be selected for the final value of the Record Property node being evaluated."
+        },
+        {
+            "style": "List",
+            "text": "Min: This means that the mathematical minimun value from all values found in the time range will be used as the value for the Record Proeprty."
+        },
+        {
+            "style": "List",
+            "text": "Max: This means that the mathematical maximun value from all values found in the time range will be used as the value for the Record Proeprty."
+        },
+        {
+            "style": "List",
+            "text": "Sum: This means that the sum of all values found in the time range will be used as the value for the Record Proeprty."
+        },
+        {
+            "style": "List",
+            "text": "Avg: This means that the average of all values found in the time range will be used as the value for the Record Proeprty."
+        },
+        {
+            "style": "Block",
+            "text": "Step By Step"
+        },
+        {
+            "style": "Title",
+            "text": "Step by step guite to fetch data from an API"
+        },
+        {
+            "style": "Text",
+            "text": "This is a detailed guide on how to setup a data mining operation fetching data from an Internet API.",
+            "updated": 1621975839421
+        },
+        {
+            "style": "List",
+            "text": "Step #1: Open the API Data Fetcher Bot Demo Plugin Workspace and study it for a while. Run the tasks in there, watch while data is processed, check the charts and finally run the Trading Task that uses the data processed.",
+            "updated": 1621976539133
+        },
+        {
+            "style": "List",
+            "text": "Step #2: Open a second UI (this will prevent the previous one to save the changes, but you don't need to do that). You will use the first UI to check from time to time how nodes are configured there, and the second UI to build your own data mining operation from scratch. The rest of the staps you will do it at the second UI.",
+            "updated": 1621976658929
+        },
+        {
+            "style": "List",
+            "text": "Step #3: Select an API from the Internet with data that might be usefull at your Trading Strategy.",
+            "updated": 1621976668506
+        },
+        {
+            "style": "List",
+            "text": "Step #4: Read the API's documentation, indentify the endpoints, what parameters they require and what data they will provide you.",
+            "updated": 1621976675920
+        },
+        {
+            "style": "List",
+            "text": "Step #5: At Superalgos, get a clean workspace, without plugins pre-installed.",
+            "updated": 1621976682833
+        },
+        {
+            "style": "List",
+            "text": "Step #6: Create an API Map node, and start mapping the documented API by adding children to it.",
+            "updated": 1621976688595
+        },
+        {
+            "style": "List",
+            "text": "Step #7: For each node you add, read the node's doc page. This will save you time trying to guess how all this works.",
+            "updated": 1621976694068
+        },
+        {
+            "style": "List",
+            "text": "Step #8: Define one API Endpoint first, once you have the first one working, go for the rest of the enpoints available.",
+            "updated": 1621976699911
+        },
+        {
+            "style": "List",
+            "text": "Step #9: Define the query or path parameters and the the responses. Find at each node's help the details on how to do it right.",
+            "updated": 1621976710636
+        },
+        {
+            "style": "List",
+            "text": "Step #10: Create a Trading System with an empty Strategy. This will allow the Install New Market procedure to create trading tasks referencing this Trading System.",
+            "updated": 1621976716866
+        },
+        {
+            "style": "List",
+            "text": "Step #11: Create a new Data Mine woth the name of the company that host the API.",
+            "updated": 1621976722023
+        },
+        {
+            "style": "List",
+            "text": "Step #12: Create at that Data Mine and a API Data Fetcher Bot. Configure the Bot's nodes following the example at the UI #1.",
+            "updated": 1621976915804
+        },
+        {
+            "style": "List",
+            "text": "Step #13: Install a new market. This procedure is going to setup many things for you. It will detect the API Map, and the Data Mine you created and will create appropiate Tasks to run your API Fertcher Bot.",
+            "updated": 1621976960858
+        },
+        {
+            "style": "List",
+            "text": "Step #14: Pay attention to the nodePath property at the API Response Schema node of the API Map. Check the node's docs page for details."
+        },
+        {
+            "style": "List",
+            "text": "Step #15: Define at your API Data Fetcher Bot Product Definition Record Properties the fields that you would like to save from the API Server response. You don't need to save everything, only what makes sense to you."
+        },
+        {
+            "style": "List",
+            "text": "Step #16: Establish a reference from each Record Property child API Response Field Reference node to a node at the API Map of type API Response Field. Do it carefully for every Record Property."
+        },
+        {
+            "style": "List",
+            "text": "Step #17: Pay attention to the special case of the needed timeStamp Record Property, specially because it might be pointing to the header of the API response. Read the docs of the API Response Field Reference node to set up the nodePath property if necesary."
+        },
+        {
+            "style": "List",
+            "text": "Step #18: Setup an your Data Mine an Indicator Bot and configure their Process Definition nodes to be of the framework that converts from One-Min-Daily to Multi-Period-Daily and Multi-Perio-Market types of Datasets. See how it was done at the UI #1.",
+            "updated": 1621977642805
+        },
+        {
+            "style": "List",
+            "text": "Step #19: At this Dataset Type Converter Indicator Bot, defined the Record Property nodes based on the information you would lile to convert from Raw Data. Add a begin and end Record Property even if it does not exist at the Raw Data. "
+        },
+        {
+            "style": "List",
+            "text": "Step #20: Configure at each Record Proeprty the aggregationMode property. Check the docs of that node for details."
+        },
+        {
+            "style": "List",
+            "text": "Step #21: If you wish you can create multiple other Indicator Bots to cut the dataset in different groups of Record Properties."
+        },
+        {
+            "style": "List",
+            "text": "Step #22: Setup a Plotter at your new Data Mine, and reference it from the Product Definition of the latest Indicator Bot created."
+        },
+        {
+            "style": "List",
+            "text": "Step #23: Now you are ready to run the Task where the Data Fetcher Bot Process Instance is defined. "
+        },
+        {
+            "style": "List",
+            "text": "Step #24: If something did not work, check the workspace map to see if there is a node with an error. Locate the node, click on the error and see the error's page at the doc. Try to understand what is wrong, fix it and try again."
+        },
+        {
+            "style": "List",
+            "text": "Step #25: Iterate through step #24 until everything is working. Re read node's documentations until you get the details of what it explains there. Reread this guide or this whole Topic from the start, until the concepts explained sinks in. "
+        },
+        {
+            "style": "List",
+            "text": "Step #26: If after digesting the docs, and following all this advice you stil can not put it to work, go to the Telegram Support group and ask for help."
+        },
+        {
+            "style": "List",
+            "text": "Step #27: Once all is running well, set up your Trading Stragegy in a way that allows you to test if you have acces to the prcessed data, the one provided by the latest Indicator Bots created."
+        },
+        {
+            "style": "List",
+            "text": "Step #29: Once the data is correctly consumed by your Trading Strategy iterate throug this guide by adding another API Endpoint to the API Map. You will need for that a new API Fetcher Bot.",
+            "updated": 1621978484377
+        },
+        {
+            "style": "List",
+            "text": "Step #30: Once you have sucessfully mapped the whole API and you have all the needed API Data Fetcher, Indicator Bots and Plotters tested, you are ready to share your API Map and Data Mine with the community as plugins, and get in exchange SA tokens for your contribution. "
+        },
+        {
+            "style": "Text",
+            "text": ""
+        },
+        {
+            "style": "Text",
+            "text": ""
+        }
+    ]
+}
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "14",
+    "pageNumber": 15,
    "type": "Data Mining - Consuming Data From Strategies",
    "definition": {
        "text": "Traing Bots can have as Data Dependencies datasets from any Indicator Bots. This allows Trading Strategies to act upon the data processed by Indicator Bots.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "15",
+    "pageNumber": 16,
    "type": "Data Mining - Data Mine Hierarchy",
    "definition": {
        "text": "Data mines enable the processing of data by consuming datasets others have produced, performing calculations, and outputting a new value-added data product that others may consume.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "16",
+    "pageNumber": 17,
    "type": "Data Mining - Indicator Bots",
    "definition": {
        "text": "Indicator bots enable the processing of data by consuming datasets others have produced, performing calculations, and outputting a new value-added data product that others may consume.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "17",
+    "pageNumber": 18,
    "type": "Data Mining - Indicator Bots - Process Definition",
    "definition": {
        "text": "The process definition is the compilation of configurations, dependencies, and references that make up a process.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "18",
+    "pageNumber": 19,
    "type": "Data Mining - Indicator Bots - Product Definition",
    "definition": {
        "text": "The product definition is the compilation of configurations, references, and JavaScript code that make up the data product.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "19",
+    "pageNumber": 20,
    "type": "Data Mining - Indicator Bots - Data Mining Workflow",
    "definition": {
        "text": "When running on a data mining operation, indicators follow a cycle designed to fulfill their purpose: to process data others have produced and make the result available for others to consume.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "20",
+    "pageNumber": 21,
    "type": "Data Mining - Indicator Bots - Coding the Indicator's Logic",
    "definition": {
        "text": "Learn how to use data dependencies, the variable object, and how to deal with indicators with irregular periods.",
@@ -1,70 +0,0 @@
-{
-    "topic": "Data Mining - Data Mines",
-    "pageNumber": "22",
-    "type": "Data Mining - Plotters - Plotter Panel",
-    "definition": {
-        "text": "A plotter panel allows building panels with numerical representations of data.",
-        "updated": 1611432021846
-    },
-    "paragraphs": [
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Plotter Panel->Definition",
-            "updated": 1611432049796
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Plotter Panel->Content",
-            "updated": 1611432052734
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Concept->Reusable Snippets->Note for Hierarchy Tables"
-        },
-        {
-            "style": "Hierarchy",
-            "text": "Superalgos->Plotter Panel->4",
-            "updated": 1611432061472
-        },
-        {
-            "style": "Title",
-            "text": "Panel Data",
-            "updated": 1611432322437
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Panel Data->Definition",
-            "updated": 1611432326601
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Panel Data->Content",
-            "updated": 1611432330023
-        },
-        {
-            "style": "Subtitle",
-            "text": "Data Formula",
-            "updated": 1611432338583
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Data Formula->Definition",
-            "updated": 1611432348376
-        },
-        {
-            "style": "Subtitle",
-            "text": "Text Style",
-            "updated": 1611432394440
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Text Style->Definition",
-            "updated": 1611432402298
-        },
-        {
-            "style": "Include",
-            "text": "Superalgos->Node->Text Style->Content",
-            "updated": 1611432405386
-        }
-    ]
-}
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "21",
+    "pageNumber": 22,
    "type": "Data Mining - Plotters",
    "definition": {
        "text": "Plotters read data products created by bots and map certain coordinate points to use them as vertices for polygons, which are assigned styles to create graphic representations of the underlying data.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "23",
+    "pageNumber": 23,
    "type": "Data Mining - Plotters - Shapes",
    "definition": {
        "text": "Shapes define coordinate points to use as vertex for polygons, which are assigned styles.",
@@ -1,6 +1,6 @@
 {
    "topic": "Data Mining",
-    "pageNumber": "24",
+    "pageNumber": 24,
    "type": "Data Mining - How to Build an Indicator - Video Series",
    "definition": {
        "text": "This video series goes through the entire process of building an indicator, step by step.",
@@ -1,57 +0,0 @@
-{
-    "topic": "Data Mining",
-    "pageNumber": "13.1",
-    "type": "Data Mining - Fetching Data From APIs",
-    "definition": {
-        "text": "Superalgos allow it's users to fetch data from any API on the Internet without needing to write code to do it. Instead, users design an API Map, which is a set of definitions of how a Bot can access a certain API and how to interpret the API Server responses. With this API Maps, a API Data Fetcher Bot can be configured to extract data from any API Server and stored inside the system as Raw Data that later can processed and consumed by other Bots.",
-        "updated": 1621906349521
-    },
-    "paragraphs": [
-        {
-            "style": "Block",
-            "text": "Main Workflow",
-            "updated": 1621951016327
-        },
-        {
-            "style": "Title",
-            "text": "How to fetch data from an API?",
-            "updated": 1621951311144
-        },
-        {
-            "style": "Text",
-            "text": "The main workflow to pull this out is quite straight forward and it does not require coding at all. Follow this guide and you will be using data from any source at the Internet from your trading strategies in no time."
-        },
-        {
-            "style": "List",
-            "text": "1. Choose an API: Current version is limited to public APIs only. Secured APIs comming soon. "
-        },
-        {
-            "style": "List",
-            "text": "2. Design an API Map: Read the APIs online documentation and identify the Endpoints that returns the data you would like to fetch. Carefully map those Endpoints defining the parameters the API Server expects at each call, and also how the API Server response is formatted."
-        },
-        {
-            "style": "List",
-            "text": "3. Setup an API Data Fetcher Bot: Create a Data Mine for this API and create there an API Data Fetcher Bot. From that Bot's Record Properties you will need to reference the fields returned by the API Server defined at the API Map, defining in this way which fields from each response you are going to save to disk. You might also need to define query or path parameters here. "
-        },
-        {
-            "style": "List",
-            "text": "4. Setup an Indicator Bot for Dataset Type Conversion: The Raw Data saved to disk needs to be organized for consumption. You will need to setup an Indicator Bot that will do a Dataset Type Conversion from Raw Data to a standard Dataset Type."
-        },
-        {
-            "style": "List",
-            "text": "5. Setup Indicator Bots: Once the data is converted, you can extract from it specific Datasets defining a set of indicators you can build from that data. These Indicators can be just a reorganization of the data, with some adjustments on the names of fields and splitting the data in different indicators, or it can also include some data processing / transformations if you wish (this would require some coding)."
-        },
-        {
-            "style": "List",
-            "text": "6. Setup Plotters: These last indicators can have their plotters if you wish."
-        },
-        {
-            "style": "List",
-            "text": "7. Consume Data from Trading Strategies: These indicators can be consumed from your trading strategies."
-        },
-        {
-            "style": "Success",
-            "text": "There is a Workspace called API Data Fetcher Bot Demo with a simple working example of the previous steps you can take as reference."
-        }
-    ]
-}
@@ -78,7 +78,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {

                    if (TS.projects.superalgos.globals.taskConstants.TASK_NODE.bot.processes[processIndex].apiMapReference === undefined) {
                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                            processIndex, 
+                            processIndex,
                            MODULE_NAME,
                            'API Data Fetcher Bot',
                            undefined,
@@ -90,7 +90,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                    let apiMap = TS.projects.superalgos.globals.taskConstants.TASK_NODE.bot.processes[processIndex].apiMapReference.referenceParent
                    if (apiMap === undefined) {
                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                            processIndex, 
+                            processIndex,
                            MODULE_NAME,
                            'API Data Fetcher Bot',
                            undefined,
@@ -106,7 +106,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {

                    if (apiMap.config.hostName === undefined || apiMap.config.hostName === "") {
                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                            processIndex, 
+                            processIndex,
                            MODULE_NAME,
                            'API Data Fetcher Bot',
                            { missingProperty: "hostName" },
@@ -131,7 +131,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {

                    if (apiMap.apiVersions.length === 0) {
                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                            processIndex, 
+                            processIndex,
                            MODULE_NAME,
                            'API Data Fetcher Bot',
                            undefined,
@@ -141,7 +141,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                    }
                    if (processNode.processOutput === undefined) {
                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                            processIndex, 
+                            processIndex,
                            MODULE_NAME,
                            'API Data Fetcher Bot',
                            undefined,
@@ -204,7 +204,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                        */
                                        if (endpointNodeFound === undefined) {
                                            TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                                                processIndex, 
+                                                processIndex,
                                                MODULE_NAME,
                                                'API Data Fetcher Bot',
                                                undefined,
@@ -226,7 +226,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                            We can not reference fields from different enpoints.
                                            */
                                            TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                                                processIndex, 
+                                                processIndex,
                                                MODULE_NAME,
                                                'API Data Fetcher Bot',
                                                undefined,
@@ -245,7 +245,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                API Response Field.
                                */
                                TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                                    processIndex, 
+                                    processIndex,
                                    MODULE_NAME,
                                    'API Data Fetcher Bot',
                                    undefined,
@@ -478,7 +478,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                    if (apiResponseSchemaNode === undefined) {
                                        errorCodeReceived = true
                                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                                            processIndex, 
+                                            processIndex,
                                            MODULE_NAME,
                                            'API Data Fetcher Bot',
                                            { resposeCodeReceived: response.statusCode, detais: 'The response code ' + response.statusCode + ' received is not associated with any API Query Response node.' },
@@ -604,7 +604,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                }
                                default: {
                                    TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                                        processIndex, 
+                                        processIndex,
                                        MODULE_NAME,
                                        'API Data Fetcher Bot',
                                        { datasetType: dataset.config.codeName },
@@ -786,7 +786,7 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                        There must be a Record Property with codeName 'timestamp' defined.
                                        */
                                        TS.projects.superalgos.utilities.errorHandlingFunctions.throwHandledException(
-                                            processIndex, 
+                                            processIndex,
                                            MODULE_NAME,
                                            'API Data Fetcher Bot',
                                            undefined,
@@ -866,7 +866,10 @@ exports.newSuperalgosBotModulesFetchingProcess = function (processIndex) {
                                            2) It was calculated automatically.
                                            */

-                                            let nodePath = recordProperty.config.nodePath
+                                            let nodePath
+                                            if (recordProperty.apiResponseFieldReference.config.nodePath !== "") {
+                                                nodePath = recordProperty.apiResponseFieldReference.config.nodePath
+                                            }

                                            if (nodePath === undefined) {
                                                nodePath = recordPropertiesNodePathMap.get(recordProperty.config.codeName)