Merge pull request #979 from Luis-Fernando-Molina/develop

Contribution: API Data Fetcher Docs improved

Projects/Superalgos/Schemas/Docs-Nodes/api-endpoint.json

@@ -5,9 +5,23 @@
         "updated": 1621017376741
     },
     "paragraphs": [
+        {
+            "style": "Subtitle",
+            "text": "How to use this node?",
+            "updated": 1621949863470
+        },
         {
             "style": "Text",
-            "text": "Right click and select the pencil button to enter edit mode."
+            "text": "Read the docs of the API you are trying to map, and indentigy the commands that it support (endpoints)."
+        },
+        {
+            "style": "Include",
+            "text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Main Workflow",
+            "updated": 1621951345309
+        },
+        {
+            "style": "Title",
+            "text": "Configuration"
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-map.json

@@ -12,6 +12,23 @@
         },
         {
             "style": "Subtitle",
+            "text": "How is it used?"
+        },
+        {
+            "style": "Text",
+            "text": "To use an API Map users need to setup a Task for an API Data Fetcher Bot. This set up is done automatically as part of the Install New Market procedure if an API Map is detected at the Worksapce. From within that Task, a reference to an API Map is established. "
+        },
+        {
+            "style": "Text",
+            "text": "When the Task is run, the UI will send to the Task Server all the definitions inside the API Map so that the API Data Fetcher Bot can use them to connect to the API Server, fetch data from it, and correctly interpret the responses."
+        },
+        {
+            "style": "Include",
+            "text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Main Workflow",
+            "updated": 1621951793419
+        },
+        {
+            "style": "Title",
             "text": "Configuration"
         },
         {

Projects/Superalgos/Schemas/Docs-Nodes/api-path-parameter.json

@@ -16,8 +16,16 @@
         },
         {
             "style": "List",
-            "text": "replaceBy: this property allows you to instruct the API Fetcher Bot to set a value to this parameter. The possible values are: @BaseAsset and @QuotedAsset",
-            "updated": 1621018303336
+            "text": "codeName: here you must write the exact string that is going to be used to form the URL path. ",
+            "updated": 1621950838675
+        },
+        {
+            "style": "List",
+            "text": "replaceBy: this property allows you to instruct the API Fetcher Bot to set a value to this parameter. The possible values are: @BaseAsset and @QuotedAsset"
+        },
+        {
+            "style": "Important",
+            "text": "The replacedBy property is the way users have to send some contextual values to the API Server like for example the Base Asset or Quoted Asset of the Defautl Market."
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-path-parameters.json

@@ -9,6 +9,19 @@
             "style": "Text",
             "text": "When found at an API Map they define the default values for each parameter, whicle when found at a Data Mine, they define the actual values to be used, which overrides the defaults.",
             "updated": 1621022867854
+        },
+        {
+            "style": "Title",
+            "text": "How to identify them?"
+        },
+        {
+            "style": "Text",
+            "text": "The API your are trying to map shold have it's API Path Parameters documented. If not, check any example provided there of how to request the data and see if there are inner folders below the folder of the API Enpoint. These folders are, each one, one API Path Parameter that needs to be defined. As everything else in Superalgos, this current node children are considered clockwise, that defines the order in which the path will be constructed at runtime."
+        },
+        {
+            "style": "Include",
+            "text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Main Workflow",
+            "updated": 1621951167309
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-query-parameters.json

@@ -7,8 +7,16 @@
     "paragraphs": [
         {
             "style": "Text",
-            "text": "When found at the API Map will define the default values for each parameter. When found at the Data Mine will define the actual values to be used, these would replace the defaults.",
-            "updated": 1621022566230
+            "text": "In other words, Query Parameters are the parameters sent at any URL after the question mark ?",
+            "updated": 1621954104568
+        },
+        {
+            "style": "Text",
+            "text": "Query Parameters in general do not need to be defined in any specific orders, so feel free to add children for this node in whatever order you like."
+        },
+        {
+            "style": "Success",
+            "text": "When found at the API Map will define the default values for each parameter. When found at the Data Mine will define the actual values to be used, these would replace the defaults."
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-query-response.json

@@ -16,7 +16,8 @@
         },
         {
             "style": "Success",
-            "text": "If you wish to be able to react to other error codes, you should also have more of this nodes with the codeName of each of the status you want to cover."
+            "text": "If you wish to be able to react to other error codes, you should also have more of these nodes with the codeName of each of the status you want to cover. If the API Data Fetcher Bot receives a response code that has not been defined, it will stop there the processing.",
+            "updated": 1621952346052
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-query-responses.json

@@ -7,7 +7,12 @@
     "paragraphs": [
         {
             "style": "Text",
-            "text": "Right click and select the pencil button to enter edit mode."
+            "text": "For a single API Endpoint an API Server might have different possible responses. Each response type is differentiated from each other by the response code / status.",
+            "updated": 1621952454860
+        },
+        {
+            "style": "Text",
+            "text": "Each child of this node is a different response code / status from the API Server. Users need to add at least the most common response code (200) in order to start testing this API Endpoint. After that is recommended to add all other response codes as well, otherwise the API Data Fetcher Bot will stop when an unhandled response code is received."
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-response-field.json

@@ -19,6 +19,39 @@
             "style": "List",
             "text": "fieldType: This property declares if the data is number, a string an object or an array, those are the 4 possible values.",
             "updated": 1621022104672
+        },
+        {
+            "style": "List",
+            "text": "description: The API Map is expected to be used not only by the user who created, but once contributed as a Plugin, by all other users. This property is useful to store the description of this field as it was written at the API Server documentation, to avoid other users to go and look for it."
+        },
+        {
+            "style": "Subtitle",
+            "text": "Field Type Possible Values"
+        },
+        {
+            "style": "Text",
+            "text": "This property is very important and needs to be defined very well. This property defines the structure of the response data structure and needs to be very precise. It's possible values are:",
+            "updated": 1621953383545
+        },
+        {
+            "style": "List",
+            "text": "number: Any type of numbers fall into this category, including dates timestamps in numbers.",
+            "updated": 1621953444809
+        },
+        {
+            "style": "List",
+            "text": "string: Any strings falls into this category, including dates in string format.",
+            "updated": 1621953470163
+        },
+        {
+            "style": "List",
+            "text": "object: Use this value when you find that the field holds a JSON object enclosed in { and }",
+            "updated": 1621953515163
+        },
+        {
+            "style": "List",
+            "text": "array: Use this type when the value of the field is a JSON array enclosed in [ and ]",
+            "updated": 1621953569329
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Nodes/api-response-schema.json

@@ -5,10 +5,14 @@
         "updated": 1621021486718
     },
     "paragraphs": [
+        {
+            "style": "Text",
+            "text": "The API Map assumes the API Server will respond with string formatted in JSON. This means that the response will be a data structure of nestled objects and arrays. Users will need to map the response format into a API Response Schema by adding children to this node and to this node's children. All these descendants are of the same Node Type API Response Field. ",
+            "updated": 1621952655717
+        },
         {
             "style": "Subtitle",
-            "text": "Configuration",
-            "updated": 1621021492973
+            "text": "Configuration"
         },
         {
             "style": "List",

Projects/Superalgos/Schemas/Docs-Nodes/api-version.json

@@ -6,8 +6,18 @@
     },
     "paragraphs": [
         {
-            "style": "Text",
-            "text": "Right click and select the pencil button to enter edit mode."
+            "style": "Note",
+            "text": "We expect every API to have at leat one version.",
+            "updated": 1621951647515
+        },
+        {
+            "style": "Include",
+            "text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Main Workflow",
+            "updated": 1621951484955
+        },
+        {
+            "style": "Title",
+            "text": "Configuration"
         }
     ]
 }

Projects/Superalgos/Schemas/Docs-Topics/data-mining-3.1-data-mining-fetching-data-from-apis.json

@@ -7,9 +7,51 @@
         "updated": 1621906349521
     },
     "paragraphs": [
+        {
+            "style": "Block",
+            "text": "Main Workflow",
+            "updated": 1621951016327
+        },
+        {
+            "style": "Title",
+            "text": "How to fetch data from an API?",
+            "updated": 1621951311144
+        },
         {
             "style": "Text",
-            "text": "Right click and Edit to enter edit mode and change this text. ENTER to write new paragraphs. ESC to exit edit mode. Then you can use docs.save to save your changes and app.contribute to send it to the main repository."
+            "text": "The main workflow to pull this out is quite straight forward and it does not require coding at all. Follow this guide and you will be using data from any source at the Internet from your trading strategies in no time."
+        },
+        {
+            "style": "List",
+            "text": "1. Choose an API: Current version is limited to public APIs only. Secured APIs comming soon. "
+        },
+        {
+            "style": "List",
+            "text": "2. Design an API Map: Read the APIs online documentation and identify the Endpoints that returns the data you would like to fetch. Carefully map those Endpoints defining the parameters the API Server expects at each call, and also how the API Server response is formatted."
+        },
+        {
+            "style": "List",
+            "text": "3. Setup an API Data Fetcher Bot: Create a Data Mine for this API and create there an API Data Fetcher Bot. From that Bot's Record Properties you will need to reference the fields returned by the API Server defined at the API Map, defining in this way which fields from each response you are going to save to disk. You might also need to define query or path parameters here. "
+        },
+        {
+            "style": "List",
+            "text": "4. Setup an Indicator Bot for Dataset Type Conversion: The Raw Data saved to disk needs to be organized for consumption. You will need to setup an Indicator Bot that will do a Dataset Type Conversion from Raw Data to a standard Dataset Type."
+        },
+        {
+            "style": "List",
+            "text": "5. Setup Indicator Bots: Once the data is converted, you can extract from it specific Datasets defining a set of indicators you can build from that data. These Indicators can be just a reorganization of the data, with some adjustments on the names of fields and splitting the data in different indicators, or it can also include some data processing / transformations if you wish (this would require some coding)."
+        },
+        {
+            "style": "List",
+            "text": "6. Setup Plotters: These last indicators can have their plotters if you wish."
+        },
+        {
+            "style": "List",
+            "text": "7. Consume Data from Trading Strategies: These indicators can be consumed from your trading strategies."
+        },
+        {
+            "style": "Success",
+            "text": "There is a Workspace called API Data Fetcher Bot Demo with a simple working example of the previous steps you can take as reference."
         }
     ]
 }