Merge branch 'develop' into develop

This commit is contained in:
Julian Molina authored and GitHub committed 2021-06-05 12:21:23 +02:00
commit 1d2dc5a646
12 files changed
+378 -151

No files matched your search

@@ -1,18 +1,23 @@
{
"type": "API Endpoint",
"definition": {
"text": "Endpoints are important aspects of interacting with server-side web APIs, as they specify where resources lie that can be accessed by third party software. Usually the access is via a URI to which HTTP requests are posted, and from which the response is thus expected.",
"updated": 1621017376741
"text": "Endpoints are important aspects of interacting with server-side web APIs, as they specify where resources can be accessed. Usually the access is via a URL to which HTTP requests are posted. Once these requests are posted correctly, the API will respond with the requested data.",
"updated": 1622854645690
},
"paragraphs": [
{
"style": "Subtitle",
"text": "How to use this node?",
"updated": 1621949863470
"text": "How to Use an Endpoint Node",
"updated": 1622854703461
},
{
"style": "Text",
"text": "Read the docs of the API you are trying to map, and indentigy the commands that it support (endpoints)."
"text": "Read the docs of the API you are trying to map, and indentify which endpoints they support. From there you can choose which endpoint holds the information you are interested in and begin to map it. For a detailed description of how to do this, please read the following description. In addtion please see the API Map page as well as the Data Mining topic for other important concepts surrounding setting up an api map. ",
"updated": 1622855071509
},
{
"style": "Note",
"text": "The first page of the Data Mining topic is Data Mining - Introduction."
},
{
"style": "Include",
@@ -21,7 +26,12 @@
},
{
"style": "Title",
"text": "Configuration"
"text": "Configuration Properties",
"updated": 1622863183444
},
{
"style": "List",
"text": "codeName: Define the endpoint this branch of the map will handle."
}
]
}
@@ -7,22 +7,152 @@
"paragraphs": [
{
"style": "Text",
"text": "The way to create an API Map is to read the API's documentation and for each command and their responses, generate the node structure that represents it.",
"updated": 1621016234649
},
{
"style": "Title",
"text": "How is it used?"
"text": "The way to create an API Map is to read the API's documentation for each endpoint as well as the Data Mining Topic. These sources will provide you with all the information you need to make a successful API Map.",
"updated": 1622847308932
},
{
"style": "Text",
"text": "To use an API Map users need to setup a Task for an API Data Fetcher Bot. This set up is done automatically as part of the Install Market procedure if an API Map is detected at the Worksapce. From within that Task, a reference to an API Map is established.",
"updated": 1622626543279
"text": "The basic way you can think about an API Map is as a representation of the URL link needed to request information from the API as well as the layout of the data returned by the API. This means that an API Map hold all the information needed to request data and decipher the data once it is returned.",
"updated": 1622847337343
},
{
"style": "Title",
"text": "How are API Maps used?",
"updated": 1622847013897
},
{
"style": "Text",
"text": "API Maps are used by API Data Fetcher Bots to request and save the data from an API. In order to do this a user needs to set up a task to run the bot. This set up is done automatically as part of the Install Market procedure if both an API Map and its data mine is detected within the Worksapce. From within that Task, a reference to the API Map is established.",
"updated": 1622847596702
},
{
"style": "Text",
"text": "When the Task is run, the UI will send to the Task Server all the definitions inside the API Map so that the API Data Fetcher Bot can use them to connect to the API Server, fetch data from it, and correctly interpret the responses."
},
{
"style": "Title",
"text": "How to Read an API Request URL",
"updated": 1622847733042
},
{
"style": "Text",
"text": "The API you are trying to map should have it's API Endpoints documented. If not, check any examples they may provide for how to request the data and see if there are inner folders below the main folder of the API Enpoint. Each one of these folders are an API Path Parameter that needs to be defined.",
"updated": 1622847657548
},
{
"style": "Text",
"text": "Here are some examples of API request URLs:"
},
{
"style": "Javascript",
"text": " https://api.coingecko.com/api/v3/coins/bitcoin?"
},
{
"style": "Javascript",
"text": "https://data.messari.io/api/v1/assets/btc?fields=id,slug,symbol,metrics/market_data",
"updated": 1622848130436
},
{
"style": "Subtitle",
"text": "Sections of an API Request URL"
},
{
"style": "Text",
"text": "Let's break down one of these URLs and identify each part of it."
},
{
"style": "List",
"text": "Internet Protocol: This is the section on the very front of almost any internet link you will see."
},
{
"style": "Javascript",
"text": " https"
},
{
"style": "Note",
"text": "You define this using the protocol config property in the main API Map node.",
"updated": 1622847750228
},
{
"style": "List",
"text": "Root Domain Name: This is the section of the URL that is before the first backslash:"
},
{
"style": "Javascript",
"text": " data.messari.io"
},
{
"style": "Note",
"text": "You define this using the hostName config property in the main API Map node.",
"updated": 1622847770913
},
{
"style": "List",
"text": "API Path: This is the section that defines the main path to the API:"
},
{
"style": "Javascript",
"text": " api"
},
{
"style": "Note",
"text": "You define this using the path config property in the main API Map node.",
"updated": 1622847800732
},
{
"style": "List",
"text": "API Version Number: This is the section that defines the API version to which you are making requests."
},
{
"style": "Javascript",
"text": " v1"
},
{
"style": "Note",
"text": "You define this using the codeName config property in the API Version node.",
"updated": 1622847982153
},
{
"style": "List",
"text": "API Endpoint: This is the section that tells the API what kind of data you wish to request.",
"updated": 1622847877226
},
{
"style": "Javascript",
"text": "assets"
},
{
"style": "Note",
"text": "You define this using the config codeName property of the API Endpoint node.",
"updated": 1622847959761
},
{
"style": "List",
"text": "Path Parameters: These are the remaning sections of the path before the question mark they further narrow down what data you wish to request from the API. In this case we want to receive data about bitcoin:",
"updated": 1622848189591
},
{
"style": "Javascript",
"text": " btc"
},
{
"style": "Note",
"text": "You define each of these using the codeName config property in the API Path Parameter node.",
"updated": 1622848212517
},
{
"style": "List",
"text": "Query Parameters: These are all the sections after the question mark. They are often used to pair down the amount of data returned by the API so that you only receive what you are interested in:",
"updated": 1622848284105
},
{
"style": "Javascript",
"text": "fields=id,slug,symbol,metrics/market_data"
},
{
"style": "Note",
"text": "You define these using the codeName and value config properties of the API Query Parameter node."
},
{
"style": "Include",
"text": "Superalgos->Topic->Data Mining - Fetching Data From APIs->Main Workflow",
@@ -35,7 +165,8 @@
},
{
"style": "Title",
"text": "Configuration"
"text": "Configuration Properties",
"updated": 1622863551147
},
{
"style": "List",
@@ -61,7 +192,17 @@
},
{
"style": "List",
"text": "millisecondsBetweenCalls: [Optional] the number of milliseconds to wait before calling the remote host. This is particuary usefull if there is pagination, so that there is a delay before requesting each page."
"text": "millisecondsBetweenCalls: [Optional] the number of milliseconds to wait before calling the remote host. This is particuary useful if there is pagination, so that there is a delay before requesting each page.",
"updated": 1622863477228
},
{
"style": "Text",
"text": "Here is an example API Map config:"
},
{
"style": "Json",
"text": "{\n \"codeName\": \"CoinGecko\",\n \"protocol\": \"https\",\n \"portNumber\": \"\",\n \"hostName\": \"api.coingecko.com\",\n \"path\": \"api\",\n \"millisecondsBetweenCalls\": 4000\n}",
"updated": 1622863537968
}
]
}
@@ -27,8 +27,20 @@
},
{
"style": "List",
"text": "replaceBy: This property allows you to instruct the API Fetcher Bot to set a value to this parameter replacing the current default and using contextual information available only at runtime.",
"updated": 1622773172219
"text": "isString: Define if the path parameter is a string or not."
},
{
"style": "List",
"text": "isOptional: Define if this path parameter is optional or not."
},
{
"style": "List",
"text": "description: Define a basic description of what this path parameter accomplishes. "
},
{
"style": "List",
"text": "replaceBy: [Optional] This property allows you to instruct the API Fetcher Bot to set a value to this parameter replacing the current default and using contextual information available only at runtime.",
"updated": 1622863592804
},
{
"style": "Text",
@@ -20,99 +20,6 @@
"text": "As with everything else in Superalgos, node children are considered clockwise. This is important because it will affect the order in which Superalgos builds the URL request you will send to the API.",
"updated": 1622773291926
},
{
"style": "Text",
"text": "Here are some examples of API request URLs:"
},
{
"style": "Javascript",
"text": "https://api.coingecko.com/api/v3/coins/bitcoin?",
"updated": 1622773573541
},
{
"style": "Javascript",
"text": "https://data.messari.io/api/v1/assets/btc?",
"updated": 1622773581349
},
{
"style": "Subtitle",
"text": "Sections of an API Request URL",
"updated": 1622773855472
},
{
"style": "Text",
"text": "Let's break down one of these URLs and identify each part of it."
},
{
"style": "List",
"text": "Internet Protocol: This is the section on the very front of almost any internet link you will see.",
"updated": 1622774168610
},
{
"style": "Javascript",
"text": "https",
"updated": 1622774200879
},
{
"style": "Note",
"text": "You define this using the protocol config property in the main API Map node."
},
{
"style": "List",
"text": "Root Domain Name: This is the section of the URL that is before the first backslash:"
},
{
"style": "Javascript",
"text": "data.messari.io"
},
{
"style": "Note",
"text": "You define this using the hostName config property in the main API Map node. ",
"updated": 1622774031960
},
{
"style": "List",
"text": "API Path: This is the section that defines the main path to the API:"
},
{
"style": "Javascript",
"text": "api"
},
{
"style": "Note",
"text": "You define this using the path config property in the main API Map node.",
"updated": 1622773958109
},
{
"style": "List",
"text": "API Version Number: This is the section that defines the API version to which you are making requests."
},
{
"style": "Javascript",
"text": "v1"
},
{
"style": "Note",
"text": "You define this using the codename config property in the API Versioin node."
},
{
"style": "List",
"text": "Path Parameters: These are the remaning sections of the path before the question mark:"
},
{
"style": "Javascript",
"text": "assets ",
"updated": 1622774373224
},
{
"style": "Javascript",
"text": "btc"
},
{
"style": "Note",
"text": "You define each of these using the codeName config property in the API Path Parameter node.",
"updated": 1622774459925
},
{
"style": "Text",
"text": ""
@@ -11,13 +11,26 @@
"updated": 1622765045028
},
{
"style": "Subtitle",
"text": "Configuration"
"style": "Title",
"text": "Configuration Properties",
"updated": 1622863199996
},
{
"style": "List",
"text": "codeName: This must match with the name of the query parameter defined at the API docs."
},
{
"style": "List",
"text": "isString: Define if the path parameter is a string or not."
},
{
"style": "List",
"text": "isOptional: Define if this path parameter is optional or not."
},
{
"style": "List",
"text": "description: Define a basic description of what this path parameter accomplishes."
},
{
"style": "List",
"text": "value: This is the value to be used by the parameter. ",
@@ -30,8 +43,8 @@
},
{
"style": "List",
"text": "replaceBy: This property allows you to instruct the API Fetcher Bot to set a value to this parameter replacing the current default and using contextual information available only at runtime.",
"updated": 1622773230818
"text": "replaceBy: [Optional] This property allows you to instruct the API Fetcher Bot to set a value to this parameter replacing the current default and using contextual information available only at runtime.",
"updated": 1622863577107
},
{
"style": "Text",
@@ -1,23 +1,38 @@
{
"type": "API Query Response",
"definition": {
"text": "This node enables the definition of a particular response of the API call. ",
"updated": 1621021001505
"text": "This node enables the definition of a particular response from the API call.",
"updated": 1622862747985
},
"paragraphs": [
{
"style": "Subtitle",
"text": "Configuration",
"updated": 1621021155356
"style": "Title",
"text": "Configuration Properties",
"updated": 1622863221142
},
{
"style": "List",
"text": "codeName: At the codeName property, the API Fetcher Bot will expect to find the http response code for the API call. You will need to have at least one of these nodes with a codeName = \"200\" that is the standard code for success."
"text": "codeName: At the codeName property, the API Fetcher Bot will expect to find the http response code from the API call handled by this node. You will need to have at least one of these nodes with a codeName = \"200\" (that is the standard code for success).",
"updated": 1622862811491
},
{
"style": "List",
"text": "isError: Define if the status code should be considered an error or not.",
"updated": 1622862913882
},
{
"style": "Success",
"text": "If you wish to be able to react to other error codes, you should also have more of these nodes with the codeName of each of the status you want to cover. If the API Data Fetcher Bot receives a response code that has not been defined, it will stop there the processing.",
"updated": 1621952346052
"text": "If you wish to be able to react to other error codes, you should also have more of these nodes with the codeName of each status you want to cover. If the API Data Fetcher Bot receives a response code that has not been defined, it will stop processing and error out.",
"updated": 1622862870506
},
{
"style": "Text",
"text": "Here is an example API Query Response config:"
},
{
"style": "Json",
"text": "{\n \"codeName\": \"200\",\n \"isError\": false\n}",
"updated": 1622863659433
}
]
}
@@ -1,8 +1,8 @@
{
"type": "API Query Responses",
"definition": {
"text": "API calls produce responses. Under this section they are defined.",
"updated": 1622295146024
"text": "API calls produce responses. Under this node you can define how your map will interpret different responces from the API.",
"updated": 1622862463288
},
"paragraphs": [
{
@@ -10,9 +10,15 @@
"text": "For a single API Endpoint an API Server might have different possible responses. Each response type is differentiated from each other by the response code / status.",
"updated": 1621952454860
},
{
"style": "Success",
"text": "A few common examples are: 200 meaning successful response, 404 meaning page not found and 429 meaning api rate limit exceeded.",
"updated": 1622862627312
},
{
"style": "Text",
"text": "Each child of this node is a different response code / status from the API Server. Users need to add at least the most common response code (200) in order to start testing this API Endpoint. After that is recommended to add all other response codes as well, otherwise the API Data Fetcher Bot will stop when an unhandled response code is received."
"text": "Each child of this node will handle a different response code / status from the API Server. Users need to add at least the most common response code (200) in order to start testing this API Endpoint. After that is recommended to add all other response codes as well, otherwise the API Data Fetcher Bot will stop when an unhandled response code is received.",
"updated": 1622862684442
}
]
}
@@ -1,24 +1,28 @@
{
"type": "API Response Field",
"definition": {
"text": "The API responds with a JSON object. Each property of the object is mapped as node of this type. This node can have children of the same type allowing users to map a complex data structure.",
"updated": 1621021740145
"text": "The API responds with a JSON object. Each property of the object is mapped as a node of this type. This node can have children of the same type allowing users to map a complex data structure.",
"updated": 1622864246927
},
"paragraphs": [
{
"style": "Subtitle",
"text": "Configuration",
"updated": 1621021763335
"style": "Title",
"text": "Configuration Properties",
"updated": 1622864255383
},
{
"style": "List",
"text": "codeName: The codeName needs to be the exact name of the property at the response JSON object holding the value. For the first object, since it is a special case, we usually use the codeName of rootObject.",
"updated": 1621021843683
"text": "codeName: The codeName needs to be the exact name of the property held in the response JSON object. ",
"updated": 1622864298993
},
{
"style": "Note",
"text": "For the first object, since it is a special case, we usually use the codeName of rootObject."
},
{
"style": "List",
"text": "fieldType: This property declares if the data is number, a string, an object, or an array, those are the 4 possible values.",
"updated": 1622315306276
"text": "fieldType: This property declares if the data is a number, string, object, or array.",
"updated": 1622864334731
},
{
"style": "List",
@@ -26,7 +30,8 @@
},
{
"style": "Subtitle",
"text": "Field Type Possible Values"
"text": "Possible Values for Field Type Property",
"updated": 1622864403887
},
{
"style": "Text",
@@ -1,8 +1,8 @@
{
"type": "API Response Schema",
"definition": {
"text": "This node represents the mapping of the JSON response of the API into a data structure built by API Response Field nodes.",
"updated": 1621021486718
"text": "This node represents the map for the API response. APIs send ther data using a JSON object. The children of this node will map this JSON object so that the API Fetcher Bot knows how to extract the data you want.",
"updated": 1622863829380
},
"paragraphs": [
{
@@ -10,29 +10,40 @@
"text": "The API Map assumes the API Server will respond with string formatted in JSON. This means that the response will be a data structure of nestled objects and arrays. Users will need to map the response format into an API Response Schema by adding children to this node and to this node's children. All these descendants are of the same Node Type API Response Field.",
"updated": 1622312357188
},
{
"style": "Success",
"text": "When reading documentation, or looking at raw data held in JSON, it is important to remember that curly brackets { } define objects while square brackets [ ] define arrays.",
"updated": 1622863958740
},
{
"style": "Title",
"text": "Configuration"
"text": "Configuration Properties",
"updated": 1622863966010
},
{
"style": "List",
"text": "nodePath: the response from the server might include some header type of data and the actual data requested. Since the response is a JSON object, then the actual data might be located somewhere within the structure of that object. This property allows you to declare the path to the actual data requested, so as to jump the header-like information."
"text": "nodePath: the response from the server might include some header type data in addition to the actual data requested. Since the response is a JSON object, then the actual data might be located somewhere within the structure of that object. This property allows you to declare the path to the actual data requested, so as to jump the header-like information.",
"updated": 1622864019058
},
{
"style": "Subtitle",
"text": "Why do we need to jump the header?"
"text": "Why do We Need to Jump the Header?",
"updated": 1622864036535
},
{
"style": "Text",
"text": "In many cases it will be necesary to make several queries to the API Server in order to fetch all the data (think of pagination). Each page of data arrives at a certain location within the reponse data structure. That location is the one considered the Data Root. "
"text": "In many cases it will be necessary to make several queries to the API Server in order to fetch all the data (think of pagination). Each page of data arrives at a certain location within the reponse data structure. That location is what we considered the Data Root.",
"updated": 1622864086051
},
{
"style": "Text",
"text": "If you Endpoint does not support pagination, it is still a good practice to define which is the Data Root."
"text": "If your Endpoint does not support pagination, it is still a good practice to define which part of the JSON object is the Data Root.",
"updated": 1622864107476
},
{
"style": "Important",
"text": "Most Record Properties will be referencing API Response Field nodes that are under the Data Root you defined, but there might be one or two that are not there, but under the response data structure in general, or at a section that could be considered a header. In that case, there is some special step you need to follow. At the API Response Field Reference node cofing, you will need to set up a nodePath property as explained below."
"text": "Most Record Properties will be referencing API Response Field nodes that are under the Data Root you defined, but there might be one or two that are not there. Rather these values may be under the response data structure in general, or at a section that could be considered a header. In that case, there is some special steps you need to follow. At the API Response Field Reference node config, you will need to set up a nodePath property as explained below.",
"updated": 1622864199297
},
{
"style": "Text",
@@ -1,14 +1,14 @@
{
"type": "API Version",
"definition": {
"text": "An API might have different versions. This node is used for specifying the version that it's children are defining. At the codeName property we expect the version string as needed on the URL when calling the API Server.",
"updated": 1621016746266
"text": "An API might have different versions. This node is used for specifying the version that it's children are defining. At the codeName property we expect the version string that will be needed in the URL when calling the API Server.",
"updated": 1622848609952
},
"paragraphs": [
{
"style": "Note",
"text": "We expect every API to have at least one version.",
"updated": 1622291417472
"text": "We expect every API to have at least one version. Others may have three or more.",
"updated": 1622848637394
},
{
"style": "Include",
@@ -17,7 +17,12 @@
},
{
"style": "Title",
"text": "Configuration"
"text": "Configuration Properties",
"updated": 1622863171533
},
{
"style": "List",
"text": "codeName: Define the version of the API this branch of the map will handle."
}
]
}
@@ -0,0 +1,102 @@
{
"topic": "CoinGecko API Map and Mine",
"pageNumber": 2,
"type": "CoinGecko API Map",
"definition": {
"text": "The current CoinGecko API Map accesses the following endpoints of the CoinGecko API.",
"updated": 1622686989766
},
"paragraphs": [
{
"style": "Text",
"text": "Documentation for the CoinGecko API can be found at the following link:",
"updated": 1622687107891
},
{
"style": "Link",
"text": "www.coingecko.com/en/api->www.coingecko.com/en/api",
"updated": 1622687459394
},
{
"style": "Text",
"text": "The base request URL is:"
},
{
"style": "Javascript",
"text": "https://api.coingecko.com/api/v3/"
},
{
"style": "Title",
"text": "Mapped Endpoints",
"updated": 1622687143514
},
{
"style": "Json",
"text": "/coins/{id}",
"updated": 1622688221280
},
{
"style": "Subtitle",
"text": "Raw Returned Values",
"updated": 1622688289637
},
{
"style": "List",
"text": "id: name of returned coin (string).",
"updated": 1622688305150
},
{
"style": "List",
"text": "market_cap_rank: Current ranking of coin based on its overall share of the crypto market cap (number)."
},
{
"style": "List",
"text": "sentiment_votes_up_percentage: Percentage of up votes for coin through CoinGecko platforms (number).",
"updated": 1622688575235
},
{
"style": "List",
"text": "sentiment_votes_down_percentage: Percentage of down votes for coin through CoinGecko platforms (number).",
"updated": 1622688586166
},
{
"style": "List",
"text": "block_time_in_minutes: Current time in minutes taken to mine the last block of coin (number).",
"updated": 1622688666930
},
{
"style": "List",
"text": "coingecko_score: Overall trust score given to coin by the CoinGecko platform (number)."
},
{
"style": "Note",
"text": "CoinGecko score is an aggregated score combining liquidity, community, and developer scores.",
"updated": 1622846885139
},
{
"style": "List",
"text": "liquidity_score: Liquidity health score given by the CoinGecko platform (number).",
"updated": 1622688975074
},
{
"style": "List",
"text": "community_score: Community strength score given by the CoinGecko platform (number).",
"updated": 1622689004490
},
{
"style": "List",
"text": "developer_score: Developer strength score given by the CoinGecko platform (number).",
"updated": 1622689029728
},
{
"style": "List",
"text": "public_interest_score: Level of public interest score given by the CoinGecko platform (number).",
"updated": 1622689104410
},
{
"style": "List",
"text": "last_updated: Date content of current responce was last updated (string).",
"updated": 1622689075868
}
]
}
@@ -43,8 +43,8 @@
},
{
"style": "Note",
"text": "CoinGecko score is an aggregated score combining liquidity, community, developer, and public interest scores.",
"updated": 1622689922164
"text": "CoinGecko score is an aggregated score combining liquidity, community, and developer scores.",
"updated": 1622846902289
},
{
"style": "List",