span8
span4
Live Data Feed:
Input:https://environment.data.gov.uk/flood-monitoring/id/floods?min-severity=4
or
http://environment.data.gov.uk/flood-monitoring/id/floodAreas/061WAF06Coln/polygon
完成的工作区:EX3-READADV-FLOODAER-JSON2KML.FMWT
输出:ex3-readadv-live-output.kml
File based scenario:
输入:Flood_ID_APR7.JSON洪水ID-APR7.JSON
完成的工作区:EX3-READADV-FILE-FLOODAREAS-JSON2KML.FMWT
输出:EX3-READADV-FILE-OUTPUT.KML
JSON is a simple structured text format to store data. These data sets in JSON format come in many shapes. GeoJSON also uses the JSON (JavaScript Object Notation) specification. GeoJSON has the ability to specify data in a way that a Javascript web browser can instantly display it. Whether you download a file from a website or subscribe to a web feed of your interest, you may be accessing data via JSON, or consuming GeoJSON.
A GeoJSON object may represent a geometry, a feature or a collection of features. The features include points, polygons, and multi-part collections of these types. The geometry could be stored in form of coordinates or in form of addresses and locations that can be geocoded. While GeoJSON can have geometry, by default FME does not display it. More complex JSON services are needed where multiple queries are involved with accessing the nested data. In this article of advanced writing we need to use queries to extract subsets of the JSON that we are interested in.
本文中的高级阅读JSON示例将涵盖使用数据检查器使用JSON读取器格式读取的数据。数据检查器将用于检查数据。然后,我们将创建一个工作空间,使我们能够阅读洪水警告API的JSON消息,并在有时存在洪水多边形或不发出洪水警告的情况下对多种情况说明。输出将写入KML文件,以在Google Earth中查看。
The structure of the file and how deeply nested the features are, will make the difference on how easy it is to read. This exercise uses data from the UK government about flood warnings, flood alerts, measurements of water levels and flows. The Environment Agency flood-monitoring API provides developers with access to near real time information. We make no disclaimer as to the availability of this data as it changes continuously. Note that this article uses Environment Agency flood and river level data from the real-time data API (Beta) which is licensed under theOpen Government Licence v3.0. The APIs provide a REST style access to the data via simple HTTP GET requests which returns data in JSON format. This data structure example is nested, which means it has objects which contain child objects. An array is a special type of object that contains repeating elements. The “items” element in the JSON response will contain either a description of a single item or an array of items.
Since we have no control over the example data, and there may be times when there are no floods, you may encounter that the list of floods has items [], or zero items. We have included error handling in this case and we are including a file-based scenario that you can use if following the link returns no items.
第一个Web链接https://environment.data.gov.uk/flood-monitoring/id/floods?min-severity=4
包含许多嵌入的Weblinks,其中多边形数据在第二个Web链接中http://environ.data.gov.uk/flood-monitoring/id/id/floodareas/061waf06coln/polygon。从特征数组查询时,可以在表中表示数据结构。通过更改最后一个数字,可以将严重程度从4更改为降低。
Table 1 shows the data structure of flood area link (the first link)
Table 2 shows the continued data structure of flood area link
http://environment.data.gov.uk/flood-monitoring/id/floodAreas/061WAF06Coln/polygon
Table 3 shows the data structure of embedded link of polygons (second link)
The message in JSON format that we are reading is nested, which means arrays are within arrays. So this means we do not want to flatten the file before we have extracted the geometries.
在创建这个示例的那一天,我们看到了英国政府对格洛斯特郡地区的洪水警告信息。
Open the Data Inspector and make sure you have Stamen background maps or equivalent loaded. Select JSON as the reader format and enter the file location or link (https://environment.data.gov.uk/flood-monitoring/id/floods?min-severity=4) of the data. Parameters will need to be specified before clicking ok as the Features Tree can only be accessed at this point, this is a new feature as of the 2017.0 release. Click on parameters, set Schema Scan Mode to JSON Query and then on the ellipsis button [...] behind JSON Query for Feature Objects.
JSON reader format and URL as the dataset
In the dialog box that opens,click on “json” and pay attention to the text in the “Selected Items” section, you may want to copy it for later use.
读者配置树
一旦你设置参数,点击record and you will see that this file is deeply nested. Note that there are several attributes but no geometry in this view. In the feature information window on the right you can see that it contains links to other sites that contain the flood area polygons.
Data Inspector view has no geometry and shows this deeply nested file contains links to polygon data
We will take a closer look at the data to see how the polygon data is stored. As mentioned the items element in the JSON response will contain either a description of a single item or an array of items. It may however also contain items [] or zero, which happens during a season of no flood warnings. We will look at that scenario under error handling at the end.
{ "items" : [ { "@id" : "http://environment.data.gov.uk/flood-monitoring/id/floods/99528" , "description" : "River Coln and its tributaries" , "eaAreaName" : "West Thames" , "eaRegionName" : "South East" , "floodArea" : { "@id" : "http://environment.data.gov.uk/flood-monitoring/id/floodAreas/061WAF06Coln" , "county" : "Gloucestershire" , "notation" : "061WAF06Coln" , "polygon" : "http://environment.data.gov.uk/flood-monitoring/id/floodAreas/061WAF06Coln/polygon" , "riverOrSea" : "River Coln, Dudgrove Brook" }
Example of nested JSON data with polygons
多边形数据存储在一个多边形避署ment link at http://environment.data.gov.uk/flood-monitoring/id/floodAreas/061WAF06Coln/polygon and contains the coordinate pairs that define the polygon. An excerpt from the polygon data is listed below. Note that the value pairs are stored under geometry.
"geometry": { "type": "MultiPolygon", "coordinates": [[[ [-1.758934661685646, 51.68604258983194], [-1.758826005513945, 51.68598898294504], [-1.758905679948403, 51.68588139134682], [-1.760186905019262, 51.6852519605945], [-1.760259206697431, 51.685342301483914], [-1.759446753960602, 51.68571307418105], [-1.759578091859035, 51.685836831189455], [-1.759519936229592, 51.68589262464122], [-1.758934661685646, 51.68604258983194] ]]]}
Example of polygon data stored in a separate link
In this example a JSON feed was read and displayed in the Data Inspector. The Feature Information window showed that there is no geometry in this view, however, shows the external link that contains polygon data. When investigating the external link, we find coordinate pairs that define the flood polygons in JSON format.
For a nested JSON file like this, we will need to set up a workspace in order to access the geometry. The first URL only gives us an id and a URL which we need to resolve to get to the actual polygon data. This section will walk you through automating these steps.
Open the FME Workbench and generate a workspace with a data set that demonstrates a more elaborate workflow of advanced reading of messages in JSON. Add a Creator transformer, an AttributeCreator and a FeatureReader transformer to create the initial request. The input is a message in JSON format, accessed by the URL, and the output is a KML file that will be opened in Google Earth.
Add a Creator transformer to the canvas. This transformer creates a feature, even if the data does not produce a feature result. This takes care of the possibility that no flood polygon data is in the link at the time.
In the Navigator Pane on the left, under Published Parameters, right-click to create a new User Parameter. Enter the following:
Type | Text |
Name | Severity |
Published | Yes |
Optional | Yes |
Prompt | Severity level 1-4 |
属性分配 | Default |
默认值 | 4 |
Adds User Parameters Dialog Box
添加一个属性转换器,这将创建一个新属性,我们将称为“ _request1”,并包含具有严重性级别的稍微修改的URL作为属性值,我们将在属性值下粘贴到第一个字段中。URL是:https://environment.data.gov.uk/flood-monitoring/id/floods?min-severity=$(Severity)
The flood warning has levels 1-4. We will display severity level 3 and 4 in this exercise to demonstrate that data may come in two error conditions or the expected flood polygons depending on the weather. In case you are running this workspace at a time when no flood warnings or alerts are in effect due to the season see the Alternative: File-based workflow at the end of this article on how to use a file-based workflow instead.
The results may have:
AttributeCreator Parameters
In the Feature Reader parameters enter JSON under format and paste the dataset URL https://environment.data.gov.uk/flood-monitoring/id/floods?min-severity=4 into the section under reader dataset.
点击读者参数,这将打开再保险ader Parameter Dialog Box. Here we have the option to select from the tree once we specify JSON Query as Schema Scan Mode and none as Geometry Format. Select the ellipsis [...] to open the tree. Enter “json[“items”][*]” as JSON Query by clicking on
功能阅读器参数
读者配置树
After clicking ok 3 times, your features are now read into the workspace and you can save your file.
The AttributeExposer transformer exposes hidden attributes so that they can be used by other transformers. Connect this transformer to the FeatureReader
添加StringLengthCalculator并将其连接到上一个变压器。将“ Floodarea.polygon”分配为参数中的源属性,并接受“ _length”为StringLength属性默认值。
We will demonstrate the option to read this JSON message in case the resulting feature contains no geometry. When adding this tester transformer, the output of the tester will be read in by two transformers. If there is geometry it passes the features to the HTTPCaller. If the Tester fails and contains no geometry we want to pass on a message via an AttributeCreator to the output display. This is one error handling scenario.
Add a Tester transformer and connect it to the StringLengthCalculator. In the Left Value type or chose from the pull down menu select “_length”. Chose “>” as operator and type “0” as Right Value. Click ok.
测试仪参数
Connect an AttributeCreator to the Failed Output port on the Tester. Then connect its Output port to the Writer. In the parameters enter the following:
New Attribute | Attribute Value |
message | no flood feature polygons for severity level $(Severity) |
kml_id | no flood features |
kml_description | no flood features for severity level $(Severity) |
AttributeCreator handles messages
Connect this second AttributeCreator to the writer as it passes the tester’s failed port results to the writer.
Add an HTTPCaller transformer and connect it to the Passed Output port on the Tester transformer. Specify under Request URL “floodArea.polygon”, the HTTP Method should be set to GET and ensure the Response Body Attribute isset to “_response_body”. Accept the defaults for everything else. Click ok. The structure of this Request URL comes from the file.
"floodArea" : { "polygon" : "http://environment.data.gov.uk/flood-monitoring/id/floodAreas/061WAF06Coln/polygon" }
多边形数据结构的摘录
接下来,我们将添加jsonfragmenter。由于该文件的结构是嵌套的,因此我们将需要另一个查询才能深入到文件的结构中,以访问几何形状。坐标数据在功能和几何形状范围内。输入以下内容:
JSON属性 | _response_body |
JSON查询 | json[“features”][*][“geometry”] |
将查询结果变平属性 | Yes |
递归平坦的对象/数组 | No |
JSONFragmenter parameters
The reason for this query structure is the nesting.
{ "type": "FeatureCollection", "features": [{"type": "Feature"}, "geometry": {"type": "MultiPolygon", "coordinates": [ [ [ [-1.758934661685646, 51.68604258983194]]]}}
Condensed data structure
接下来,将添加另一个属性XPOSER变压器。亚搏在线在参数中添加“几何形状”和“洪水泛滥”。单击“确定”一旦输入空白字段。
将编码的几何形状设置为Geojson,然后单击几何源旁边的省略号[...]符号以查看文本编辑器。通过双击文本编辑器中的“ _Response_body”,将源参数设置为@Value(_Response_body),您将在右侧的字段中看到“访问”参数。最后,将删除属性选项设置为:“否”。
几何定位参数
Add a CoodinateSystemSetter to the Output port on the GeometryReplacer and set the Coordinate System to ESPG:4326.
Next we will duplicate the previous AttributeCreator that was added. Right-click on the AttributeCreator that is attached to the Failed Output port on the Tester, and click Duplicate. Connect the duplicated AttributeCreator to the Output port on the CoordinateSystemSetter.
AttributeCreator Parameters
将输出连接到kmlstyler,然后将输出连接到Floodareas Writer。将KML Styler的颜色参数设置为您选择的颜色。例如颜色绿色,填充颜色蓝色,不透明度0.3,填充不透明度0.4。将工作空间重定向到数据检查器。作者格式将是KML,并且将设置数据集路径。
完成的工作区
The completed workspace will look like this when running at a time when flood warnings of severity 3 is selected and there are flood warnings in effect. When running the workspace you will be prompted which severity level you would like to chose. If you only get 1 result in the FeatureReader initiator port, this means that no flood warnings are in effect. We will handle this in the error handling section at the end.
When Redirect to the Data Inspector is turned on your result will look similar to this, depending on how many areas have flood warnings and alerts posted. The Data Inspector will display the resulting feature records of the flood polygons. This example showed how to read a complex message in JSON format using a URL and writing to a KML file.
洪水多边形用数据检查员查看
When Redirect to the Data Inspector is turned off, your result will be a KML file. Click on JSON Feature Writer to open containing folder and double click the KML file that will open in Google Earth. You will see the flood polygons highlighted in the colour you chose in the KML styler or a message alerting you that no flood warnings are issued at the current time.
Google Earth中的KML文件显示您选择的颜色突出显示的洪水多边形
Flood warnings occur only during certain seasons and cause flood polygons not necessarily present at all times. If you are testing this workflow in a season of low flood expectancy, you will receive a message as an output. When flood alerts or warnings exist, the flood polygons associated with flood features will be handled with the following transformer sequence.
当Google Earth中的输出KML文件在此严重性水平找不到功能时,将看起来像这样。但是,可能有其他严重程度列出的多边形。
The message that will be displayed in this case in Google Earth when no features are found
使用原始的工作区中创建这个导师ial, add a Tester transformer to the
The one AttributeCreator will be connected to the passed tester port to handle no result features. This transformer connects the failed port to the writer and passes on the message that there are no features. This case is in effect when the array is empty.
The other AttributeCreator will handle the results from the failed tester port and will send output messages in case the link has no flood polygons and no features are found. Set the parameters as follows:
Duplicate the AttributeCreator_2 and adjust the message. Then connect it to the Passed Output port on the Tester. This AttributeCreator will handle the results from the passed tester port and will send output messages in case the array is empty.
New Attribute | Attribute Value |
message | no features for severity level $(Severity) |
kml_id | no features found |
kml_description | no features for severity level $(Severity) |
Duplicate this new AttributeCreator and attach it to the failed port of the Tester. Then connect it to the writer. This case is in effect when a severity level is chosen that does not have flood warnings for example level 1. Connect both these newly created AttributeCreators to the Writer. The resulting messages can be seen in the Data Inspector or in Google Earth.
Connect a Logger to the
This exercise demonstrated how to read a message in JSON format that is complex and nested. It can therefore be considered an advanced reading example and showed how to access the hidden geometry in the embedded link of the UK environment agency’s Flood warnings API.
在洪水警告API不包含洪水多边形的时候,上述工作流程仅表明没有几何形状,您就可以使用我们的基于文件的数据集Flood_ID_APR7.JSON。亚搏在线使用此完成的工作空间:EX3-READADV-FILE-FLOODAEREAS-JSON2KML.FMWT和输入:Flood_ID_APR7.JSON。输出为:ex3-readadv-file-output.kml。
JSON入门
Converting from JSON to a spatial format (GIS)
JSON Writing with JSONTemplater
Note that this article uses Environment Agency flood and river level data from the real-time data API (Beta) which is licensed under the Open Government Licence v3.0.
© 2019 Safe Software Inc |Legal