SlideRule Core Module
1 Service Architecture
A typical SlideRule deployment includes three components:
Access to the NASA Common Metadata Repository (CMR) system
A service discovery server
A set of processing nodes
When a client makes a processing request to SlideRule, it is typical that all three components are involved as follows:
The client first makes a spatial and temporal query to the CMR system to retrieve a set of resources (i.e. H5 files) that correspond to the region and time period of interest.
The client then makes a request to the service discovery server to retrieve a list of IP addresses of the available SlideRule processing nodes.
Finally, the client creates a thread pool of workers and fans out the processing of each resource over the available SlideRule processing nodes.
Of course, these steps do not have to be taken in this order, nor is it a problem to just use one component of the system without the others. But these components are designed to compliment each other and provide all of the necessary services needed to perform large processing requests.
2 De-serialization
There are two types of SlideRule services distinguished by the type of response they provide: (1) normal services, (2) stream services.
- Normal
Normal services are accessed via the GET HTTP method and return a discrete block of ASCII text, typically formatted as JSON.
These services can easily be accessed via curl or other HTTP-based tools, and contain self-describing data. When using the SlideRule Python client, they are accessed via the
sliderule.source(..., stream=False)
call.- Stream
Stream services are accessed via the POST HTTP method and return a serialized stream of binary records containing the results of the processing request.
These services are more difficult to work with third-party tools since the returned data must be parsed and the data itself is not self-describing. When using the SlideRule Python client, they are accessed via the
sliderule.source(..., stream=True)
call. Inside this call, the SlideRule Python client will take care of any additional service calls needed in order to parse the results and return a self-describing Python data structure (i.e. the elements of the data structure are named in such a way as to indicate the type and content of the returned data).If you want to process streamed results outside of the SlideRule Python client, then a brief description of the format of the data follows. For additional guidance, the hidden __parse function inside the sliderule.py source code provides the code which performs the stream processing for the SlideRule Python client.
Each response record is formatted as: <record length><record type><record data> where,
- record length
32-bit little endian integer providing the total length of the record that follows (type and data)
- record type
null-terminated ASCII string containing the name of the record type
- record data
binary contents of data
In order to know how to process the contents of the record data, the user must perform an additional query to the SlideRule
definition
service, providing the record type. The definition service returns a JSON response object that provides a format definition of the record type that can be used by the client to decode the binary record data. The format of the definition response object is:{ "__datasize": # minimum size of record "field_1": { "type": # data type (see sliderule.basictypes for full definition), or record type if a nested structure "elements": # number of elements, 1 if not an array "offset": # starting bit offset into record data "flags": # processing flags - LE: little endian, BE: big endian, PTR: pointer }, ... "field_n": { ... } }
3 Endpoints
definition
GET /source/definition <request payload>
Gets the record definition of a record type; used to parse binary record data
Request Payload (application/json)
parameter
description
default
record-type
the name of the record type
required
HTTP Example
GET /source/definition HTTP/1.1 Host: my-sliderule-server:9081 Content-Length: 23 {"rectype": "atl03rec"}Python Example
# Request Record Definition rsps = sliderule.source("definition", {"rectype": "atl03rec"}, stream=False)
Response Payload (application/json)
JSON object defining the on-the-wire binary format of the record data contained in the specified record type.
See De-serialization for a description of how to use the record definitions.
event
POST /source/event <request payload>
Return event messages (logs, traces, and metrics) in real-time that have occurred during the time the request is active
Request Payload (application/json)
parameter
description
default
type
type of event message to monitor: “LOG”, “TRACE”, “METRIC”
“LOG”
level
minimum event level to monitor: “DEBUG”, “INFO”, “WARNING”, “ERROR”, “CRITICAL”
“INFO”
format
the format of the event message: “FMT_TEXT”, “FMT_JSON”; empty for binary record representation
optional
duration
seconds to hold connection open
0
HTTP Example
POST /source/event HTTP/1.1 Host: my-sliderule-server:9081 Content-Length: 48 {"type": "LOG", "level": "INFO", "duration": 30}Python Example
# Build Logging Request rqst = { "type": "LOG", "level" : "INFO", "duration": 30 } # Retrieve logs rsps = sliderule.source("event", rqst, stream=True)
Response Payload (application/octet-stream)
Serialized stream of event records of the type
eventrec
. See De-serialization for a description of how to process binary response records.
geo
GET /source/geo <request payload>
Perform geospatial operations on spherical and polar coordinates
Request Payload (application/json)
parameter
description
default
asset
data source (see Assets)
required
pole
polar orientation of indexing operations: “north”, “south”
“north”
lat
spherical latitude coordinate to project onto a polar coordinate system, -90.0 to 90.0
optional
lon
spherical longitude coordinate to project onto a polar coordinate system, -180.0 to 180.0
optional
x
polar x coordinate to project onto a spherical coordinate system
optional
y
polar y coordinate to project onto a spherical coordinate system
optional
span
a box defined by a lower left latitude/longitude pair, and an upper right lattitude/longitude pair
optional
span1
a span used for intersection with the span2
optional
span2
a span used for intersection with the span1
optional
span definition parameter
description
default
lat0
smallest latitude (starting at -90.0)
required
lon0
smallest longitude (starting at -180.0)
required
lat1
largest latitude (ending at 90.0)
required
lon1
largest longitude (ending at 180.0)
required
HTTP Example
GET /source/geo HTTP/1.1 Host: my-sliderule-server:9081 Content-Length: 115 {"asset": "atlas-local", "pole": "north", "lat": 30.0, "lon": 100.0, "x": -0.20051164424058, "y": -1.1371580426033}Python Example
rqst = { "asset": "atlas-local", "pole": "north", "lat": 30.0, "lon": 100.0, "x": -0.20051164424058, "y": -1.1371580426033, } rsps = sliderule.source("geo", rqst)
Response Payload (application/json)
JSON object with elements populated by the inferred operations being requested
parameter
description
default
intersect
true if span1 and span2 intersect, false otherwise
optional
combine
the combined span of span1 and span 2
optional
split
the split of span
optional
lat
spherical latitude coordinate projected from the polar coordinate system, -90.0 to 90.0
optional
lon
spherical longitude coordinate projected from the polar coordinate system, -180.0 to 180.0
optional
x
polar x coordinate projected from the spherical coordinate system
optional
y
polar y coordinate projected from the spherical coordinate system
optional
HTTP Example
HTTP/1.1 200 OK Server: sliderule/0.5.0 Content-Type: text/plain Content-Length: 76 {"y":1.1371580426033,"x":-0.20051164424058,"lat":29.999999999998,"lon":-100}
h5
POST /source/h5 <request payload>
Reads a dataset from an HDF5 file and return the values of the dataset in a list.
See icesat2.h5 function for a convient method for accessing HDF5 datasets.
Request Payload (application/json)
parameter
description
default
asset
data source asset (see Assets)
required
resource
HDF5 filename
required
dataset
full path to dataset variable
required
datatype
the type of data the returned dataset values should be in
“DYNAMIC”
col
the column to read from the dataset for a multi-dimensional dataset
0
startrow
the first row to start reading from in a multi-dimensional dataset
0
numrows
the number of rows to read when reading from a multi-dimensional dataset
-1 (all rows)
id
value to echo back in the records being returned
0
HTTP Example
POST /source/h5 HTTP/1.1 Host: my-sliderule-server:9081 Content-Length: 189 {"asset": "atlas-local", "resource": "ATL03_20181019065445_03150111_003_01.h5", "dataset": "/gt1r/geolocation/segment_ph_cnt", "datatype": 2, "col": 0, "startrow": 0, "numrows": 5, "id": 0}Python Example
>>> import sliderule >>> sliderule.set_url("icesat2sliderule.org") >>> asset = "nsidc-s3" >>> resource = "ATL03_20181019065445_03150111_003_01.h5" >>> dataset = "/gt1r/geolocation/segment_ph_cnt" >>> rqst = { "asset" : asset, "resource": resource, "dataset": dataset, "datatype": sliderule.datatypes["INTEGER"], "col": 0, "startrow": 0, "numrows": 5, "id": 0 } >>> rsps = sliderule.source("h5", rqst, stream=True) >>> print(rsps) [{'__rectype': 'h5dataset', 'datatype': 2, 'data': (245, 0, 0, 0, 7, 1, 0, 0, 17, 1, 0, 0, 1, 1, 0, 0, 4, 1, 0, 0), 'size': 20, 'offset': 0, 'id': 0}]
Response Payload (application/octet-stream)
Serialized stream of H5 dataset records of the type
h5dataset
. See De-serialization for a description of how to process binary response records.
h5p
POST /source/h5p <request payload>
Reads a list of datasets from an HDF5 file and returns the values of the datasets in a dictionary of lists.
See icesat2.h5p function for a convient method for accessing HDF5 datasets.
Request Payload (application/json)
parameter
description
default
asset
data source asset (see Assets)
required
resource
HDF5 filename
required
datasets
list of datasets (see h5 for a list of parameters for each dataset)
required
Python Example
>>> import sliderule >>> sliderule.set_url("icesat2sliderule.org") >>> asset = "nsidc-s3" >>> resource = "ATL03_20181019065445_03150111_003_01.h5" >>> dataset = "/gt1r/geolocation/segment_ph_cnt" >>> datasets = [ {"dataset": dataset, "col": 0, "startrow": 0, "numrows": 5} ] >>> rqst = { "asset" : asset, "resource": resource, "datasets": datasets, } >>> rsps = sliderule.source("h5p", rqst, stream=True) >>> print(rsps) [{'__rectype': 'h5file', 'dataset': '/gt1r/geolocation/segment_ph_cnt', 'elements': 5, 'size': 20, 'datatype': 2, 'data': (245, 0, 0, 0, 7, 1, 0, 0, 17, 1, 0, 0, 1, 1, 0, 0, 4, 1, 0, 0)}]
Response Payload (application/octet-stream)
Serialized stream of H5 file data records of the type
h5file
. See De-serialization for a description of how to process binary response records.
health
GET /source/health
Provides status on the health of the node.
Response Payload (application/json)
JSON object containing a true|false indicator of the health of the node.
{ "healthy": true|false }
index
GET /source/index <request payload>
Return list of resources (i.e H5 files) that match the query criteria.
Since the way resources are indexed (e.g. which meta-data to use), is very dependent upon the actual resources available; this endpoint is not necessarily useful in and of itself. It is expected that data specific indexes will be built per SlideRule deployment, and higher level routines will be constructed that take advantage of this endpoint and provide a more meaning interface.
Request Payload (application/json)
{ "or"|"and": { "<index name>": { <index parameters>... } ... } }
parameter
description
default
index name
name of server-side index to use (deployment specific)
required
index parameters
an index span represented in the format native to the index selected
required
Response Payload (application/json)
JSON object containing a list of the resources available to the SlideRule deployment that match the query criteria.
{ "resources": ["<resource name>", ...] }
metric
GET /source/metric <request payload>
Return a list of metric values associated with a provided system attribute.
Each SlideRule server node maintains internal metrics on a variety of things. Each metric is associated with an attribute that identifies a set of metrics.
When querying metrics, you provide the metric attribute, and the server will respond with the set of metrics associated with that attribute.
Request Payload (application/json)
{ "attr": <metric attribute> }
parameter
description
default
metric attribute
name of the attribute that is being queried
required
Response Payload (application/json)
JSON object containing a set of the metric names and values.
{ "<metric name>": <metric value>, ... }
tail
GET /source/tail <request payload>
Retrieve the most recent log messages generated by the server.
The number of log message saved by the server is configured at startup. This endpoint will return up to the maximum number of log messages that are saved.
Request Payload (application/json)
{ "monitor": "<monitor name>" }
parameter
description
default
monitor
name of the monitor to tail, should almost always be “EventMonitor”
required
Response Payload (application/json)
JSON object containing a list of log messages.
[ "<log message 1>", "<log message 2>", ... "<log message N>" ]
time
GET /source/time <request payload>
Converts times from one format to another
Request Payload (application/json)
parameter
description
default
time
time value
required
input
format of above time value: “NOW”, “CDS”, “GMT”, “GPS”
required
output
desired format of return value: same as above
required
Sliderule supports the following time specifications
- NOW
If supplied for either input or time then grab the current time
- CDS
CCSDS 6-byte packet timestamp represented as [<day>, <ms>]
days = 2 bytes of days since GPS epoch
ms = 4 bytes of milliseconds in the current day
- GMT
UTC time represented as a one of two date strings
“<year>:<month>:<day of month>:<hour in day>:<minute in hour>:<second in minute>””
“<year>:<day of year>:<hour in day>:<minute in hour>:<second in minute>”
- GPS
seconds since GPS epoch “January 6, 1980”
HTTP Example
GET /source/time HTTP/1.1 Host: my-sliderule-server:9081 Content-Length: 48 {"time": "NOW", "input": "NOW", "output": "GPS"}Python Example
rqst = { "time": "NOW", "input": "NOW", "output": "GPS" } rsps = sliderule.source("time", rqst)
Response Payload (application/json)
JSON object describing the results of the time conversion
{ "time": <time value> "format": "<format of time value>" }
version
GET /source/version
Get the version information of the server.
Response Payload (application/json)
JSON object containing the version information.
{ "server": { "packages": [ "<package 1>", "<package 2>", ... "<package n>" ], "version": "<version string>", "launch": "<date of launch>", "commit": "<commit id of code>", "duration": <seconds since launch> } "<package 1>": { "version": "<version string>", "commit": "<commit id of code>" }, "<package 2>": { "version": "<version string>", "commit": "<commit id of code>" }, ... "<package n>": { "version": "<version string>", "commit": "<commit id of code>" } }