The Spark integration uses OpenLineage to collect and send metadata to Grai. Complete documentation for integrating OpenLineage with Spark can be found here (opens in a new tab).
Web App
Fields
| Field | Value | Example |
|---|---|---|
| source | The name of the source, see sources | Spark |
| Name | Name for connection | production |
| Namespace | Namespace for the connection, see namespaces | default |
| namespaces | Optional |
Webhook
âšī¸
In order to authenticate your webhook, you will need to create an API key in the Grai web app.
Grai will open an endpoint at /api/v1/openlineage/<connection_id>/ which can be used to receive OpenLineage events.
See the OpenLineage docs (opens in a new tab) for how to configure your client to send OpenLineage events. As part of the configuration settings you'll need to to provide configuration settings via SparkConf parameters.
| Field | Value |
|---|---|
| spark.openlineage.transport.type | http |
| spark.openlineage.transport.endpoint | api/v1/openlineage/<connection_id> |
| spark.openlineage.transport.url | https://api.grai.io (opens in a new tab) |
| spark.openlineage.transport.auth.type | <your_grai_api_key> |
Did this doc help you?