gilles/scrutiny
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added instructions for how to create a Scope restricted InfluxDB API token for use with Scrutiny.

Browse Source 
- fixes #249
This commit is contained in:
Jason Kulatunga
2022-07-10 11:30:38 -07:00
parent 2361c329e2
commit c6579864b8
2 changed files with 321 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/TROUBLESHOOTING_INFLUXDB.md
+311 -4
View File
@@ -66,12 +66,319 @@ panic: failed to check influxdb setup status - parse "://:": missing protocol sc
As discussed in [#248](https://github.com/AnalogJ/scrutiny/issues/248) and [#234](https://github.com/AnalogJ/scrutiny/issues/234),
this usually related to either:
- Upgrading from the LSIO Scrutiny image to the Official Scrutiny image, without removing LSIO specific environmental variables
- remove the `SCRUTINY_WEB=true` and `SCRUTINY_COLLECTOR=true` environmental variables. They were used by the LSIO image, but are unnecessary and cause issues with the official Scrutiny image.
- Updated versions of the [LSIO Scrutiny images are broken](https://github.com/linuxserver/docker-scrutiny/issues/22), as they have not installed InfluxDB which is a required dependency of Scrutiny v0.4.x
- You can revert to an earlier version of the LSIO image (`lscr.io/linuxserver/scrutiny:060ac7b8-ls34`), or just change to the official Scrutiny image (`ghcr.io/analogj/scrutiny:master-omnibus`)
- Upgrading from the LSIO Scrutiny image to the Official Scrutiny image, without removing LSIO specific environmental
variables
- remove the `SCRUTINY_WEB=true` and `SCRUTINY_COLLECTOR=true` environmental variables. They were used by the LSIO
image, but are unnecessary and cause issues with the official Scrutiny image.
- Updated versions of the [LSIO Scrutiny images are broken](https://github.com/linuxserver/docker-scrutiny/issues/22),
as they have not installed InfluxDB which is a required dependency of Scrutiny v0.4.x
- You can revert to an earlier version of the LSIO image (`lscr.io/linuxserver/scrutiny:060ac7b8-ls34`), or just
change to the official Scrutiny image (`ghcr.io/analogj/scrutiny:master-omnibus`)
Here's a couple of confirmed working docker-compose files that you may want to look at:
- https://github.com/AnalogJ/scrutiny/blob/master/docker/example.hubspoke.docker-compose.yml
- https://github.com/AnalogJ/scrutiny/blob/master/docker/example.omnibus.docker-compose.yml
## Bring your own InfluxDB
> WARNING: Most users should not follow these steps. This is ONLY for users who have an EXISTING InfluxDB installation which contains data from multiple services.
> The Scrutiny Docker omnibus image includes an empty InfluxDB instance which it can configure.
> If you're deploying manually or via Hub/Spoke, you can just follow the installation instructions, Scrutiny knows how
> to run the first-time setup automatically.
The goal here is to create an InfluxDB API key with minimal permissions for use by Scrutiny.
- Create Scrutiny buckets (`metrics`, `metrics_weekly`, `metrics_monthly`, `metrics_yearly`) with placeholder config
- Create Downsampling tasks (`tsk-weekly-aggr`, `tsk-monthly-aggr`, `tsk-yearly-aggr`) with placeholder script.
- Create API token with restricted scope
- NOTE: Placeholder bucket & task configuration will be replaced automatically by Scrutiny during startup
The placeholder buckets and tasks need to be created before the API token can be created, as the resource ID's need to
exist for the scope restriction to work.
Scopes:
- `orgs`: read - required for scrutiny to find it's configured org_id
- `tasks`: scrutiny specific read/write access - Scrutiny only needs access to the downsampling tasks you created above
- `buckets`: scrutiny specific read/write access - Scrutiny only needs access to the buckets you created above
### Setup Environmental Variables
```bash
# replace the following values with correct values for your InfluxDB installation
export INFLUXDB_ADMIN_TOKEN=pCqRq7xxxxxx-FZgNLfstIs0w==
export INFLUXDB_ORG_ID=b2495xxxxx
export INFLUXDB_HOSTNAME=http://localhost:8086
# if you want to change the bucket name prefix below, you'll also need to update the setting in the scrutiny.yaml config file.
export INFLUXDB_SCRUTINY_BUCKET_BASENAME=metrics
```
### Create placeholder buckets
<details>
<summary>Click to expand!</summary>
```bash
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/buckets \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"name": "${INFLUXDB_SCRUTINY_BUCKET_BASENAME}",
"orgID": "${INFLUXDB_ORG_ID}",
"retentionRules": []
}
EOF
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/buckets \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"name": "${INFLUXDB_SCRUTINY_BUCKET_BASENAME}_weekly",
"orgID": "${INFLUXDB_ORG_ID}",
"retentionRules": []
}
EOF
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/buckets \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"name": "${INFLUXDB_SCRUTINY_BUCKET_BASENAME}_monthly",
"orgID": "${INFLUXDB_ORG_ID}",
"retentionRules": []
}
EOF
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/buckets \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"name": "${INFLUXDB_SCRUTINY_BUCKET_BASENAME}_yearly",
"orgID": "${INFLUXDB_ORG_ID}",
"retentionRules": []
}
EOF
```
</details>
### Create placeholder tasks
<details>
<summary>Click to expand!</summary>
```bash
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/tasks \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"orgID": "${INFLUXDB_ORG_ID}",
"flux": "option task = {name: \"tsk-weekly-aggr\", every: 1y} \nyield now()"
}
EOF
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/tasks \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"orgID": "${INFLUXDB_ORG_ID}",
"flux": "option task = {name: \"tsk-monthly-aggr\", every: 1y} \nyield now()"
}
EOF
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/tasks \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"orgID": "${INFLUXDB_ORG_ID}",
"flux": "option task = {name: \"tsk-yearly-aggr\", every: 1y} \nyield now()"
}
EOF
```
</details>
### Create InfluxDB API Token
<details>
<summary>Click to expand!</summary>
```bash
# replace these values with placeholder bucket and task ids from your InfluxDB installation.
export INFLUXDB_SCRUTINY_BASE_BUCKET_ID=1e0709xxxx
export INFLUXDB_SCRUTINY_WEEKLY_BUCKET_ID=1af03dexxxxx
export INFLUXDB_SCRUTINY_MONTHLY_BUCKET_ID=b3c59c7xxxxx
export INFLUXDB_SCRUTINY_YEARLY_BUCKET_ID=f381d8cxxxxx
export INFLUXDB_SCRUTINY_WEEKLY_TASK_ID=09a64ecxxxxx
export INFLUXDB_SCRUTINY_MONTHLY_TASK_ID=09a64xxxxx
export INFLUXDB_SCRUTINY_YEARLY_TASK_ID=09a64ecxxxxx
curl -sS -X POST ${INFLUXDB_HOSTNAME}/api/v2/authorizations \
-H "Content-Type: application/json" \
-H "Authorization: Token ${INFLUXDB_ADMIN_TOKEN}" \
--data-binary @- << EOF
{
"description": "scrutiny - restricted scope token",
"orgID": "${INFLUXDB_ORG_ID}",
"permissions": [
{
"action": "read",
"resource": {
"type": "orgs"
}
},
{
"action": "read",
"resource": {
"type": "tasks"
}
},
{
"action": "write",
"resource": {
"type": "tasks",
"id": "${INFLUXDB_SCRUTINY_WEEKLY_TASK_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "write",
"resource": {
"type": "tasks",
"id": "${INFLUXDB_SCRUTINY_MONTHLY_TASK_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "write",
"resource": {
"type": "tasks",
"id": "${INFLUXDB_SCRUTINY_YEARLY_TASK_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "read",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_BASE_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "write",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_BASE_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "read",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_WEEKLY_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "write",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_WEEKLY_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "read",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_MONTHLY_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "write",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_MONTHLY_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "read",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_YEARLY_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
},
{
"action": "write",
"resource": {
"type": "buckets",
"id": "${INFLUXDB_SCRUTINY_YEARLY_BUCKET_ID}",
"orgID": "${INFLUXDB_ORG_ID}"
}
}
]
}
EOF
```
</details>
### Save InfluxDB API Token
After running the Curl command above, you'll see a JSON response that looks like the following:
```json
{
"token": "ksVU2t5SkQwYkvIxxxxxxxYt2xUt0uRKSbSF1Po0UQ==",
"status": "active",
"description": "scrutiny - restricted scope token",
"orgID": "b2495586xxxx",
"org": "my-org",
"user": "admin",
"permissions": [
{
"action": "read",
"resource": {
"type": "orgs"
}
},
{
"action": "read",
"resource": {
"type": "tasks"
}
},
{
"action": "write",
"resource": {
"type": "tasks",
"id": "09a64exxxxx",
"orgID": "b24955860xxxxx",
"org": "my-org"
}
},
...
]
}
```
You must copy the token field from the JSON response, and save it in your `scrutiny.yaml` config file. After that's
done, you can start the Scrutiny server
webapp/backend/pkg/database/scrutiny_repository.go
+10 -2
View File
@@ -242,21 +242,29 @@ func (sr *scrutinyRepository) EnsureBuckets(ctx context.Context, org *domain.Org
//create buckets (used for downsampling)
weeklyBucket := fmt.Sprintf("%s_weekly", sr.appConfig.GetString("web.influxdb.bucket"))
if _, foundErr := sr.influxClient.BucketsAPI().FindBucketByName(ctx, weeklyBucket); foundErr != nil {
if foundWeeklyBucket, foundErr := sr.influxClient.BucketsAPI().FindBucketByName(ctx, weeklyBucket); foundErr != nil {
// metrics_weekly bucket will have a retention period of 8+1 weeks (since it will be down-sampled once a month)
_, err := sr.influxClient.BucketsAPI().CreateBucketWithName(ctx, org, weeklyBucket, weeklyBucketRetentionRule)
if err != nil {
return err
}
} else if sr.appConfig.GetBool("web.influxdb.retention_policy") {
//correctly set the retention period for the bucket (may not be able to do it during setup/creation)
foundWeeklyBucket.RetentionRules = domain.RetentionRules{weeklyBucketRetentionRule}
sr.influxClient.BucketsAPI().UpdateBucket(ctx, foundWeeklyBucket)
}
monthlyBucket := fmt.Sprintf("%s_monthly", sr.appConfig.GetString("web.influxdb.bucket"))
if _, foundErr := sr.influxClient.BucketsAPI().FindBucketByName(ctx, monthlyBucket); foundErr != nil {
if foundMonthlyBucket, foundErr := sr.influxClient.BucketsAPI().FindBucketByName(ctx, monthlyBucket); foundErr != nil {
// metrics_monthly bucket will have a retention period of 24+1 months (since it will be down-sampled once a year)
_, err := sr.influxClient.BucketsAPI().CreateBucketWithName(ctx, org, monthlyBucket, monthlyBucketRetentionRule)
if err != nil {
return err
}
} else if sr.appConfig.GetBool("web.influxdb.retention_policy") {
//correctly set the retention period for the bucket (may not be able to do it during setup/creation)
foundMonthlyBucket.RetentionRules = domain.RetentionRules{monthlyBucketRetentionRule}
sr.influxClient.BucketsAPI().UpdateBucket(ctx, foundMonthlyBucket)
}
yearlyBucket := fmt.Sprintf("%s_yearly", sr.appConfig.GetString("web.influxdb.bucket"))