(0.2.1) Automated packaging of release by Packagr

Signed-off-by: Jason Kulatunga <jason@thesparktree.com>
fix link
2020-09-24 03:26:38 +00:00 · 2020-09-23 20:51:13 -06:00 · 2020-09-23 20:49:56 -06:00 · 2020-09-23 11:01:53 -06:00 · 2020-09-23 10:37:59 -06:00 · 2020-09-23 09:54:33 -06:00
85 changed files with 17506 additions and 12856 deletions
@@ -0,0 +1,41 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Log Files**
+If related to missing devices or SMART data, please run the `collector` in DEBUG mode, and attach the log file.
+
+```
+docker run -it --rm -p 8080:8080 \
+-v /run/udev:/run/udev:ro \
+--cap-add SYS_RAWIO \
+--device=/dev/sda \
+--device=/dev/sdb \
+-e DEBUG=true \
+-e COLLECTOR_LOG_FILE=/tmp/collector.log \
+-e SCRUTINY_LOG_FILE=/tmp/web.log \
+--name scrutiny \
+analogj/scrutiny
+
+# in another terminal trigger the collector
+docker exec scrutiny scrutiny-collector-metrics run
+
+# then use docker cp to copy the log files out of the container.
+docker cp scrutiny:/tmp/collector.log collector.log
+docker cp scrutiny:/tmp/web.log web.log
+
+```
@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[FEAT]"
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
@@ -23,7 +23,7 @@ jobs:

          go mod vendor

-          go test -v -tags "static" $(go list ./... | grep -v /vendor/)
+          go test -race -coverprofile=coverage.txt -covermode=atomic -v -tags "static" $(go list ./... | grep -v /vendor/)

          go build -ldflags "-X main.goos=linux -X main.goarch=amd64" -o scrutiny-web-linux-amd64 -tags "static" webapp/backend/cmd/scrutiny/scrutiny.go
          go build -ldflags "-X main.goos=linux -X main.goarch=amd64" -o scrutiny-collector-metrics-linux-amd64 -tags "static" collector/cmd/collector-metrics/collector-metrics.go
@@ -37,6 +37,11 @@ jobs:
          path: |
            ${{ env.PROJECT_PATH }}/scrutiny-web-linux-amd64
            ${{ env.PROJECT_PATH }}/scrutiny-collector-metrics-linux-amd64
+      - uses: codecov/codecov-action@v1
+        with:
+          file: ${{ env.PROJECT_PATH }}/coverage.txt
+          flags: unittests
+          fail_ci_if_error: true
  build-docker:
    name: Build Docker
    runs-on: ubuntu-latest
@@ -1,34 +0,0 @@
-#TODO: once scrutiny is public, this file can be deleted.
-# builds a docker image and attaches it to the latest release.
-name: Release Docker
-
-on:
-  release:
-    # Only use the types keyword to narrow down the activity types that will trigger your workflow.
-    types: [published]
-jobs:
-  docker-release:
-    name: Docker Release
-    runs-on: ubuntu-latest
-    container: docker:19.03.2
-    env:
-      PROJECT_PATH: /go/src/github.com/analogj/scrutiny
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-        with:
-          ref: ${{github.event.release.tag_name}}
-      - name: Build Docker
-        run: |
-          docker build -t analogj/scrutiny:latest -t analogj/scrutiny:${{ github.event.release.tag_name }} -f docker/Dockerfile .
-          docker save -o docker-analogj-scrutiny-${{ github.event.release.tag_name }}.tar analogj/scrutiny
-      - name: Upload Collector Release Asset
-        id: upload-release-asset3
-        uses: actions/upload-release-asset@v1
-        env:
-          GITHUB_TOKEN: ${{ secrets.SCRUTINY_GITHUB_TOKEN }}
-        with:
-          upload_url: ${{ github.event.release.upload_url }} # This pulls from the CREATE RELEASE step above, referencing it's ID to get its outputs object, which include a `upload_url`. See this blog post for more info: https://jasonet.co/posts/new-features-of-github-actions/#passing-data-to-future-steps
-          asset_path: docker-analogj-scrutiny-${{ github.event.release.tag_name }}.tar
-          asset_name: docker-analogj-scrutiny-${{ github.event.release.tag_name }}.tar
-          asset_content_type: application/octet-stream
@@ -62,3 +62,5 @@ vendor
 /scrutiny-web-linux-amd64
 scrutiny-*.db
 scrutiny_test.db
+scrutiny.yaml
+coverage.txt
@@ -7,7 +7,12 @@ There are multiple ways to develop on the scrutiny codebase locally. The two mos
 ## Docker Development
 ```
 docker build -f docker/Dockerfile . -t analogj/scrutiny
-docker run -it --rm -p 9090:8080 -v /run:/run  -v /dev/disk:/dev/disk --privileged analogj/scrutiny
+docker run -it --rm -p 8080:8080 \
+-v /run/udev:/run/udev:ro \
+--cap-add SYS_RAWIO \
+--device=/dev/sda \
+--device=/dev/sdb \
+analogj/scrutiny
 /scrutiny/bin/scrutiny-collector-metrics run
 ```

@@ -29,9 +34,38 @@ cd webapp/frontend && ng build --watch --output-path=../../dist --deploy-url="/w
 > Note: if you do not add `--prod` flag, app will display mocked data for api calls.

 ### Backend
+
+If you're using the `ng build` command above to generate your frontend, you'll need to create a custom config file and
+override the `web.src.frontend.path` value.
+
 ```
-go run webapp/backend/cmd/scrutiny/scrutiny.go start --config ./example.scrutiny.yaml
+# config file for local development. store as scrutiny.yaml
+version: 1
+
+web:
+  listen:
+    port: 8080
+    host: 0.0.0.0
+  database:
+    # can also set absolute path here
+    location: ./scrutiny.db
+  src:
+    frontend:
+      path: ./dist
+
+
+log:
+  file: 'web.log' #absolute or relative paths allowed, eg. web.log
+  level: DEBUG
+
 ```
+
+Once you've created a config file, you can pass it to the scrutiny binary during startup.
+
+```
+go run webapp/backend/cmd/scrutiny/scrutiny.go start --config ./scrutiny.yaml
+```
+
 Now visit http://localhost:8080


@@ -40,3 +74,24 @@ Now visit http://localhost:8080
 brew install smartmontools
 go run collector/cmd/collector-metrics/collector-metrics.go run --debug
 ```
+
+
+## Debugging
+
+If you need more verbose logs for debugging, you can use the following environmental variables:
+
+- `DEBUG=true` - enables debug level logging on both the `collector` and `webapp`
+- `COLLECTOR_DEBUG=true` - enables debug level logging on the `collector`
+- `SCRUTINY_DEBUG=true` - enables debug level logging on the `webapp`
+
+In addition, you can instruct scrutiny to write its logs to a file using the following environmental variables:
+
+- `COLLECTOR_LOG_FILE=/tmp/collector.log` - write the `collector` logs to a file
+- `SCRUTINY_LOG_FILE=/tmp/web.log` - write the `webapp` logs to a file
+
+Finally, you can copy the files from the scrutiny container to your host using the following command(s)
+
+```
+docker cp scrutiny:/tmp/collector.log collector.log
+docker cp scrutiny:/tmp/web.log web.log
+```
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Jason Kulatunga
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -6,6 +6,15 @@


 # scrutiny
+
+[![CI](https://github.com/AnalogJ/scrutiny/workflows/CI/badge.svg?branch=master)](https://github.com/AnalogJ/scrutiny/actions?query=workflow%3ACI)
+[![codecov](https://codecov.io/gh/AnalogJ/scrutiny/branch/master/graph/badge.svg)](https://codecov.io/gh/AnalogJ/scrutiny)
+[![GitHub license](https://img.shields.io/github/license/AnalogJ/scrutiny.svg?style=flat-square)](https://github.com/AnalogJ/scrutiny/blob/master/LICENSE)
+[![Godoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](https://godoc.org/github.com/analogj/scrutiny)
+[![Go Report Card](https://goreportcard.com/badge/github.com/AnalogJ/scrutiny?style=flat-square)](https://goreportcard.com/report/github.com/AnalogJ/scrutiny)
+[![GitHub release](http://img.shields.io/github/release/AnalogJ/scrutiny.svg?style=flat-square)](https://github.com/AnalogJ/scrutiny/releases)
+
+
 WebUI for smartd S.M.A.R.T monitoring

 > NOTE: Scrutiny is a Work-in-Progress and still has some rough edges.
@@ -50,13 +59,17 @@ If you're using Docker, getting started is as simple as running the following co
 ```bash
 docker run -it --rm -p 8080:8080 \
 -v /run/udev:/run/udev:ro \
-v /dev/disk:/dev/disk \
+--cap-add SYS_RAWIO \
+--device=/dev/sda \
+--device=/dev/sdb \
 --name scrutiny \
--privileged analogj/scrutiny
+analogj/scrutiny
 ```

- `/run/udev` and `/dev/disk` are necessary to provide the Scrutiny collector with access to your drive metadata.
- `--privileged` is required to ensure that your hard disk devices are accessible within the container (this will be changed in a future release)
+- `/run/udev` is necessary to provide the Scrutiny collector with access to your device metadata
+- `--cap-add SYS_RAWIO` is necessary to allow `smartctl` permission to query your device SMART data
+    - NOTE: If you have NVMe drives, you must use `--cap-add SYS_ADMIN` instead. See [#26](https://github.com/AnalogJ/scrutiny/issues/26)
+- `--device` entries are required to ensure that your hard disk devices are accessible within the container
 - `analogj/scrutiny` is a omnibus image, containing both the webapp server (frontend & api) as well as the S.M.A.R.T metric collector. (see below)

 ### Hub/Spoke Deployment
@@ -66,6 +79,21 @@ In addition to the Omnibus image (available under the `latest` tag) there are 2
 - `analogj/scrutiny:collector` - Contains the Scrutiny data collector, `smartctl` binary and cron-like scheduler. You can run one collector on each server.
 - `analogj/scrutiny:web` - Contains the Web UI, API and Database. Only one container necessary

+```bash
+docker run -it --rm -p 8080:8080 \
+--name scrutiny-web \
+analogj/scrutiny:web
+
+docker run -it --rm \
+-v /run/udev:/run/udev:ro \
+--cap-add SYS_RAWIO \
+--device=/dev/sda \
+--device=/dev/sdb \
+-e SCRUTINY_API_ENDPOINT=http://SCRUTINY_WEB_IPADDRESS:8080 \
+--name scrutiny-collector \
+analogj/scrutiny:collector
+```
+

 ## Usage

@@ -100,9 +128,10 @@ We use SemVer for versioning. For the versions available, see the tags on this r

 Jason Kulatunga - Initial Development - @AnalogJ

-# License
+# Licenses

-MIT
+- MIT
+- Logo: [Glasses by matias porta lezcano](https://thenounproject.com/term/glasses/775232)

 # Sponsors

@@ -5,6 +5,7 @@ import (
 	"github.com/analogj/scrutiny/collector/pkg/collector"
 	"github.com/analogj/scrutiny/webapp/backend/pkg/version"
 	"github.com/sirupsen/logrus"
+	"io"
 	"log"
 	"os"
 	"time"
@@ -85,6 +86,16 @@ OPTIONS:
 						logrus.SetLevel(logrus.InfoLevel)
 					}

+					if c.IsSet("log-file") {
+						logFile, err := os.OpenFile(c.String("log-file"), os.O_CREATE|os.O_WRONLY, 0644)
+						if err != nil {
+							logrus.Errorf("Failed to open log file %s for output: %s", c.String("log-file"), err)
+							return err
+						}
+						defer logFile.Close()
+						logrus.SetOutput(io.MultiWriter(os.Stderr, logFile))
+					}
+
 					metricCollector, err := collector.CreateMetricsCollector(
 						collectorLogger,
 						c.String("api-endpoint"),
@@ -99,14 +110,23 @@ OPTIONS:

 				Flags: []cli.Flag{
 					&cli.StringFlag{
-						Name:  "api-endpoint",
-						Usage: "The api server endpoint",
-						Value: "http://localhost:8080",
+						Name:    "api-endpoint",
+						Usage:   "The api server endpoint",
+						Value:   "http://localhost:8080",
+						EnvVars: []string{"SCRUTINY_API_ENDPOINT"},
+					},
+
+					&cli.StringFlag{
+						Name:    "log-file",
+						Usage:   "Path to file for logging. Leave empty to use STDOUT",
+						Value:   "",
+						EnvVars: []string{"COLLECTOR_LOG_FILE"},
 					},

 					&cli.BoolFlag{
-						Name:  "debug",
-						Usage: "Enable debug logging",
+						Name:    "debug",
+						Usage:   "Enable debug logging",
+						EnvVars: []string{"COLLECTOR_DEBUG", "DEBUG"},
 					},
 				},
 			},
@@ -5,6 +5,7 @@ import (
 	"github.com/analogj/scrutiny/collector/pkg/collector"
 	"github.com/analogj/scrutiny/webapp/backend/pkg/version"
 	"github.com/sirupsen/logrus"
+	"io"
 	"log"
 	"os"
 	"time"
@@ -85,6 +86,16 @@ OPTIONS:
 						logrus.SetLevel(logrus.InfoLevel)
 					}

+					if c.IsSet("log-file") {
+						logFile, err := os.OpenFile(c.String("log-file"), os.O_CREATE|os.O_WRONLY, 0644)
+						if err != nil {
+							logrus.Errorf("Failed to open log file %s for output: %s", c.String("log-file"), err)
+							return err
+						}
+						defer logFile.Close()
+						logrus.SetOutput(io.MultiWriter(os.Stderr, logFile))
+					}
+
 					stCollector, err := collector.CreateSelfTestCollector(
 						collectorLogger,
 						c.String("api-endpoint"),
@@ -99,14 +110,23 @@ OPTIONS:

 				Flags: []cli.Flag{
 					&cli.StringFlag{
-						Name:  "api-endpoint",
-						Usage: "The api server endpoint",
-						Value: "http://localhost:8080",
+						Name:    "api-endpoint",
+						Usage:   "The api server endpoint",
+						Value:   "http://localhost:8080",
+						EnvVars: []string{"SCRUTINY_API_ENDPOINT"},
+					},
+
+					&cli.StringFlag{
+						Name:    "log-file",
+						Usage:   "Path to file for logging. Leave empty to use STDOUT",
+						Value:   "",
+						EnvVars: []string{"COLLECTOR_LOG_FILE"},
 					},

 					&cli.BoolFlag{
-						Name:  "debug",
-						Usage: "Enable debug logging",
+						Name:    "debug",
+						Usage:   "Enable debug logging",
+						EnvVars: []string{"COLLECTOR_DEBUG", "DEBUG"},
 					},
 				},
 			},
@@ -3,15 +3,8 @@ package collector
 import (
 	"bytes"
 	"encoding/json"
-	"errors"
-	"github.com/analogj/scrutiny/collector/pkg/models"
-	"github.com/jaypipes/ghw"
 	"github.com/sirupsen/logrus"
-	"io"
 	"net/http"
-	"os"
-	"os/exec"
-	"path"
 	"time"
 )

@@ -21,72 +14,6 @@ type BaseCollector struct {
 	logger *logrus.Entry
 }

-func (c *BaseCollector) DetectStorageDevices() ([]models.Device, error) {
-
-	block, err := ghw.Block()
-	if err != nil {
-		c.logger.Errorf("Error getting block storage info: %v", err)
-		return nil, err
-	}
-
-	approvedDisks := []models.Device{}
-	for _, disk := range block.Disks {
-
-		// ignore optical drives and floppy disks
-		if disk.DriveType == ghw.DRIVE_TYPE_FDD || disk.DriveType == ghw.DRIVE_TYPE_ODD {
-			c.logger.Debugf(" => Ignore: Optical or floppy disk - (found %s)\n", disk.DriveType.String())
-			continue
-		}
-
-		// ignore removable disks
-		if disk.IsRemovable {
-			c.logger.Debugf(" => Ignore: Removable disk (%v)\n", disk.IsRemovable)
-			continue
-		}
-
-		// ignore virtual disks & mobile phone storage devices
-		if disk.StorageController == ghw.STORAGE_CONTROLLER_VIRTIO || disk.StorageController == ghw.STORAGE_CONTROLLER_MMC {
-			c.logger.Debugf(" => Ignore: Virtual/multi-media storage controller - (found %s)\n", disk.StorageController.String())
-			continue
-		}
-
-		// ignore NVMe devices (not currently supported) TBA
-		if disk.StorageController == ghw.STORAGE_CONTROLLER_NVME {
-			c.logger.Debugf(" => Ignore: NVMe storage controller - (found %s)\n", disk.StorageController.String())
-			continue
-		}
-
-		// Skip unknown storage controllers, not usually S.M.A.R.T compatible.
-		if disk.StorageController == ghw.STORAGE_CONTROLLER_UNKNOWN {
-			c.logger.Debugf(" => Ignore: Unknown storage controller - (found %s)\n", disk.StorageController.String())
-			continue
-		}
-
-		diskModel := models.Device{
-			WWN:           disk.WWN,
-			Manufacturer:  disk.Vendor,
-			ModelName:     disk.Model,
-			InterfaceType: disk.StorageController.String(),
-			//InterfaceSpeed: string
-			SerialNumber: disk.SerialNumber,
-			Capacity:     int64(disk.SizeBytes),
-			//Firmware       string
-			//RotationSpeed  int
-
-			DeviceName: disk.Name,
-		}
-		if len(diskModel.WWN) == 0 {
-			//(macOS and some other os's) do not provide a WWN, so we're going to fallback to
-			//diskname as identifier if WWN is not present
-			diskModel.WWN = disk.Name
-		}
-
-		approvedDisks = append(approvedDisks, diskModel)
-	}
-
-	return approvedDisks, nil
-}
-
 func (c *BaseCollector) getJson(url string, target interface{}) error {

 	r, err := httpClient.Get(url)
@@ -113,29 +40,6 @@ func (c *BaseCollector) postJson(url string, body interface{}, target interface{
 	return json.NewDecoder(r.Body).Decode(target)
 }

-func (c *BaseCollector) ExecCmd(cmdName string, cmdArgs []string, workingDir string, environ []string) (string, error) {
-
-	cmd := exec.Command(cmdName, cmdArgs...)
-	var stdBuffer bytes.Buffer
-	mw := io.MultiWriter(os.Stdout, &stdBuffer)
-
-	cmd.Stdout = mw
-	cmd.Stderr = mw
-
-	if environ != nil {
-		cmd.Env = environ
-	}
-	if workingDir != "" && path.IsAbs(workingDir) {
-		cmd.Dir = workingDir
-	} else if workingDir != "" {
-		return "", errors.New("Working Directory must be an absolute path")
-	}
-
-	err := cmd.Run()
-	return stdBuffer.String(), err
-
-}
-
 func (c *BaseCollector) LogSmartctlExitCode(exitCode int) {
 	if exitCode&0x01 != 0 {
 		c.logger.Errorln("smartctl could not parse commandline")
@@ -2,14 +2,17 @@ package collector

 import (
 	"bytes"
+	"encoding/json"
 	"fmt"
+	"github.com/analogj/scrutiny/collector/pkg/common"
+	"github.com/analogj/scrutiny/collector/pkg/detect"
 	"github.com/analogj/scrutiny/collector/pkg/errors"
 	"github.com/analogj/scrutiny/collector/pkg/models"
 	"github.com/sirupsen/logrus"
 	"net/url"
+	"os"
 	"os/exec"
 	"strings"
-	"sync"
 )

 type MetricsCollector struct {
@@ -43,13 +46,18 @@ func (mc *MetricsCollector) Run() error {
 	apiEndpoint.Path = "/api/devices/register"

 	deviceRespWrapper := new(models.DeviceWrapper)
-	detectedStorageDevices, err := mc.DetectStorageDevices()
+
+	deviceDetector := detect.Detect{
+		Logger: mc.logger,
+	}
+	detectedStorageDevices, err := deviceDetector.Start()
 	if err != nil {
 		return err
 	}

 	mc.logger.Infoln("Sending detected devices to API, for filtering & validation")
-	mc.logger.Debugf("Detected devices: %v", detectedStorageDevices)
+	jsonObj, _ := json.Marshal(detectedStorageDevices)
+	mc.logger.Debugf("Detected devices: %v", string(jsonObj))
 	err = mc.postJson(apiEndpoint.String(), models.DeviceWrapper{
 		Data: detectedStorageDevices,
 	}, &deviceRespWrapper)
@@ -62,16 +70,19 @@ func (mc *MetricsCollector) Run() error {
 		return errors.ApiServerCommunicationError("An error occurred while retrieving filtered devices")
 	} else {
 		mc.logger.Debugln(deviceRespWrapper)
-		var wg sync.WaitGroup
-
+		//var wg sync.WaitGroup
 		for _, device := range deviceRespWrapper.Data {
 			// execute collection in parallel go-routines
-			wg.Add(1)
-			go mc.Collect(&wg, device.WWN, device.DeviceName)
+			//wg.Add(1)
+			//go mc.Collect(&wg, device.WWN, device.DeviceName, device.DeviceType)
+			mc.Collect(device.WWN, device.DeviceName, device.DeviceType)
+
+			// TODO: we may need to sleep for between each call to smartctl -a
+			//time.Sleep(30 * time.Millisecond)
 		}

-		mc.logger.Infoln("Main: Waiting for workers to finish")
-		wg.Wait()
+		//mc.logger.Infoln("Main: Waiting for workers to finish")
+		//wg.Wait()
 		mc.logger.Infoln("Main: Completed")
 	}

@@ -89,11 +100,19 @@ func (mc *MetricsCollector) Validate() error {
 	return nil
 }

-func (mc *MetricsCollector) Collect(wg *sync.WaitGroup, deviceWWN string, deviceName string) {
-	defer wg.Done()
+//func (mc *MetricsCollector) Collect(wg *sync.WaitGroup, deviceWWN string, deviceName string, deviceType string) {
+func (mc *MetricsCollector) Collect(deviceWWN string, deviceName string, deviceType string) {
+	//defer wg.Done()
 	mc.logger.Infof("Collecting smartctl results for %s\n", deviceName)

-	result, err := mc.ExecCmd("smartctl", []string{"-a", "-j", fmt.Sprintf("/dev/%s", deviceName)}, "", nil)
+	args := []string{"-a", "-j"}
+	//only include the device type if its a non-standard one. In some cases ata drives are detected as scsi in docker, and metadata is lost.
+	if len(deviceType) > 0 && deviceType != "scsi" && deviceType != "ata" {
+		args = append(args, "-d", deviceType)
+	}
+	args = append(args, fmt.Sprintf("%s%s", detect.DevicePrefix(), deviceName))
+
+	result, err := common.ExecCmd(mc.logger, "smartctl", args, "", os.Environ())
 	resultBytes := []byte(result)
 	if err != nil {
 		if exitError, ok := err.(*exec.ExitError); ok {
@@ -121,6 +140,7 @@ func (mc *MetricsCollector) Publish(deviceWWN string, payload []byte) error {

 	resp, err := httpClient.Post(apiEndpoint.String(), "application/json", bytes.NewBuffer(payload))
 	if err != nil {
+		mc.logger.Errorf("An error occurred while publishing SMART data for device (%s): %v", deviceWWN, err)
 		return err
 	}
 	defer resp.Body.Close()
@@ -0,0 +1,35 @@
+package common
+
+import (
+	"bytes"
+	"errors"
+	"github.com/sirupsen/logrus"
+	"io"
+	"os/exec"
+	"path"
+	"strings"
+)
+
+func ExecCmd(logger *logrus.Entry, cmdName string, cmdArgs []string, workingDir string, environ []string) (string, error) {
+	logger.Infof("Executing command: %s %s", cmdName, strings.Join(cmdArgs, " "))
+
+	cmd := exec.Command(cmdName, cmdArgs...)
+	var stdBuffer bytes.Buffer
+	mw := io.MultiWriter(logger.Logger.Out, &stdBuffer)
+
+	cmd.Stdout = mw
+	cmd.Stderr = mw
+
+	if environ != nil {
+		cmd.Env = environ
+	}
+	if workingDir != "" && path.IsAbs(workingDir) {
+		cmd.Dir = workingDir
+	} else if workingDir != "" {
+		return "", errors.New("Working Directory must be an absolute path")
+	}
+
+	err := cmd.Run()
+	return stdBuffer.String(), err
+
+}
@@ -1,7 +1,8 @@
-package collector_test
+package common_test

 import (
-	"github.com/analogj/scrutiny/collector/pkg/collector"
+	"github.com/analogj/scrutiny/collector/pkg/common"
+	"github.com/sirupsen/logrus"
 	"github.com/stretchr/testify/require"
 	"os/exec"
 	"testing"
@@ -11,10 +12,9 @@ func TestExecCmd(t *testing.T) {
 	t.Parallel()

 	//setup
-	bc := collector.BaseCollector{}

 	//test
-	result, err := bc.ExecCmd("echo", []string{"hello world"}, "", nil)
+	result, err := common.ExecCmd(logrus.WithField("exec", "test"), "echo", []string{"hello world"}, "", nil)

 	//assert
 	require.NoError(t, err)
@@ -25,10 +25,9 @@ func TestExecCmd_Date(t *testing.T) {
 	t.Parallel()

 	//setup
-	bc := collector.BaseCollector{}

 	//test
-	_, err := bc.ExecCmd("date", []string{}, "", nil)
+	_, err := common.ExecCmd(logrus.WithField("exec", "test"), "date", []string{}, "", nil)

 	//assert
 	require.NoError(t, err)
@@ -56,10 +55,9 @@ func TestExecCmd_InvalidCommand(t *testing.T) {
 	t.Parallel()

 	//setup
-	bc := collector.BaseCollector{}

 	//test
-	_, err := bc.ExecCmd("invalid_binary", []string{}, "", nil)
+	_, err := common.ExecCmd(logrus.WithField("exec", "test"), "invalid_binary", []string{}, "", nil)

 	//assert
 	_, castOk := err.(*exec.ExitError)
@@ -0,0 +1,116 @@
+package detect
+
+import (
+	"encoding/json"
+	"fmt"
+	"github.com/analogj/scrutiny/collector/pkg/common"
+	"github.com/analogj/scrutiny/collector/pkg/models"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/models/collector"
+	"github.com/denisbrodbeck/machineid"
+	"github.com/sirupsen/logrus"
+	"os"
+	"strings"
+)
+
+type Detect struct {
+	Logger *logrus.Entry
+}
+
+//private/common functions
+
+// This function calls smartctl --scan which can be used to detect storage devices.
+// It has a couple of issues however:
+// - --scan does not return any results on mac
+//
+// To handle these issues, we have OS specific wrapper functions that update/modify these detected devices.
+// models.Device returned from this function only contain the minimum data for smartctl to execute: device type and device name (device file).
+func (d *Detect) smartctlScan() ([]models.Device, error) {
+	//we use smartctl to detect all the drives available.
+	detectedDeviceConnJson, err := common.ExecCmd(d.Logger, "smartctl", []string{"--scan", "-j"}, "", os.Environ())
+	if err != nil {
+		d.Logger.Errorf("Error scanning for devices: %v", err)
+		return nil, err
+	}
+
+	var detectedDeviceConns models.Scan
+	err = json.Unmarshal([]byte(detectedDeviceConnJson), &detectedDeviceConns)
+	if err != nil {
+		d.Logger.Errorf("Error decoding detected devices: %v", err)
+		return nil, err
+	}
+
+	detectedDevices := []models.Device{}
+
+	for _, detectedDevice := range detectedDeviceConns.Devices {
+		detectedDevices = append(detectedDevices, models.Device{
+			DeviceType: detectedDevice.Type,
+			DeviceName: strings.TrimPrefix(detectedDevice.Name, DevicePrefix()),
+		})
+	}
+
+	return detectedDevices, nil
+}
+
+//updates a device model with information from smartctl --scan
+// It has a couple of issues however:
+// - WWN is provided as component data, rather than a "string". We'll have to generate the WWN value ourselves
+// - WWN from smartctl only provided for ATA protocol drives, NVMe and SCSI drives do not include WWN.
+func (d *Detect) smartCtlInfo(device *models.Device) error {
+
+	args := []string{"--info", "-j"}
+	//only include the device type if its a non-standard one. In some cases ata drives are detected as scsi in docker, and metadata is lost.
+	if len(device.DeviceType) > 0 && device.DeviceType != "scsi" && device.DeviceType != "ata" {
+		args = append(args, "-d", device.DeviceType)
+	}
+	args = append(args, fmt.Sprintf("%s%s", DevicePrefix(), device.DeviceName))
+
+	availableDeviceInfoJson, err := common.ExecCmd(d.Logger, "smartctl", args, "", os.Environ())
+	if err != nil {
+		d.Logger.Errorf("Could not retrieve device information for %s: %v", device.DeviceName, err)
+		return err
+	}
+
+	var availableDeviceInfo collector.SmartInfo
+	err = json.Unmarshal([]byte(availableDeviceInfoJson), &availableDeviceInfo)
+	if err != nil {
+		d.Logger.Errorf("Could not decode device information for %s: %v", device.DeviceName, err)
+		return err
+	}
+
+	//DeviceType and DeviceName are already populated.
+	//WWN
+	//InterfaceType:
+	device.ModelName = availableDeviceInfo.ModelName
+	device.InterfaceSpeed = availableDeviceInfo.InterfaceSpeed.Current.String
+	device.SerialNumber = availableDeviceInfo.SerialNumber
+	device.Firmware = availableDeviceInfo.FirmwareVersion
+	device.RotationSpeed = availableDeviceInfo.RotationRate
+	device.Capacity = availableDeviceInfo.UserCapacity.Bytes
+	device.FormFactor = availableDeviceInfo.FormFactor.Name
+	device.DeviceProtocol = availableDeviceInfo.Device.Protocol
+	if len(availableDeviceInfo.Vendor) > 0 {
+		device.Manufacturer = availableDeviceInfo.Vendor
+	}
+
+	//populate WWN is possible if present
+	if availableDeviceInfo.Wwn.Naa != 0 { //valid values are 1-6 (5 is what we handle correctly)
+		d.Logger.Info("Generating WWN")
+		wwn := Wwn{
+			Naa: availableDeviceInfo.Wwn.Naa,
+			Oui: availableDeviceInfo.Wwn.Oui,
+			Id:  availableDeviceInfo.Wwn.ID,
+		}
+		device.WWN = strings.ToLower(wwn.ToString())
+		d.Logger.Debugf("NAA: %d OUI: %d Id: %d => WWN: %s", wwn.Naa, wwn.Oui, wwn.Id, device.WWN)
+	} else {
+		d.Logger.Info("Using WWN Fallback")
+		d.wwnFallback(device)
+	}
+
+	return nil
+}
+
+//uses https://github.com/denisbrodbeck/machineid to get a OS specific unique machine ID.
+func (d *Detect) getMachineId() (string, error) {
+	return machineid.ProtectedID("scrutiny")
+}
@@ -0,0 +1,111 @@
+package detect
+
+import (
+	"github.com/analogj/scrutiny/collector/pkg/models"
+	"github.com/jaypipes/ghw"
+	"strings"
+)
+
+func DevicePrefix() string {
+	return "/dev/"
+}
+
+func (d *Detect) Start() ([]models.Device, error) {
+	// call the base/common functionality to get a list of devicess
+	detectedDevices, err := d.smartctlScan()
+	if err != nil {
+		return nil, err
+	}
+
+	//smartctl --scan doesn't seem to detect mac nvme drives, lets see if we can detect them manually.
+	missingDevices, err := d.findMissingDevices(detectedDevices) //we dont care about the error here, just continue retrieving device info.
+	if err == nil {
+		detectedDevices = append(detectedDevices, missingDevices...)
+	}
+
+	//inflate device info for detected devices.
+	for ndx, _ := range detectedDevices {
+		d.smartCtlInfo(&detectedDevices[ndx]) //ignore errors.
+	}
+
+	return detectedDevices, nil
+}
+
+func (d *Detect) findMissingDevices(detectedDevices []models.Device) ([]models.Device, error) {
+
+	missingDevices := []models.Device{}
+
+	block, err := ghw.Block()
+	if err != nil {
+		d.Logger.Errorf("Error getting block storage info: %v", err)
+		return nil, err
+	}
+
+	for _, disk := range block.Disks {
+
+		// ignore optical drives and floppy disks
+		if disk.DriveType == ghw.DRIVE_TYPE_FDD || disk.DriveType == ghw.DRIVE_TYPE_ODD {
+			d.Logger.Debugf(" => Ignore: Optical or floppy disk - (found %s)\n", disk.DriveType.String())
+			continue
+		}
+
+		// ignore removable disks
+		if disk.IsRemovable {
+			d.Logger.Debugf(" => Ignore: Removable disk (%v)\n", disk.IsRemovable)
+			continue
+		}
+
+		// ignore virtual disks & mobile phone storage devices
+		if disk.StorageController == ghw.STORAGE_CONTROLLER_VIRTIO || disk.StorageController == ghw.STORAGE_CONTROLLER_MMC {
+			d.Logger.Debugf(" => Ignore: Virtual/multi-media storage controller - (found %s)\n", disk.StorageController.String())
+			continue
+		}
+
+		// Skip unknown storage controllers, not usually S.M.A.R.T compatible.
+		if disk.StorageController == ghw.STORAGE_CONTROLLER_UNKNOWN {
+			d.Logger.Debugf(" => Ignore: Unknown storage controller - (found %s)\n", disk.StorageController.String())
+			continue
+		}
+
+		//check if device is already detected.
+		alreadyDetected := false
+		diskName := strings.TrimPrefix(disk.Name, DevicePrefix())
+		for _, detectedDevice := range detectedDevices {
+
+			if detectedDevice.DeviceName == diskName {
+				alreadyDetected = true
+				break
+			}
+		}
+		if !alreadyDetected {
+			missingDevices = append(missingDevices, models.Device{
+				DeviceName: diskName,
+				DeviceType: "",
+			})
+		}
+	}
+	return missingDevices, nil
+}
+
+//WWN values NVMe and SCSI
+func (d *Detect) wwnFallback(detectedDevice *models.Device) {
+	block, err := ghw.Block()
+	if err == nil {
+		for _, disk := range block.Disks {
+			if disk.Name == detectedDevice.DeviceName {
+				d.Logger.Debugf("Found matching block device. WWN: %s", disk.WWN)
+				detectedDevice.WWN = disk.WWN
+				break
+			}
+		}
+	}
+
+	//no WWN found, or could not open Block devices. Either way, fallback to serial number
+	if len(detectedDevice.WWN) == 0 {
+		d.Logger.Debugf("WWN is empty, falling back to serial number: %s", detectedDevice.SerialNumber)
+		detectedDevice.WWN = detectedDevice.SerialNumber
+	}
+
+	//wwn must always be lowercase.
+	detectedDevice.WWN = strings.ToLower(detectedDevice.WWN)
+}
@@ -0,0 +1,49 @@
+package detect
+
+import (
+	"github.com/analogj/scrutiny/collector/pkg/models"
+	"github.com/jaypipes/ghw"
+	"strings"
+)
+
+func DevicePrefix() string {
+	return "/dev/"
+}
+
+func (d *Detect) Start() ([]models.Device, error) {
+	// call the base/common functionality to get a list of devices
+	detectedDevices, err := d.smartctlScan()
+	if err != nil {
+		return nil, err
+	}
+
+	//inflate device info for detected devices.
+	for ndx, _ := range detectedDevices {
+		d.smartCtlInfo(&detectedDevices[ndx]) //ignore errors.
+	}
+
+	return detectedDevices, nil
+}
+
+//WWN values NVMe and SCSI
+func (d *Detect) wwnFallback(detectedDevice *models.Device) {
+	block, err := ghw.Block()
+	if err == nil {
+		for _, disk := range block.Disks {
+			if disk.Name == detectedDevice.DeviceName {
+				d.Logger.Debugf("Found matching block device. WWN: %s", disk.WWN)
+				detectedDevice.WWN = disk.WWN
+				break
+			}
+		}
+	}
+
+	//no WWN found, or could not open Block devices. Either way, fallback to serial number
+	if len(detectedDevice.WWN) == 0 {
+		d.Logger.Debugf("WWN is empty, falling back to serial number: %s", detectedDevice.SerialNumber)
+		detectedDevice.WWN = detectedDevice.SerialNumber
+	}
+
+	//wwn must always be lowercase.
+	detectedDevice.WWN = strings.ToLower(detectedDevice.WWN)
+}
@@ -0,0 +1,16 @@
+package detect_test
+
+import (
+	"github.com/analogj/scrutiny/collector/pkg/detect"
+	"github.com/stretchr/testify/require"
+	"testing"
+)
+
+func TestDevicePrefix(t *testing.T) {
+	//setup
+
+	//test
+
+	//assert
+	require.Equal(t, "/dev/", detect.DevicePrefix())
+}
@@ -0,0 +1,38 @@
+package detect
+
+import (
+	"github.com/analogj/scrutiny/collector/pkg/models"
+	"github.com/jaypipes/ghw"
+	"strings"
+)
+
+func DevicePrefix() string {
+	return ""
+}
+
+func (d *Detect) Start() ([]models.Device, error) {
+	// call the base/common functionality to get a list of devices
+	detectedDevices, err := d.smartctlScan()
+	if err != nil {
+		return nil, err
+	}
+
+	//inflate device info for detected devices.
+	for ndx, _ := range detectedDevices {
+		d.smartCtlInfo(&detectedDevices[ndx]) //ignore errors.
+	}
+
+	return detectedDevices, nil
+}
+
+//WWN values NVMe and SCSI
+func (d *Detect) wwnFallback(detectedDevice *models.Device) {
+
+	//fallback to serial number
+	if len(detectedDevice.WWN) == 0 {
+		detectedDevice.WWN = detectedDevice.SerialNumber
+	}
+
+	//wwn must always be lowercase.
+	detectedDevice.WWN = strings.ToLower(detectedDevice.WWN)
+}
@@ -0,0 +1,59 @@
+package detect
+
+import (
+	"fmt"
+	"strings"
+)
+
+type Wwn struct {
+	Naa        uint64 `json:"naa"`
+	Oui        uint64 `json:"oui"`
+	Id         uint64 `json:"id"`
+	VendorCode string `json:"vendor_code"`
+}
+
+// this is an incredibly basic converter, that only works for "Registered" IEEE format - NAA5
+// https://standards.ieee.org/content/dam/ieee-standards/standards/web/documents/tutorials/fibre.pdf
+// references:
+// - https://metacpan.org/pod/Device::WWN
+// - https://en.wikipedia.org/wiki/World_Wide_Name
+// - https://storagemeat.blogspot.com/2012/08/decoding-wwids-or-how-to-tell-whats-what.html
+// - https://bryanchain.com/2016/01/20/breaking-down-an-naa-id-world-wide-name/
+
+/*
+----------+---+---+---+---+---+---+---+---+
+| Byte/Bit | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
+----------+---+---+---+---+---+---+---+---+
+| 0        | NAA (5h)      | (MSB)         |
+----------+---------------+               +
+| 1        |                               |
+----------+            IEEE OUI           |
+| 2        |                               |
+----------+               +---------------+
+| 3        | (LSB)         | (MSB)         |
+----------+---------------+               +
+| 4        |                               |
+|          |                               |
+----------+                               |
+| 5        |            Vendor ID          |
+----------+                               |
+| 6        |                               |
+----------+                               |
+| 7        |                         (LSB) |
+----------+-------------------------------+
+
+
+*/
+
+func (wwn *Wwn) ToString() string {
+
+	var wwnBuffer uint64
+
+	wwnBuffer = wwn.Id           //start with vendor ID
+	wwnBuffer += (wwn.Oui << 36) //add left-shifted OUI
+	wwnBuffer += (wwn.Naa << 60) //NAA is a number from 1-6, so decimal == hex.
+
+	//TODO: may need to support additional versions in the future.
+
+	return strings.ToLower(fmt.Sprintf("%#x", wwnBuffer))
+}
@@ -0,0 +1,35 @@
+package detect_test
+
+import (
+	"fmt"
+	"github.com/analogj/scrutiny/collector/pkg/detect"
+	"github.com/stretchr/testify/require"
+	"testing"
+)
+
+func TestWwn_FromStringTable(t *testing.T) {
+
+	//setup
+	var tests = []struct {
+		wwnStr string
+		wwn    detect.Wwn
+	}{
+
+		{"0x5002538e40a22954", detect.Wwn{Naa: 5, Oui: 9528, Id: 61213911380}}, //sda
+		{"0x5000cca264eb01d7", detect.Wwn{Naa: 5, Oui: 3274, Id: 10283057623}}, //sdb
+		{"0x5000cca264ec3183", detect.Wwn{Naa: 5, Oui: 3274, Id: 10283135363}}, //sdc
+		{"0x5000cca252c859cc", detect.Wwn{Naa: 5, Oui: 3274, Id: 9978796492}},  //sdd
+		{"0x50014ee20b2a72a9", detect.Wwn{Naa: 5, Oui: 5358, Id: 8777265833}},  //sde
+		{"0x5000cca264ebc248", detect.Wwn{Naa: 5, Oui: 3274, Id: 10283106888}}, //sdf
+		{"0x5000c500673e6b5f", detect.Wwn{Naa: 5, Oui: 3152, Id: 1732143967}},  //sdg
+	}
+	//test
+	for _, tt := range tests {
+		testname := fmt.Sprintf("%s", tt.wwnStr)
+		t.Run(testname, func(t *testing.T) {
+			str := tt.wwn.ToString()
+			require.Equal(t, tt.wwnStr, str)
+		})
+	}
+
+}
@@ -14,6 +14,8 @@ type Device struct {
 	Capacity       int64  `json:"capacity"`
 	FormFactor     string `json:"form_factor"`
 	SmartSupport   bool   `json:"smart_support"`
+	DeviceProtocol string `json:"device_protocol"` //protocol determines which smart attribute types are available (ATA, NVMe, SCSI)
+	DeviceType     string `json:"device_type"`     //device type is used for querying with -d/t flag, should only be used by collector.
 }

 type DeviceWrapper struct {
@@ -0,0 +1,19 @@
+package models
+
+type Scan struct {
+	JSONFormatVersion []int `json:"json_format_version"`
+	Smartctl          struct {
+		Version      []int    `json:"version"`
+		SvnRevision  string   `json:"svn_revision"`
+		PlatformInfo string   `json:"platform_info"`
+		BuildInfo    string   `json:"build_info"`
+		Argv         []string `json:"argv"`
+		ExitStatus   int      `json:"exit_status"`
+	} `json:"smartctl"`
+	Devices []struct {
+		Name     string `json:"name"`
+		InfoName string `json:"info_name"`
+		Type     string `json:"type"`
+		Protocol string `json:"protocol"`
+	} `json:"devices"`
+}
@@ -15,7 +15,7 @@ FROM node:lts-slim as frontendbuild
 ENV NPM_CONFIG_LOGLEVEL=warn NG_CLI_ANALYTICS=false

 WORKDIR /scrutiny/src
-COPY ./webapp/frontend /scrutiny/src
+COPY webapp/frontend /scrutiny/src

 RUN npm install -g @angular/cli@9.1.4 && \
    mkdir -p /scrutiny/dist && \
@@ -34,4 +34,4 @@ COPY --from=frontendbuild /scrutiny/dist /scrutiny/web
 RUN chmod +x /scrutiny/bin/scrutiny && \
    mkdir -p /scrutiny/web && \
    mkdir -p /scrutiny/config
-CMD ["/scrutiny", "start"]
+CMD ["/scrutiny/bin/scrutiny", "start"]
@@ -22,43 +22,64 @@ web:
    host: 0.0.0.0
  database:
    # can also set absolute path here
-    location: ./scrutiny.db
+    location: /scrutiny/config/scrutiny.db
  src:
    frontend:
-      path: ./dist
+      path: /scrutiny/web

-disks:
-  include:
-    # - /dev/sda
-  exclude:
-    # - /dev/sdb

-notify:
-  level: 'warn' # 'warn' or 'error'
-  urls:
-    - "discord://token@channel"
-    - "telegram://token@telegram?channels=channel-1[,channel-2,...]"
-    - "pushover://shoutrrr:apiToken@userKey/?devices=device1[,device2, ...]"
-    - "slack://[botname@]token-a/token-b/token-c"
-    - "smtp://username:password@host:port/?fromAddress=fromAddress&toAddresses=recipient1[,recipient2,...]"
-    - "teams://token-a/token-b/token-c"
-    - "gotify://gotify-host/token"
-    - "pushbullet://api-token[/device/#channel/email]"
-    - "ifttt://key/?events=event1[,event2,...]&value1=value1&value2=value2&value3=value3"
-    - "mattermost://[username@]mattermost-host/token[/channel]"
-    - "hangouts://chat.googleapis.com/v1/spaces/FOO/messages?key=bar&token=baz"
-    - "zulip://bot-mail:bot-key@zulip-domain/?stream=name-or-id&topic=name"
-    - "join://shoutrrr:api-key@join/?devices=device1[,device2, ...][&icon=icon][&title=title]"
-    - "script:///file/path/on/disk"
-    - "https://www.example.com/path"
+log:
+  file: '' #absolute or relative paths allowed, eg. web.log
+  level: INFO

-collect:
-  metric:
-    enable: true
-    command: '-a -o on -S on'
-  long:
-    enable: false
-    command: ''
-  short:
-    enable: false
-    command: ''
+# The following commented out sections are a preview of additional configuration options that will be available soon.
+
+#disks:
+#  include:
+#    # - /dev/sda
+#  exclude:
+#    # - /dev/sdb
+
+#notify:
+#  urls:
+#    - "discord://token@channel"
+#    - "telegram://token@telegram?channels=channel-1[,channel-2,...]"
+#    - "pushover://shoutrrr:apiToken@userKey/?devices=device1[,device2, ...]"
+#    - "slack://[botname@]token-a/token-b/token-c"
+#    - "smtp://username:password@host:port/?fromAddress=fromAddress&toAddresses=recipient1[,recipient2,...]"
+#    - "teams://token-a/token-b/token-c"
+#    - "gotify://gotify-host/token"
+#    - "pushbullet://api-token[/device/#channel/email]"
+#    - "ifttt://key/?events=event1[,event2,...]&value1=value1&value2=value2&value3=value3"
+#    - "mattermost://[username@]mattermost-host/token[/channel]"
+#    - "hangouts://chat.googleapis.com/v1/spaces/FOO/messages?key=bar&token=baz"
+#    - "zulip://bot-mail:bot-key@zulip-domain/?stream=name-or-id&topic=name"
+#    - "join://shoutrrr:api-key@join/?devices=device1[,device2, ...][&icon=icon][&title=title]"
+#    - "script:///file/path/on/disk"
+#    - "https://www.example.com/path"
+
+#limits:
+#  ata:
+#    critical:
+#      error: 10
+#    standard:
+#      error: 20
+#      warn: 10
+#  scsi:
+#    critical: true
+#    standard: true
+#  nvme:
+#    critical: true
+#    standard: true
+
+
+#collect:
+#  metric:
+#    enable: true
+#    command: '-a -o on -S on'
+#  long:
+#    enable: false
+#    command: ''
+#  short:
+#    enable: false
+#    command: ''
@@ -3,14 +3,16 @@ module github.com/analogj/scrutiny
 go 1.13

 require (
+	github.com/AnalogJ/go-util v0.0.0-20200905200945-3b93d31215ae // indirect
 	github.com/analogj/go-util v0.0.0-20190301173314-5295e364eb14
+	github.com/containrrr/shoutrrr v0.0.0-20200828202222-1da53231b05a
+	github.com/denisbrodbeck/machineid v1.0.1
 	github.com/fatih/color v1.9.0
 	github.com/gin-gonic/gin v1.6.3
 	github.com/golang/mock v1.4.3
 	github.com/jaypipes/ghw v0.6.1
 	github.com/jinzhu/gorm v1.9.14
 	github.com/kvz/logstreamer v0.0.0-20150507115422-a635b98146f0 // indirect
-	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/sirupsen/logrus v1.2.0
 	github.com/spf13/viper v1.7.0
 	github.com/stretchr/testify v1.5.1
@@ -11,6 +11,8 @@ cloud.google.com/go/firestore v1.1.0/go.mod h1:ulACoGHTpvq5r8rxGJ4ddJZBZqakUQqCl
 cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
 cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
+github.com/AnalogJ/go-util v0.0.0-20200905200945-3b93d31215ae h1:iYSadgTTTmFTvZwdDImnytps8Hq9zlpWeNfYpe1RTPs=
+github.com/AnalogJ/go-util v0.0.0-20200905200945-3b93d31215ae/go.mod h1:0jFBtvNk8rNzZjL8j1b852fcka5/VJfJvRiU2w6OIkI=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
@@ -23,6 +25,7 @@ github.com/analogj/go-util v0.0.0-20190301173314-5295e364eb14 h1:wsrSjiqQtseStRI
 github.com/analogj/go-util v0.0.0-20190301173314-5295e364eb14/go.mod h1:lJQVqFKMV5/oDGYR2bra2OljcF3CvolAoyDRyOA4k4E=
 github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
 github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
+github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
 github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
 github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -31,16 +34,24 @@ github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kB
 github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/containrrr/shoutrrr v0.0.0-20200828202222-1da53231b05a h1:6ZMiughZYF6fJjFIf2X3D7AfImJeXnTMJ9qC2v75WPw=
+github.com/containrrr/shoutrrr v0.0.0-20200828202222-1da53231b05a/go.mod h1:z3pUtEhu5zOpu+Q8wZWiEq+ZLL9hM0HiFNhttaI67Ks=
 github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
+github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
+github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
 github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
 github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d h1:U+s90UTSYgptZMwQh2aRr3LuazLJIa+Pg3Kc1ylSYVY=
 github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
+github.com/cpuguy83/go-md2man/v2 v2.0.0 h1:EoUDS0afbrsXAZ9YQ9jdu/mZ2sXgT1/2yyNng4PGlyM=
+github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/denisbrodbeck/machineid v1.0.1 h1:geKr9qtkB876mXguW2X6TU4ZynleN6ezuMSRhl4D7AQ=
+github.com/denisbrodbeck/machineid v1.0.1/go.mod h1:dJUwb7PTidGDeYyUBmXZ2GphQBbjJCrnectwCyxcUSI=
 github.com/denisenkom/go-mssqldb v0.0.0-20191124224453-732737034ffd/go.mod h1:xbL0rPBG9cCiLr28tMa8zpbdarY27NDyej4t/EjAShU=
 github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
 github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
@@ -48,8 +59,12 @@ github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5/go.mod h1:a
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
 github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
+github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
+github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
 github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
+github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
@@ -98,6 +113,7 @@ github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm4
 github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
 github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
@@ -123,7 +139,10 @@ github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO
 github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ=
 github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I=
 github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
+github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
+github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/jarcoal/httpmock v1.0.4/go.mod h1:ATjnClrvW/3tijVmpL/va5Z3aAyGvqU3gCT8nX0Txik=
 github.com/jaypipes/ghw v0.6.1 h1:Ewt3mdpiyhWotGyzg1ursV/6SnToGcG4215X6rR2af8=
 github.com/jaypipes/ghw v0.6.1/go.mod h1:QOXppNRCLGYR1H+hu09FxZPqjNt09bqUZUnOL3Rcero=
 github.com/jaypipes/pcidb v0.5.0 h1:4W5gZ+G7QxydevI8/MmmKdnIPJpURqJ2JNXTzfLxF5c=
@@ -154,6 +173,7 @@ github.com/kvz/logstreamer v0.0.0-20150507115422-a635b98146f0/go.mod h1:8/LTPeDL
 github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
 github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/lib/pq v1.1.1/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
+github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
 github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
@@ -178,6 +198,8 @@ github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0Qu
 github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/mitchellh/mapstructure v1.2.2 h1:dxe5oCinTXiTIcfgmZecdCzPmAJKd46KsCWc35r0TV4=
+github.com/mitchellh/mapstructure v1.2.2/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -186,9 +208,15 @@ github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9
 github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
+github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
+github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
 github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
 github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/pelletier/go-toml v1.7.0 h1:7utD74fnzVc/cpcyy8sjrlFr5vYpypUixARcHIMIGuI=
+github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -220,14 +248,25 @@ github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4k
 github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
+github.com/spf13/afero v1.2.2 h1:5jhuqJyZCZf2JRofRvN/nIFgIWNzPa3/Vz8mYylgbWc=
+github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
 github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
+github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
 github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
+github.com/spf13/cobra v0.0.7/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
 github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
+github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
 github.com/spf13/pflag v1.0.2/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
+github.com/spf13/viper v1.6.3/go.mod h1:jUMtyi0/lB5yZH/FjyGAoH7IMNrIhlBf6pXZmbMDvzw=
 github.com/spf13/viper v1.7.0 h1:xVKxvI7ouOI5I+U9s2eeiUfMaWBVoXA3AWskkrqK0VM=
 github.com/spf13/viper v1.7.0/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -240,6 +279,7 @@ github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5
 github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
 github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
+github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
 github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
 github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
 github.com/ugorji/go/codec v1.1.7 h1:2SvQaVZ1ouYrrKKwoSk2pzd4A9evlKJb9oTL+OaLUSs=
@@ -247,6 +287,7 @@ github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLY
 github.com/urfave/cli/v2 v2.2.0 h1:JTTnM6wKzdA0Jqodd966MVj4vWbbquZykeX1sKbe2C4=
 github.com/urfave/cli/v2 v2.2.0/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2caLfQ=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
+github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
 go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
@@ -283,6 +324,7 @@ golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
 golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -293,6 +335,7 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -308,6 +351,7 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -319,10 +363,13 @@ golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd h1:xhmwyvizuTgC2qz7ZlMluP20uW+C3Rm0FD/WLDX8884=
 golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200409092240-59c9f1ba88fa h1:mQTN3ECqfsViCNBgq+A40vdwhkGykrrQlYe3mPj6BoU=
+golang.org/x/sys v0.0.0-20200409092240-59c9f1ba88fa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -347,7 +394,10 @@ golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtn
 golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.0.0-20191112195655-aa38f8e97acc/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
 google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
@@ -368,14 +418,21 @@ google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBr
 google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
+google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
 google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
+gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
+gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
 gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
 gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
+gopkg.in/ini.v1 v1.55.0 h1:E8yzL5unfpW3M6fz/eB7Cb5MQAYSZ7GKo4Qth+N2sgQ=
+gopkg.in/ini.v1 v1.55.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
 gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -383,6 +440,8 @@ gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
 gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gosrc.io/xmpp v0.1.1 h1:iMtE9W3fx254+4E6rI34AOPJDqWvpfQR6EYaVMzhJ4s=
+gosrc.io/xmpp v0.1.1/go.mod h1:4JgaXzw4MnEv2sGltONtK3GMhj+h9gpQ7cO8nwbFJLU=
 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
@@ -19,9 +19,9 @@ resultSinks:

 jobs:
  MetricsJob:
-    cmd: /scrutiny/bin/scrutiny-collector-metrics run --api-endpoint ${SCRUTINY_API_ENDPOINT:-http://localhost:8080}
+    cmd: /scrutiny/bin/scrutiny-collector-metrics run
    # run daily at midnight.
-    time: '0 0 * * *'
+    time: '0 0 0 * * *'
    onError: Backoff
    notifyOnSuccess:
      - *filesystemSink
@@ -95,10 +95,18 @@ OPTIONS:
 						if err != nil {                             // Handle errors reading the config file
 							//ignore "could not find config file"
 							fmt.Printf("Could not find config file at specified path: %s", c.String("config"))
-							os.Exit(1)
+							return err
 						}
 					}

+					if c.Bool("debug") {
+						config.Set("log.level", "DEBUG")
+					}
+
+					if c.IsSet("log-file") {
+						config.Set("log.file", c.String("log-file"))
+					}
+
 					webServer := web.AppEngine{Config: config}

 					return webServer.Start()
@@ -109,6 +117,18 @@ OPTIONS:
 						Name:  "config",
 						Usage: "Specify the path to the config file",
 					},
+					&cli.StringFlag{
+						Name:    "log-file",
+						Usage:   "Path to file for logging. Leave empty to use STDOUT",
+						Value:   "",
+						EnvVars: []string{"SCRUTINY_LOG_FILE"},
+					},
+
+					&cli.BoolFlag{
+						Name:    "debug",
+						Usage:   "Enable debug logging",
+						EnvVars: []string{"SCRUTINY_DEBUG", "DEBUG"},
+					},
 				},
 			},
 		},
@@ -30,22 +30,24 @@ func (c *configuration) Init() error {
 	c.SetDefault("web.listen.port", "8080")
 	c.SetDefault("web.listen.host", "0.0.0.0")
 	c.SetDefault("web.src.frontend.path", "/scrutiny/web")
-
 	c.SetDefault("web.database.location", "/scrutiny/config/scrutiny.db")

-	c.SetDefault("disks.include", []string{})
-	c.SetDefault("disks.exclude", []string{})
+	c.SetDefault("log.level", "INFO")
+	c.SetDefault("log.file", "")

-	c.SetDefault("notify.metric.script", "/scrutiny/config/notify-metrics.sh")
-	c.SetDefault("notify.long.script", "/scrutiny/config/notify-long-test.sh")
-	c.SetDefault("notify.short.script", "/scrutiny/config/notify-short-test.sh")
+	//c.SetDefault("disks.include", []string{})
+	//c.SetDefault("disks.exclude", []string{})

-	c.SetDefault("collect.metric.enable", true)
-	c.SetDefault("collect.metric.command", "-a -o on -S on")
-	c.SetDefault("collect.long.enable", true)
-	c.SetDefault("collect.long.command", "-a -o on -S on")
-	c.SetDefault("collect.short.enable", true)
-	c.SetDefault("collect.short.command", "-a -o on -S on")
+	//c.SetDefault("notify.metric.script", "/scrutiny/config/notify-metrics.sh")
+	//c.SetDefault("notify.long.script", "/scrutiny/config/notify-long-test.sh")
+	//c.SetDefault("notify.short.script", "/scrutiny/config/notify-short-test.sh")
+
+	//c.SetDefault("collect.metric.enable", true)
+	//c.SetDefault("collect.metric.command", "-a -o on -S on")
+	//c.SetDefault("collect.long.enable", true)
+	//c.SetDefault("collect.long.command", "-a -o on -S on")
+	//c.SetDefault("collect.short.enable", true)
+	//c.SetDefault("collect.short.command", "-a -o on -S on")

 	//if you want to load a non-standard location system config file (~/drawbridge.yml), use ReadConfig
 	c.SetConfigType("yaml")
@@ -1,32 +0,0 @@
-package database
-
-import (
-	"fmt"
-	"github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
-	"github.com/gin-gonic/gin"
-	"github.com/jinzhu/gorm"
-	_ "github.com/jinzhu/gorm/dialects/sqlite"
-)
-
-func DatabaseHandler(dbPath string) gin.HandlerFunc {
-	//var database *gorm.DB
-	fmt.Printf("Trying to connect to database stored: %s", dbPath)
-	database, err := gorm.Open("sqlite3", dbPath)
-
-	if err != nil {
-		panic("Failed to connect to database!")
-	}
-
-	database.AutoMigrate(&db.Device{})
-	database.AutoMigrate(&db.SelfTest{})
-	database.AutoMigrate(&db.Smart{})
-	database.AutoMigrate(&db.SmartAtaAttribute{})
-	database.AutoMigrate(&db.SmartNvmeAttribute{})
-	database.AutoMigrate(&db.SmartScsiAttribute{})
-
-	//TODO: detrmine where we can call defer database.Close()
-	return func(c *gin.Context) {
-		c.Set("DB", database)
-		c.Next()
-	}
-}
@@ -24,3 +24,10 @@ type DependencyMissingError string
 func (str DependencyMissingError) Error() string {
 	return fmt.Sprintf("DependencyMissingError: %q", string(str))
 }
+
+// Raised when the notification system is incorrectly configured
+type NotificationValidationError string
+
+func (str NotificationValidationError) Error() string {
+	return fmt.Sprintf("NotificationValidationError: %q", string(str))
+}
@@ -23,9 +23,9 @@ type SmartInfo struct {
 	ModelName    string `json:"model_name"`
 	SerialNumber string `json:"serial_number"`
 	Wwn          struct {
-		Naa int   `json:"naa"`
-		Oui int   `json:"oui"`
-		ID  int64 `json:"id"`
+		Naa uint64 `json:"naa"`
+		Oui uint64 `json:"oui"`
+		ID  uint64 `json:"id"`
 	} `json:"wwn"`
 	FirmwareVersion string `json:"firmware_version"`
 	UserCapacity    struct {
@@ -34,8 +34,8 @@ type Device struct {
 	Capacity       int64   `json:"capacity"`
 	FormFactor     string  `json:"form_factor"`
 	SmartSupport   bool    `json:"smart_support"`
-	DeviceProtocol string  `json:"device_protocol"`
-	DeviceType     string  `json:"device_type"` //device type is used for querying with -d/t flag
+	DeviceProtocol string  `json:"device_protocol"` //protocol determines which smart attribute types are available (ATA, NVMe, SCSI)
+	DeviceType     string  `json:"device_type"`     //device type is used for querying with -d/t flag, should only be used by collector.
 	SmartResults   []Smart `gorm:"foreignkey:DeviceWWN" json:"smart_results"`
 }

@@ -152,16 +152,6 @@ func (dv *Device) ApplyMetadataRules() error {
 }

 func (dv *Device) UpdateFromCollectorSmartInfo(info collector.SmartInfo) error {
-	dv.InterfaceSpeed = info.InterfaceSpeed.Current.String
 	dv.Firmware = info.FirmwareVersion
-	dv.RotationSpeed = info.RotationRate
-	dv.Capacity = info.UserCapacity.Bytes
-	dv.FormFactor = info.FormFactor.Name
-	dv.DeviceProtocol = info.Device.Protocol
-	dv.DeviceType = info.Device.Type
-	if len(info.Vendor) > 0 {
-		dv.Manufacturer = info.Vendor
-	}
-
 	return nil
 }
@@ -0,0 +1,105 @@
+{
+  "json_format_version": [
+    1,
+    0
+  ],
+  "smartctl": {
+    "version": [
+      7,
+      0
+    ],
+    "svn_revision": "4883",
+    "platform_info": "x86_64-linux-4.19.107-Unraid",
+    "build_info": "(local build)",
+    "argv": [
+      "smartctl",
+      "-a",
+      "-j",
+      "-d",
+      "nvme",
+      "/dev/nvme0"
+    ],
+    "exit_status": 0
+  },
+  "device": {
+    "name": "/dev/nvme0",
+    "info_name": "/dev/nvme0",
+    "type": "nvme",
+    "protocol": "NVMe"
+  },
+  "model_name": "Force MP510",
+  "serial_number": "yes",
+  "firmware_version": "ECFM12.3",
+  "nvme_pci_vendor": {
+    "id": 6535,
+    "subsystem_id": 6535
+  },
+  "nvme_ieee_oui_identifier": 6584743,
+  "nvme_total_capacity": 480103981056,
+  "nvme_unallocated_capacity": 0,
+  "nvme_controller_id": 1,
+  "nvme_number_of_namespaces": 1,
+  "nvme_namespaces": [
+    {
+      "id": 1,
+      "size": {
+        "blocks": 937703088,
+        "bytes": 480103981056
+      },
+      "capacity": {
+        "blocks": 937703088,
+        "bytes": 480103981056
+      },
+      "utilization": {
+        "blocks": 937703088,
+        "bytes": 480103981056
+      },
+      "formatted_lba_size": 512,
+      "eui64": {
+        "oui": 6584743,
+        "ext_id": 171819811633
+      }
+    }
+  ],
+  "user_capacity": {
+    "blocks": 937703088,
+    "bytes": 480103981056
+  },
+  "logical_block_size": 512,
+  "local_time": {
+    "time_t": 1600619090,
+    "asctime": "Sun Sep 20 16:24:50 2020 Europe"
+  },
+  "smart_status": {
+    "passed": true,
+    "nvme": {
+      "value": 0
+    }
+  },
+  "nvme_smart_health_information_log": {
+    "critical_warning": 0,
+    "temperature": 38,
+    "available_spare": 100,
+    "available_spare_threshold": 5,
+    "percentage_used": 1,
+    "data_units_read": 6932144,
+    "data_units_written": 16093122,
+    "host_reads": 29878811,
+    "host_writes": 17533252,
+    "controller_busy_time": 305,
+    "power_cycles": 4,
+    "power_on_hours": 6487,
+    "unsafe_shutdowns": 4,
+    "media_errors": 0,
+    "num_err_log_entries": 8382,
+    "warning_temp_time": 0,
+    "critical_comp_time": 0
+  },
+  "temperature": {
+    "current": 38
+  },
+  "power_cycle_count": 4,
+  "power_on_time": {
+    "hours": 6487
+  }
+}
@@ -0,0 +1,161 @@
+package notify
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"github.com/analogj/go-util/utils"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/config"
+	"github.com/containrrr/shoutrrr"
+	log "github.com/sirupsen/logrus"
+	"net/http"
+	"os"
+	"strings"
+	"sync"
+	"time"
+)
+
+type Payload struct {
+	Mailer       string `json:"mailer"`
+	Subject      string `json:"subject"`
+	Date         string `json:"date"`
+	FailureType  string `json:"failure_type"`
+	Device       string `json:"device"`
+	DeviceType   string `json:"device_type"`
+	DeviceString string `json:"device_string"`
+	Message      string `json:"message"`
+}
+
+type Notify struct {
+	Config  config.Interface
+	Payload Payload
+}
+
+func (n *Notify) Send() error {
+	//validate that the Payload is populated
+	sendDate := time.Now()
+	n.Payload.Date = sendDate.Format(time.RFC3339)
+
+	//retrieve list of notification endpoints from config file
+	configUrls := n.Config.GetStringSlice("notify.urls")
+
+	//remove http:// https:// and script:// prefixed urls
+	notifyWebhooks := []string{}
+	notifyScripts := []string{}
+	notifyShoutrrr := []string{}
+
+	for _, url := range configUrls {
+		if strings.HasPrefix(url, "https://") || strings.HasPrefix(url, "http://") {
+			notifyWebhooks = append(notifyWebhooks, url)
+		} else if strings.HasPrefix(url, "script://") {
+			notifyScripts = append(notifyScripts, url)
+		} else {
+			notifyShoutrrr = append(notifyShoutrrr, url)
+		}
+	}
+
+	//run all scripts, webhooks and shoutrr commands in parallel
+	var wg sync.WaitGroup
+
+	for _, notifyWebhook := range notifyWebhooks {
+		// execute collection in parallel go-routines
+		wg.Add(1)
+		go n.SendWebhookNotification(&wg, notifyWebhook)
+	}
+	for _, notifyScript := range notifyScripts {
+		// execute collection in parallel go-routines
+		wg.Add(1)
+		go n.SendScriptNotification(&wg, notifyScript)
+	}
+	if len(notifyScripts) > 0 {
+		wg.Add(1)
+		go n.SendShoutrrrNotification(&wg, notifyShoutrrr)
+	}
+
+	//and wait for completion, error or timeout.
+	if waitTimeout(&wg, time.Minute) { //wait for 1 minute
+		fmt.Println("Timed out while sending notifications")
+	} else {
+		fmt.Println("Sent notifications. Check logs for more information.")
+	}
+	return nil
+}
+
+func (n *Notify) SendWebhookNotification(wg *sync.WaitGroup, webhookUrl string) {
+	defer wg.Done()
+	log.Infof("Sending Webhook to %s", webhookUrl)
+	requestBody, err := json.Marshal(n.Payload)
+	if err != nil {
+		log.Errorf("An error occurred while sending Webhook to %s: %v", webhookUrl, err)
+		return
+	}
+
+	resp, err := http.Post(webhookUrl, "application/json", bytes.NewBuffer(requestBody))
+	if err != nil {
+		log.Errorf("An error occurred while sending Webhook to %s: %v", webhookUrl, err)
+		return
+	}
+	defer resp.Body.Close()
+	//we don't care about resp body content, but maybe we should log it?
+}
+
+func (n *Notify) SendScriptNotification(wg *sync.WaitGroup, scriptUrl string) {
+	defer wg.Done()
+
+	//check if the script exists.
+	scriptPath := strings.TrimPrefix(scriptUrl, "script://")
+	log.Infof("Executing Script %s", scriptPath)
+
+	if !utils.FileExists(scriptPath) {
+		log.Errorf("Script does not exist: %s", scriptPath)
+		return
+	}
+
+	copyEnv := os.Environ()
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_MAILER=%s", n.Payload.Mailer))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_SUBJECT=%s", n.Payload.Subject))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_DATE=%s", n.Payload.Date))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_FAILURE_TYPE=%s", n.Payload.FailureType))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_DEVICE=%s", n.Payload.Device))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_DEVICE_TYPE=%s", n.Payload.DeviceType))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_DEVICE_STRING=%s", n.Payload.DeviceString))
+	copyEnv = append(copyEnv, fmt.Sprintf("SCRUTINY_MESSAGE=%s", n.Payload.Message))
+	err := utils.CmdExec(scriptPath, []string{}, "", copyEnv, "")
+	if err != nil {
+		log.Errorf("An error occurred while executing script %s: %v", scriptPath, err)
+	}
+	return
+}
+
+func (n *Notify) SendShoutrrrNotification(wg *sync.WaitGroup, shoutrrrUrls []string) {
+	log.Infof("Sending notifications to %v", shoutrrrUrls)
+
+	defer wg.Done()
+	sender, err := shoutrrr.CreateSender(shoutrrrUrls...)
+	if err != nil {
+		log.Errorf("An error occurred while sending notifications %v: %v", shoutrrrUrls, err)
+		return
+	}
+
+	errs := sender.Send(n.Payload.Subject, nil) //structs.Map(n.Payload).())
+	if len(errs) > 0 {
+		log.Errorf("One or more errors occurred  occurred while sending notifications %v:\n %v", shoutrrrUrls, errs)
+	}
+}
+
+//utility functions
+// waitTimeout waits for the waitgroup for the specified max timeout.
+// Returns true if waiting timed out.
+func waitTimeout(wg *sync.WaitGroup, timeout time.Duration) bool {
+	c := make(chan struct{})
+	go func() {
+		defer close(c)
+		wg.Wait()
+	}()
+	select {
+	case <-c:
+		return false // completed normally
+	case <-time.After(timeout):
+		return true // timed out
+	}
+}
@@ -2,4 +2,4 @@ package version

 // VERSION is the app-global version string, which will be replaced with a
 // new value during packaging
-const VERSION = "0.1.13"
+const VERSION = "0.2.1"
@@ -5,25 +5,40 @@ import (
 	dbModels "github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
 	"github.com/gin-gonic/gin"
 	"github.com/jinzhu/gorm"
+	"github.com/sirupsen/logrus"
 	"net/http"
 )

 func GetDeviceDetails(c *gin.Context) {
 	db := c.MustGet("DB").(*gorm.DB)
+	logger := c.MustGet("LOGGER").(logrus.FieldLogger)
 	device := dbModels.Device{}

-	db.Debug().
-		Preload("SmartResults", func(db *gorm.DB) *gorm.DB {
-			return db.Order("smarts.created_at DESC").Limit(40)
-		}).
+	if err := db.Preload("SmartResults", func(db *gorm.DB) *gorm.DB {
+		return db.Order("smarts.created_at DESC").Limit(40)
+	}).
 		Preload("SmartResults.AtaAttributes").
 		Preload("SmartResults.NvmeAttributes").
 		Preload("SmartResults.ScsiAttributes").
 		Where("wwn = ?", c.Param("wwn")).
-		First(&device)
+		First(&device).Error; err != nil {

-	device.SquashHistory()
-	device.ApplyMetadataRules()
+		logger.Errorln("An error occurred while retrieving device details", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
+	}
+
+	if err := device.SquashHistory(); err != nil {
+		logger.Errorln("An error occurred while squashing device history", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
+	}
+
+	if err := device.ApplyMetadataRules(); err != nil {
+		logger.Errorln("An error occurred while applying scrutiny thresholds & rules", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
+	}

 	var deviceMetadata interface{}
 	if device.IsAta() {
@@ -4,20 +4,25 @@ import (
 	dbModels "github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
 	"github.com/gin-gonic/gin"
 	"github.com/jinzhu/gorm"
+	"github.com/sirupsen/logrus"
 	"net/http"
 )

 func GetDevicesSummary(c *gin.Context) {
 	db := c.MustGet("DB").(*gorm.DB)
+	logger := c.MustGet("LOGGER").(logrus.FieldLogger)
 	devices := []dbModels.Device{}

 	//We need the last x (for now all) Smart objects for each Device, so that we can graph Temperature
 	//We also need the last
-	db.Debug().
-		Preload("SmartResults", func(db *gorm.DB) *gorm.DB {
-			return db.Order("smarts.created_at DESC") //OLD: .Limit(devicesCount)
-		}).
-		Find(&devices)
+	if err := db.Preload("SmartResults", func(db *gorm.DB) *gorm.DB {
+		return db.Order("smarts.created_at DESC") //OLD: .Limit(devicesCount)
+	}).
+		Find(&devices).Error; err != nil {
+		logger.Errorln("Could not get device summary from DB", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
+	}

 	c.JSON(http.StatusOK, gin.H{
 		"success": true,
@@ -4,36 +4,43 @@ import (
 	dbModels "github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
 	"github.com/gin-gonic/gin"
 	"github.com/jinzhu/gorm"
-	log "github.com/sirupsen/logrus"
+	"github.com/sirupsen/logrus"
 	"net/http"
 )

 // filter devices that are detected by various collectors.
 func RegisterDevices(c *gin.Context) {
 	db := c.MustGet("DB").(*gorm.DB)
+	logger := c.MustGet("LOGGER").(logrus.FieldLogger)

 	var collectorDeviceWrapper dbModels.DeviceWrapper
 	err := c.BindJSON(&collectorDeviceWrapper)
 	if err != nil {
-		log.Error("Cannot parse detected devices")
-		c.JSON(http.StatusOK, gin.H{"success": false})
+		logger.Errorln("Cannot parse detected devices", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
 	}

 	//TODO: filter devices here (remove excludes, force includes)
-
+	errs := []error{}
 	for _, dev := range collectorDeviceWrapper.Data {
 		//insert devices into DB if not already there.
-		db.Where(dbModels.Device{WWN: dev.WWN}).FirstOrCreate(&dev)
+		if err := db.Where(dbModels.Device{WWN: dev.WWN}).FirstOrCreate(&dev).Error; err != nil {
+			errs = append(errs, err)
+		}
 	}

-	if err != nil {
-		c.JSON(http.StatusOK, gin.H{
+	if len(errs) > 0 {
+		logger.Errorln("An error occurred while registering devices", errs)
+		c.JSON(http.StatusInternalServerError, gin.H{
 			"success": false,
 		})
+		return
 	} else {
 		c.JSON(http.StatusOK, dbModels.DeviceWrapper{
 			Success: true,
 			Data:    collectorDeviceWrapper.Data,
 		})
+		return
 	}
 }
@@ -0,0 +1,42 @@
+package handler
+
+import (
+	"fmt"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/config"
+	dbModels "github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/notify"
+	"github.com/gin-gonic/gin"
+	"github.com/sirupsen/logrus"
+	"net/http"
+	"os"
+)
+
+// Send test notification
+func SendTestNotification(c *gin.Context) {
+	appConfig := c.MustGet("CONFIG").(config.Interface)
+	logger := c.MustGet("LOGGER").(logrus.FieldLogger)
+
+	testNotify := notify.Notify{
+		Config: appConfig,
+		Payload: notify.Payload{
+			Mailer:       os.Args[0],
+			Subject:      fmt.Sprintf("Scrutiny SMART error (EmailTest) detected on disk: XXXXX"),
+			FailureType:  "EmailTest",
+			Device:       "/dev/sda",
+			DeviceType:   "ata",
+			DeviceString: "/dev/sda",
+			Message:      "TEST EMAIL from smartd for device: /dev/sda",
+		},
+	}
+	err := testNotify.Send()
+	if err != nil {
+		logger.Errorln("An error occurred while sending test notification", err)
+		c.JSON(http.StatusInternalServerError, gin.H{
+			"success": false,
+		})
+	} else {
+		c.JSON(http.StatusOK, dbModels.DeviceWrapper{
+			Success: true,
+		})
+	}
+}
@@ -5,36 +5,45 @@ import (
 	dbModels "github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
 	"github.com/gin-gonic/gin"
 	"github.com/jinzhu/gorm"
-	log "github.com/sirupsen/logrus"
+	"github.com/sirupsen/logrus"
 	"net/http"
 )

 func UploadDeviceMetrics(c *gin.Context) {
 	db := c.MustGet("DB").(*gorm.DB)
+	logger := c.MustGet("LOGGER").(logrus.FieldLogger)

 	var collectorSmartData collector.SmartInfo
 	err := c.BindJSON(&collectorSmartData)
 	if err != nil {
-		//TODO: cannot parse smart data
-		log.Error("Cannot parse SMART data")
-		c.JSON(http.StatusOK, gin.H{"success": false})
-
+		logger.Errorln("Cannot parse SMART data", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
 	}

 	//update the device information if necessary
 	var device dbModels.Device
 	db.Where("wwn = ?", c.Param("wwn")).First(&device)
 	device.UpdateFromCollectorSmartInfo(collectorSmartData)
-	db.Model(&device).Updates(device)
+	if err := db.Model(&device).Updates(device).Error; err != nil {
+		logger.Errorln("An error occurred while updating device data from smartctl metrics", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
+	}

 	// insert smart info
 	deviceSmartData := dbModels.Smart{}
 	err = deviceSmartData.FromCollectorSmartInfo(c.Param("wwn"), collectorSmartData)
 	if err != nil {
-		c.JSON(http.StatusOK, gin.H{"success": false})
+		logger.Errorln("Could not process SMART metrics", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
+		return
+	}
+	if err := db.Create(&deviceSmartData).Error; err != nil {
+		logger.Errorln("An error occurred while saving smartctl metrics", err)
+		c.JSON(http.StatusInternalServerError, gin.H{"success": false})
 		return
 	}
-	db.Create(&deviceSmartData)

 	c.JSON(http.StatusOK, gin.H{"success": true})
 }
@@ -0,0 +1,13 @@
+package middleware
+
+import (
+	"github.com/analogj/scrutiny/webapp/backend/pkg/config"
+	"github.com/gin-gonic/gin"
+)
+
+func ConfigMiddleware(appConfig config.Interface) gin.HandlerFunc {
+	return func(c *gin.Context) {
+		c.Set("CONFIG", appConfig)
+		c.Next()
+	}
+}
@@ -0,0 +1,122 @@
+package middleware
+
+import (
+	"bytes"
+	"fmt"
+	"github.com/gin-gonic/gin"
+	"github.com/sirupsen/logrus"
+	"io"
+	"io/ioutil"
+	"math"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+)
+
+// Middleware based on https://github.com/toorop/gin-logrus/blob/master/logger.go
+// Body recording based on
+// - https://github.com/gin-gonic/gin/issues/1363
+// - https://stackoverflow.com/questions/38501325/how-to-log-response-body-in-gin
+
+// 2016-09-27 09:38:21.541541811 +0200 CEST
+// 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700]
+// "GET /apache_pb.gif HTTP/1.0" 200 2326
+// "http://www.example.com/start.html"
+// "Mozilla/4.08 [en] (Win98; I ;Nav)"
+
+var timeFormat = "02/Jan/2006:15:04:05 -0700"
+
+// Logger is the logrus logger handler
+func LoggerMiddleware(logger logrus.FieldLogger) gin.HandlerFunc {
+
+	hostname, err := os.Hostname()
+	if err != nil {
+		hostname = "unknow"
+	}
+
+	return func(c *gin.Context) {
+
+		//clone the request body reader.
+		var reqBody string
+		if c.Request.Body != nil {
+			buf, _ := ioutil.ReadAll(c.Request.Body)
+			reqBodyReader1 := ioutil.NopCloser(bytes.NewBuffer(buf))
+			reqBodyReader2 := ioutil.NopCloser(bytes.NewBuffer(buf)) //We have to create a new Buffer, because reqBodyReader1 will be read.
+			c.Request.Body = reqBodyReader2
+			reqBody = readBody(reqBodyReader1)
+		}
+
+		// other handler can change c.Path so:
+		path := c.Request.URL.Path
+		blw := &responseBodyLogWriter{body: &bytes.Buffer{}, ResponseWriter: c.Writer}
+		c.Writer = blw
+		c.Set("LOGGER", logger)
+		start := time.Now()
+		c.Next()
+		stop := time.Since(start)
+		latency := int(math.Ceil(float64(stop.Nanoseconds()) / 1000000.0))
+		statusCode := c.Writer.Status()
+		clientIP := c.ClientIP()
+		clientUserAgent := c.Request.UserAgent()
+		referer := c.Request.Referer()
+		respLength := c.Writer.Size()
+		if respLength < 0 {
+			respLength = 0
+		}
+
+		entry := logger.WithFields(logrus.Fields{
+			"hostname":   hostname,
+			"statusCode": statusCode,
+			"latency":    latency, // time to process
+			"clientIP":   clientIP,
+			"method":     c.Request.Method,
+			"path":       path,
+			"referer":    referer,
+			"respLength": respLength,
+			"userAgent":  clientUserAgent,
+		})
+
+		if len(c.Errors) > 0 {
+			entry.Error(c.Errors.ByType(gin.ErrorTypePrivate).String())
+		} else {
+			msg := fmt.Sprintf("%s - %s [%s] \"%s %s\" %d %d \"%s\" \"%s\" (%dms)", clientIP, hostname, time.Now().Format(timeFormat), c.Request.Method, path, statusCode, respLength, referer, clientUserAgent, latency)
+			if statusCode >= http.StatusInternalServerError {
+				entry.Error(msg)
+			} else if statusCode >= http.StatusBadRequest {
+				entry.Warn(msg)
+			} else {
+				entry.Info(msg)
+			}
+		}
+		if strings.HasPrefix(path, "/api/") {
+			//only debug log request/response from api endpoint.
+			if len(reqBody) > 0 {
+				entry.WithField("bodyType", "request").Debugln(reqBody) // Print request body
+			}
+			entry.WithField("bodyType", "response").Debugln(blw.body.String())
+		}
+	}
+}
+
+// Response Logging
+
+type responseBodyLogWriter struct {
+	gin.ResponseWriter
+	body *bytes.Buffer
+}
+
+func (w responseBodyLogWriter) Write(b []byte) (int, error) {
+	w.body.Write(b)
+	return w.ResponseWriter.Write(b)
+}
+
+// Request Logging
+
+func readBody(reader io.Reader) string {
+	buf := new(bytes.Buffer)
+	buf.ReadFrom(reader)
+
+	s := buf.String()
+	return s
+}
@@ -0,0 +1,55 @@
+package middleware
+
+import (
+	"fmt"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/config"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
+	"github.com/gin-gonic/gin"
+	"github.com/jinzhu/gorm"
+	_ "github.com/jinzhu/gorm/dialects/sqlite"
+	"github.com/sirupsen/logrus"
+)
+
+func DatabaseMiddleware(appConfig config.Interface, logger logrus.FieldLogger) gin.HandlerFunc {
+	//var database *gorm.DB
+	fmt.Printf("Trying to connect to database stored: %s\n", appConfig.GetString("web.database.location"))
+	database, err := gorm.Open("sqlite3", appConfig.GetString("web.database.location"))
+	if err != nil {
+		panic("Failed to connect to database!")
+	}
+
+	database.SetLogger(&GormLogger{Logger: logger})
+	database.AutoMigrate(&db.Device{})
+	database.AutoMigrate(&db.SelfTest{})
+	database.AutoMigrate(&db.Smart{})
+	database.AutoMigrate(&db.SmartAtaAttribute{})
+	database.AutoMigrate(&db.SmartNvmeAttribute{})
+	database.AutoMigrate(&db.SmartScsiAttribute{})
+
+	//TODO: detrmine where we can call defer database.Close()
+	return func(c *gin.Context) {
+		c.Set("DB", database)
+		c.Next()
+	}
+}
+
+// GormLogger is a custom logger for Gorm, making it use logrus.
+type GormLogger struct{ Logger logrus.FieldLogger }
+
+// Print handles log events from Gorm for the custom logger.
+func (gl *GormLogger) Print(v ...interface{}) {
+	switch v[0] {
+	case "sql":
+		gl.Logger.WithFields(
+			logrus.Fields{
+				"module":  "gorm",
+				"type":    "sql",
+				"rows":    v[5],
+				"src_ref": v[1],
+				"values":  v[4],
+			},
+		).Debug(v[3])
+	case "log":
+		gl.Logger.WithFields(logrus.Fields{"module": "gorm", "type": "log"}).Print(v[2])
+	}
+}
@@ -3,20 +3,26 @@ package web
 import (
 	"fmt"
 	"github.com/analogj/scrutiny/webapp/backend/pkg/config"
-	"github.com/analogj/scrutiny/webapp/backend/pkg/database"
 	"github.com/analogj/scrutiny/webapp/backend/pkg/web/handler"
+	"github.com/analogj/scrutiny/webapp/backend/pkg/web/middleware"
 	"github.com/gin-gonic/gin"
+	"github.com/sirupsen/logrus"
+	"io"
 	"net/http"
+	"os"
 )

 type AppEngine struct {
 	Config config.Interface
 }

-func (ae *AppEngine) Setup() *gin.Engine {
-	r := gin.Default()
+func (ae *AppEngine) Setup(logger logrus.FieldLogger) *gin.Engine {
+	r := gin.New()

-	r.Use(database.DatabaseHandler(ae.Config.GetString("web.database.location")))
+	r.Use(middleware.LoggerMiddleware(logger))
+	r.Use(middleware.DatabaseMiddleware(ae.Config, logger))
+	r.Use(middleware.ConfigMiddleware(ae.Config))
+	r.Use(gin.Recovery())

 	api := r.Group("/api")
 	{
@@ -25,13 +31,13 @@ func (ae *AppEngine) Setup() *gin.Engine {
 				"success": true,
 			})
 		})
+		api.POST("/health/notify", handler.SendTestNotification) //check if notifications are configured correctly

-		api.POST("/devices/register", handler.RegisterDevices)
-		api.GET("/summary", handler.GetDevicesSummary)
-		api.POST("/device/:wwn/smart", handler.UploadDeviceMetrics)
+		api.POST("/devices/register", handler.RegisterDevices)      //used by Collector to register new devices and retrieve filtered list
+		api.GET("/summary", handler.GetDevicesSummary)              //used by Dashboard
+		api.POST("/device/:wwn/smart", handler.UploadDeviceMetrics) //used by Collector to upload data
 		api.POST("/device/:wwn/selftest", handler.UploadDeviceSelfTests)
-
-		api.GET("/device/:wwn/details", handler.GetDeviceDetails)
+		api.GET("/device/:wwn/details", handler.GetDeviceDetails) //used by Details
 	}

 	//Static request routing
@@ -50,7 +56,28 @@ func (ae *AppEngine) Setup() *gin.Engine {
 }

 func (ae *AppEngine) Start() error {
-	r := ae.Setup()
+
+	logger := logrus.New()
+	//set default log level
+	logLevel, err := logrus.ParseLevel(ae.Config.GetString("log.level"))
+	if err != nil {
+		return err
+	}
+	logger.SetLevel(logLevel)
+	//set the log file if present
+	if len(ae.Config.GetString("log.file")) != 0 {
+		logFile, err := os.OpenFile(ae.Config.GetString("log.file"), os.O_CREATE|os.O_WRONLY, 0644)
+		defer logFile.Close()
+		if err != nil {
+			logrus.Errorf("Failed to open log file %s for output: %s", ae.Config.GetString("log.file"), err)
+			return err
+		}
+
+		//configure the logrus default
+		logger.SetOutput(io.MultiWriter(os.Stderr, logFile))
+	}
+
+	r := ae.Setup(logger)

 	return r.Run(fmt.Sprintf("%s:%s", ae.Config.GetString("web.listen.host"), ae.Config.GetString("web.listen.port")))
 }
@@ -1,29 +1,37 @@
 package web_test

 import (
+	"encoding/json"
 	mock_config "github.com/analogj/scrutiny/webapp/backend/pkg/config/mock"
+	dbModels "github.com/analogj/scrutiny/webapp/backend/pkg/models/db"
 	"github.com/analogj/scrutiny/webapp/backend/pkg/web"
 	"github.com/golang/mock/gomock"
+	"github.com/sirupsen/logrus"
 	"github.com/stretchr/testify/require"
+	"io/ioutil"
 	"net/http"
 	"net/http/httptest"
 	"os"
+	"path"
+	"strings"
 	"testing"
 )

 func TestHealthRoute(t *testing.T) {
 	//setup
+	parentPath, _ := ioutil.TempDir("", "")
+	defer os.RemoveAll(parentPath)
 	mockCtrl := gomock.NewController(t)
 	defer mockCtrl.Finish()
 	fakeConfig := mock_config.NewMockInterface(mockCtrl)
-	fakeConfig.EXPECT().GetString("web.database.location").Return("testdata/scrutiny_test.db")
-	fakeConfig.EXPECT().GetString("web.src.frontend.path").Return("testdata")
+	fakeConfig.EXPECT().GetString("web.database.location").Return(path.Join(parentPath, "scrutiny_test.db")).AnyTimes()
+	fakeConfig.EXPECT().GetString("web.src.frontend.path").Return(parentPath).AnyTimes()

 	ae := web.AppEngine{
 		Config: fakeConfig,
 	}

-	router := ae.Setup()
+	router := ae.Setup(logrus.New())

 	//test
 	w := httptest.NewRecorder()
@@ -37,15 +45,17 @@ func TestHealthRoute(t *testing.T) {

 func TestRegisterDevicesRoute(t *testing.T) {
 	//setup
+	parentPath, _ := ioutil.TempDir("", "")
+	defer os.RemoveAll(parentPath)
 	mockCtrl := gomock.NewController(t)
 	defer mockCtrl.Finish()
 	fakeConfig := mock_config.NewMockInterface(mockCtrl)
-	fakeConfig.EXPECT().GetString("web.database.location").Return("testdata/scrutiny_test.db")
-	fakeConfig.EXPECT().GetString("web.src.frontend.path").Return("testdata")
+	fakeConfig.EXPECT().GetString("web.database.location").Return(path.Join(parentPath, "scrutiny_test.db")).AnyTimes()
+	fakeConfig.EXPECT().GetString("web.src.frontend.path").Return(parentPath).AnyTimes()
 	ae := web.AppEngine{
 		Config: fakeConfig,
 	}
-	router := ae.Setup()
+	router := ae.Setup(logrus.New())
 	file, err := os.Open("testdata/register-devices-req.json")
 	require.NoError(t, err)

@@ -60,15 +70,17 @@ func TestRegisterDevicesRoute(t *testing.T) {

 func TestUploadDeviceMetricsRoute(t *testing.T) {
 	//setup
+	parentPath, _ := ioutil.TempDir("", "")
+	defer os.RemoveAll(parentPath)
 	mockCtrl := gomock.NewController(t)
 	defer mockCtrl.Finish()
 	fakeConfig := mock_config.NewMockInterface(mockCtrl)
-	fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return("testdata/scrutiny_test.db")
-	fakeConfig.EXPECT().GetString("web.src.frontend.path").AnyTimes().Return("testdata")
+	fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return(path.Join(parentPath, "scrutiny_test.db"))
+	fakeConfig.EXPECT().GetString("web.src.frontend.path").AnyTimes().Return(parentPath)
 	ae := web.AppEngine{
 		Config: fakeConfig,
 	}
-	router := ae.Setup()
+	router := ae.Setup(logrus.New())
 	devicesfile, err := os.Open("testdata/register-devices-single-req.json")
 	require.NoError(t, err)

@@ -91,15 +103,18 @@ func TestUploadDeviceMetricsRoute(t *testing.T) {

 func TestPopulateMultiple(t *testing.T) {
 	//setup
+	parentPath, _ := ioutil.TempDir("", "")
+	defer os.RemoveAll(parentPath)
 	mockCtrl := gomock.NewController(t)
 	defer mockCtrl.Finish()
 	fakeConfig := mock_config.NewMockInterface(mockCtrl)
-	fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return("testdata/scrutiny_test.db")
-	fakeConfig.EXPECT().GetString("web.src.frontend.path").AnyTimes().Return("testdata")
+	//fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return("testdata/scrutiny_test.db")
+	fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return(path.Join(parentPath, "scrutiny_test.db"))
+	fakeConfig.EXPECT().GetString("web.src.frontend.path").AnyTimes().Return(parentPath)
 	ae := web.AppEngine{
 		Config: fakeConfig,
 	}
-	router := ae.Setup()
+	router := ae.Setup(logrus.New())
 	devicesfile, err := os.Open("testdata/register-devices-req.json")
 	require.NoError(t, err)

@@ -147,3 +162,69 @@ func TestPopulateMultiple(t *testing.T) {

 	//assert
 }
+
+func TestSendTestNotificationRoute(t *testing.T) {
+	//setup
+	parentPath, _ := ioutil.TempDir("", "")
+	defer os.RemoveAll(parentPath)
+	mockCtrl := gomock.NewController(t)
+	defer mockCtrl.Finish()
+	fakeConfig := mock_config.NewMockInterface(mockCtrl)
+	fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return(path.Join(parentPath, "scrutiny_test.db"))
+	fakeConfig.EXPECT().GetString("web.src.frontend.path").AnyTimes().Return(parentPath)
+	fakeConfig.EXPECT().GetStringSlice("notify.urls").AnyTimes().Return([]string{"https://scrutiny.requestcatcher.com/test"})
+	ae := web.AppEngine{
+		Config: fakeConfig,
+	}
+	router := ae.Setup(logrus.New())
+
+	//test
+	wr := httptest.NewRecorder()
+	req, _ := http.NewRequest("POST", "/api/health/notify", strings.NewReader("{}"))
+	router.ServeHTTP(wr, req)
+
+	//assert
+	require.Equal(t, 200, wr.Code)
+}
+
+func TestGetDevicesSummaryRoute_Nvme(t *testing.T) {
+	//setup
+	parentPath, _ := ioutil.TempDir("", "")
+	defer os.RemoveAll(parentPath)
+	mockCtrl := gomock.NewController(t)
+	defer mockCtrl.Finish()
+	fakeConfig := mock_config.NewMockInterface(mockCtrl)
+	fakeConfig.EXPECT().GetString("web.database.location").AnyTimes().Return(path.Join(parentPath, "scrutiny_test.db"))
+	fakeConfig.EXPECT().GetString("web.src.frontend.path").AnyTimes().Return(parentPath)
+	ae := web.AppEngine{
+		Config: fakeConfig,
+	}
+	router := ae.Setup(logrus.New())
+	devicesfile, err := os.Open("testdata/register-devices-req-2.json")
+	require.NoError(t, err)
+
+	metricsfile, err := os.Open("../models/testdata/smart-nvme2.json")
+	require.NoError(t, err)
+
+	//test
+	wr := httptest.NewRecorder()
+	req, _ := http.NewRequest("POST", "/api/devices/register", devicesfile)
+	router.ServeHTTP(wr, req)
+	require.Equal(t, 200, wr.Code)
+
+	mr := httptest.NewRecorder()
+	req, _ = http.NewRequest("POST", "/api/device/a4c8e8ed-11a0-4c97-9bba-306440f1b944/smart", metricsfile)
+	router.ServeHTTP(mr, req)
+	require.Equal(t, 200, mr.Code)
+
+	sr := httptest.NewRecorder()
+	req, _ = http.NewRequest("GET", "/api/summary", nil)
+	router.ServeHTTP(sr, req)
+	require.Equal(t, 200, sr.Code)
+	var device dbModels.DeviceWrapper
+	json.Unmarshal(sr.Body.Bytes(), &device)
+
+	//assert
+	require.Equal(t, "a4c8e8ed-11a0-4c97-9bba-306440f1b944", device.Data[0].WWN)
+	require.Equal(t, "passed", device.Data[0].SmartResults[0].SmartStatus)
+}
@@ -0,0 +1,20 @@
+{
+  "data": [
+    {
+      "wwn": "a4c8e8ed-11a0-4c97-9bba-306440f1b944",
+      "device_name": "nvme0",
+      "manufacturer": "",
+      "model_name": "Force MP510",
+      "interface_type": "",
+      "interface_speed": "",
+      "serial_number": "a4c8e8ed-11a0-4c97-9bba-306440f1b944",
+      "firmware": "ECFM12.3",
+      "rotational_speed": 0,
+      "capacity": 480103981056,
+      "form_factor": "",
+      "smart_support": false,
+      "device_protocol": "NVMe",
+      "device_type": "nvme"
+    }
+  ]
+}
@@ -26,8 +26,8 @@ export const appRoutes: Route[] = [
        children   : [

            // Example
-            {path: 'dashboard', loadChildren: () => import('app/modules/admin/dashboard/dashboard.module').then(m => m.DashboardModule)},
-            {path: 'device/:wwn', loadChildren: () => import('app/modules/admin/detail/detail.module').then(m => m.DetailModule)}
+            {path: 'dashboard', loadChildren: () => import('app/modules/dashboard/dashboard.module').then(m => m.DashboardModule)},
+            {path: 'device/:wwn', loadChildren: () => import('app/modules/detail/detail.module').then(m => m.DetailModule)}

            // 404 & Catch all
            // {path: '404-not-found', pathMatch: 'full', loadChildren: () => import('app/modules/admin/pages/errors/error-404/error-404.module').then(m => m.Error404Module)},
@@ -2,7 +2,11 @@ import { Injectable } from '@angular/core';
 import * as _ from 'lodash';
 import { TreoMockApi } from '@treo/lib/mock-api/mock-api.interfaces';
 import { TreoMockApiService } from '@treo/lib/mock-api/mock-api.service';
-import { details as detailsData } from 'app/data/mock/device/details/data';
+import { sda } from 'app/data/mock/device/details/sda';
+import { sdb } from 'app/data/mock/device/details/sdb';
+import { sdc } from 'app/data/mock/device/details/sdc';
+import { sdd } from 'app/data/mock/device/details/sdd';
+import { sde } from 'app/data/mock/device/details/sde';

@Injectable({
    providedIn: 'root'
@@ -21,9 +25,6 @@ export class DetailsMockApi implements TreoMockApi
        private _treoMockApiService: TreoMockApiService
    )
    {
-        // Set the data
-        this._details = detailsData;
-
        // Register the API endpoints
        this.register();
    }
@@ -37,16 +38,53 @@ export class DetailsMockApi implements TreoMockApi
     */
    register(): void
    {
-        // -----------------------------------------------------------------------------------------------------
-        // @ Sales - GET
-        // -----------------------------------------------------------------------------------------------------
        this._treoMockApiService
-            .onGet('/api/device/:wwn/details')
+            .onGet('/api/device/0x5002538e40a22954/details')
            .reply(() => {

                return [
                    200,
-                    _.cloneDeep(this._details)
+                    _.cloneDeep(sda)
+                ];
+            });
+
+        this._treoMockApiService
+            .onGet('/api/device/0x5000cca264eb01d7/details')
+            .reply(() => {
+
+                return [
+                    200,
+                    _.cloneDeep(sdb)
+                ];
+            });
+
+        this._treoMockApiService
+            .onGet('/api/device/0x5000cca264ec3183/details')
+            .reply(() => {
+
+                return [
+                    200,
+                    _.cloneDeep(sdc)
+                ];
+            });
+
+        this._treoMockApiService
+            .onGet('/api/device/0x5000cca252c859cc/details')
+            .reply(() => {
+
+                return [
+                    200,
+                    _.cloneDeep(sdd)
+                ];
+            });
+
+        this._treoMockApiService
+            .onGet('/api/device/0x5000cca264ebc248/details')
+            .reply(() => {
+
+                return [
+                    200,
+                    _.cloneDeep(sde)
                ];
            });
    }
@@ -0,0 +1,77 @@
+<h2 mat-dialog-title>Scrutiny Settings</h2>
+<mat-dialog-content class="mat-typography">
+
+    <form class="flex flex-col p-8 pb-0 overflow-hidden">
+        <div class="flex flex-col gt-xs:flex-row">
+            <mat-form-field class="flex-auto gt-xs:pr-3">
+                <mat-label>Sort By</mat-label>
+                <mat-select [value]="'status'">
+                    <mat-option  value="status">Status</mat-option>
+                    <mat-option value="name" disabled>Name</mat-option>
+                    <mat-option value="label" disabled>Label</mat-option>
+                </mat-select>
+            </mat-form-field>
+        </div>
+
+        <div class="flex">
+            <mat-tab-group mat-align-tabs="start">
+                <mat-tab label="Ata">
+
+                    <div class="flex flex-col mt-5 gt-md:flex-row">
+                        <mat-form-field class="flex-auto gt-md:pr-3">
+                            <mat-label>Critical Error Threshold</mat-label>
+                            <input matInput [value]="'10%'">
+                        </mat-form-field>
+                        <mat-form-field class="flex-auto gt-md:pl-3">
+                            <mat-label>Critical Warning Threshold</mat-label>
+                            <input matInput>
+                        </mat-form-field>
+                    </div>
+
+                    <div class="flex flex-col gt-md:flex-row">
+                        <mat-form-field class="flex-auto gt-md:pr-3">
+                            <mat-label>Error Threshold</mat-label>
+                            <input matInput [value]="'20%'">
+                        </mat-form-field>
+                        <mat-form-field class="flex-auto gt-md:pl-3">
+                            <mat-label>Warning Threshold</mat-label>
+                            <input matInput [value]="'10%'">
+                        </mat-form-field>
+                    </div>
+
+                </mat-tab>
+                <mat-tab label="NVMe">
+
+                    <div class="flex flex-col mt-5 gt-md:flex-row">
+                        <mat-form-field class="flex-auto gt-md:pr-3">
+                            <mat-label>Critical Error Threshold</mat-label>
+                            <input matInput [value]="'enabled'">
+                        </mat-form-field>
+                        <mat-form-field class="flex-auto gt-md:pl-3">
+                            <mat-label>Critical Warning Threshold</mat-label>
+                            <input matInput>
+                        </mat-form-field>
+                    </div>
+
+                </mat-tab>
+                <mat-tab label="SCSI">
+                    <div class="flex flex-col mt-5 gt-md:flex-row">
+                        <mat-form-field class="flex-auto gt-md:pr-3">
+                            <mat-label>Critical Error Threshold</mat-label>
+                            <input matInput [value]="'enabled'">
+                        </mat-form-field>
+                        <mat-form-field class="flex-auto gt-md:pl-3">
+                            <mat-label>Critical Warning Threshold</mat-label>
+                            <input matInput>
+                        </mat-form-field>
+                    </div>
+                </mat-tab>
+            </mat-tab-group>
+        </div>
+    </form>
+
+</mat-dialog-content>
+<mat-dialog-actions align="end">
+    <button mat-button mat-dialog-close>Cancel</button>
+    <button mat-button matTooltip="not yet implemented" [mat-dialog-close]="true" cdkFocusInitial>Save</button>
+</mat-dialog-actions>
@@ -0,0 +1,25 @@
+import { async, ComponentFixture, TestBed } from '@angular/core/testing';
+
+import { DashboardSettingsComponent } from './dashboard-settings.component';
+
+describe('DashboardSettingsComponent', () => {
+  let component: DashboardSettingsComponent;
+  let fixture: ComponentFixture<DashboardSettingsComponent>;
+
+  beforeEach(async(() => {
+    TestBed.configureTestingModule({
+      declarations: [ DashboardSettingsComponent ]
+    })
+    .compileComponents();
+  }));
+
+  beforeEach(() => {
+    fixture = TestBed.createComponent(DashboardSettingsComponent);
+    component = fixture.componentInstance;
+    fixture.detectChanges();
+  });
+
+  it('should create', () => {
+    expect(component).toBeTruthy();
+  });
+});
@@ -0,0 +1,17 @@
+import { Component, OnInit } from '@angular/core';
+
+@Component({
+  selector: 'app-dashboard-settings',
+  templateUrl: './dashboard-settings.component.html',
+  styleUrls: ['./dashboard-settings.component.scss']
+})
+export class DashboardSettingsComponent implements OnInit {
+
+  constructor() { }
+
+  ngOnInit(): void {
+  }
+    formatLabel(value: number) {
+        return value;
+    }
+}
@@ -0,0 +1,46 @@
+import { NgModule } from '@angular/core';
+import { RouterModule } from '@angular/router';
+import { Overlay } from '@angular/cdk/overlay';
+import { MAT_AUTOCOMPLETE_SCROLL_STRATEGY, MatAutocompleteModule } from '@angular/material/autocomplete';
+import { MatButtonModule } from '@angular/material/button';
+import { MatSelectModule } from '@angular/material/select';
+import { MatFormFieldModule } from '@angular/material/form-field';
+import { MatIconModule } from '@angular/material/icon';
+import { MatInputModule } from '@angular/material/input';
+import { SharedModule } from 'app/shared/shared.module';
+import {DashboardSettingsComponent} from 'app/layout/common/dashboard-settings/dashboard-settings.component'
+import { MatDialogModule } from "@angular/material/dialog";
+import { MatButtonToggleModule} from "@angular/material/button-toggle";
+import {MatTabsModule} from "@angular/material/tabs";
+import {MatSliderModule} from "@angular/material/slider";
+import {MatSlideToggleModule} from "@angular/material/slide-toggle";
+import {MatTooltipModule} from "@angular/material/tooltip";
+
+@NgModule({
+    declarations: [
+        DashboardSettingsComponent
+    ],
+    imports     : [
+        RouterModule.forChild([]),
+        MatAutocompleteModule,
+        MatDialogModule,
+        MatButtonModule,
+        MatSelectModule,
+        MatFormFieldModule,
+        MatIconModule,
+        MatInputModule,
+        MatButtonToggleModule,
+        MatTabsModule,
+        MatTooltipModule,
+        MatSliderModule,
+        MatSlideToggleModule,
+        SharedModule
+    ],
+    exports     : [
+        DashboardSettingsComponent
+    ],
+    providers   : []
+})
+export class DashboardSettingsModule
+{
+}
@@ -0,0 +1,30 @@
+<h2 mat-dialog-title>Scrutiny Settings</h2>
+<mat-dialog-content class="mat-typography">
+
+    <form class="flex flex-col p-8 pb-0 ">
+        <div class="flex flex-col gt-xs:flex-row">
+            <mat-form-field class="flex-auto gt-xs:pr-3">
+                <mat-label>Threshold Data</mat-label>
+                <mat-select [value]="'scrutiny'">
+                    <mat-option  value="scrutiny">Scrutiny</mat-option>
+                    <mat-option value="name" disabled>Manufacturer</mat-option>
+                </mat-select>
+            </mat-form-field>
+        </div>
+
+        <div class="flex flex-col gt-xs:flex-row">
+            <mat-form-field class="flex-auto gt-xs:pr-3">
+                <mat-label>Notifications</mat-label>
+                <mat-select [value]="'enable'">
+                    <mat-option  value="enable">Enabled</mat-option>
+                    <mat-option value="disable" disabled>Disabled</mat-option>
+                </mat-select>
+            </mat-form-field>
+        </div>
+    </form>
+
+</mat-dialog-content>
+<mat-dialog-actions align="end">
+    <button mat-button mat-dialog-close>Cancel</button>
+    <button mat-button matTooltip="not yet implemented" [mat-dialog-close]="true" cdkFocusInitial>Save</button>
+</mat-dialog-actions>
@@ -0,0 +1,25 @@
+import { async, ComponentFixture, TestBed } from '@angular/core/testing';
+
+import { DetailSettingsComponent } from './detail-settings.component';
+
+describe('DetailSettingsComponent', () => {
+  let component: DetailSettingsComponent;
+  let fixture: ComponentFixture<DetailSettingsComponent>;
+
+  beforeEach(async(() => {
+    TestBed.configureTestingModule({
+      declarations: [ DetailSettingsComponent ]
+    })
+    .compileComponents();
+  }));
+
+  beforeEach(() => {
+    fixture = TestBed.createComponent(DetailSettingsComponent);
+    component = fixture.componentInstance;
+    fixture.detectChanges();
+  });
+
+  it('should create', () => {
+    expect(component).toBeTruthy();
+  });
+});
@@ -0,0 +1,15 @@
+import { Component, OnInit } from '@angular/core';
+
+@Component({
+  selector: 'app-detail-settings',
+  templateUrl: './detail-settings.component.html',
+  styleUrls: ['./detail-settings.component.scss']
+})
+export class DetailSettingsComponent implements OnInit {
+
+  constructor() { }
+
+  ngOnInit(): void {
+  }
+
+}
@@ -0,0 +1,46 @@
+import { NgModule } from '@angular/core';
+import { RouterModule } from '@angular/router';
+import { Overlay } from '@angular/cdk/overlay';
+import { MAT_AUTOCOMPLETE_SCROLL_STRATEGY, MatAutocompleteModule } from '@angular/material/autocomplete';
+import { MatButtonModule } from '@angular/material/button';
+import { MatSelectModule } from '@angular/material/select';
+import { MatFormFieldModule } from '@angular/material/form-field';
+import { MatIconModule } from '@angular/material/icon';
+import { MatInputModule } from '@angular/material/input';
+import { SharedModule } from 'app/shared/shared.module';
+import {DetailSettingsComponent} from 'app/layout/common/detail-settings/detail-settings.component'
+import { MatDialogModule } from "@angular/material/dialog";
+import { MatButtonToggleModule} from "@angular/material/button-toggle";
+import {MatTabsModule} from "@angular/material/tabs";
+import {MatSliderModule} from "@angular/material/slider";
+import {MatSlideToggleModule} from "@angular/material/slide-toggle";
+import {MatTooltipModule} from "@angular/material/tooltip";
+
+@NgModule({
+    declarations: [
+        DetailSettingsComponent
+    ],
+    imports     : [
+        RouterModule.forChild([]),
+        MatAutocompleteModule,
+        MatDialogModule,
+        MatButtonModule,
+        MatSelectModule,
+        MatFormFieldModule,
+        MatIconModule,
+        MatInputModule,
+        MatButtonToggleModule,
+        MatTabsModule,
+        MatTooltipModule,
+        MatSliderModule,
+        MatSlideToggleModule,
+        SharedModule
+    ],
+    exports     : [
+        DetailSettingsComponent
+    ],
+    providers   : []
+})
+export class DetailSettingsModule
+{
+}
@@ -16,7 +16,7 @@ const modules = [

@NgModule({
    declarations: [
-        LayoutComponent
+        LayoutComponent,
    ],
    imports     : [
        TreoDrawerModule,
@@ -10,15 +10,13 @@
                </div>
                <!-- Action buttons -->
                <div class="flex items-center">
-                    <button class="xs:hidden"
-                            matTooltip="not yet implemented"
-                            mat-stroked-button>
+                    <button matTooltip="not yet implemented" class="xs:hidden" mat-stroked-button>
                        <mat-icon class="icon-size-20"
                                  [svgIcon]="'save'"></mat-icon>
                        <span class="ml-2">Export</span>
                    </button>
                    <button class="ml-2 xs:hidden"
-                            matTooltip="not yet implemented"
+                            (click)="openDialog()"
                            mat-stroked-button>
                        <mat-icon class="icon-size-20 rotate-90 mirror"
                                  [svgIcon]="'tune'"></mat-icon>
@@ -38,8 +36,7 @@
                                          [svgIcon]="'save'"></mat-icon>
                                <span class="ml-2">Export</span>
                            </button>
-                            <button mat-menu-item
-                                    matTooltip="not yet implemented">
+                            <button mat-menu-item (click)="openDialog()">
                                <mat-icon class="icon-size-20 rotate-90 mirror"
                                          [svgIcon]="'tune'"></mat-icon>
                                <span class="ml-2">Settings</span>
@@ -69,7 +66,7 @@
                        <div class="flex items-center">
                            <div class="flex flex-col">
                                <a [routerLink]="'/device/'+ disk.wwn"
-                                   class="font-bold text-md text-secondary uppercase tracking-wider">/dev/{{disk.device_name}} - {{disk.model_name}}</a>
+                                   class="font-bold text-md text-secondary uppercase tracking-wider">{{deviceTitle(disk)}}</a>
                                <div [ngClass]="{'text-green': disk.smart_results[0]?.smart_status == 'passed',
                                    'text-red': disk.smart_results[0]?.smart_status == 'failed' }" class="font-medium text-sm" *ngIf="disk.smart_results[0]">
                                    Last Updated on {{disk.smart_results[0]?.date | date:'MMMM dd, yyyy' }}
@@ -4,8 +4,10 @@ import { MatTableDataSource } from '@angular/material/table';
 import { Subject } from 'rxjs';
 import { takeUntil } from 'rxjs/operators';
 import { ApexOptions } from 'ng-apexcharts';
-import { DashboardService } from 'app/modules/admin/dashboard/dashboard.service';
+import { DashboardService } from 'app/modules/dashboard/dashboard.service';
 import * as moment from "moment";
+import {MatDialog} from '@angular/material/dialog';
+import { DashboardSettingsComponent } from 'app/layout/common/dashboard-settings/dashboard-settings.component';

@Component({
    selector       : 'example',
@@ -28,7 +30,8 @@ export class DashboardComponent implements OnInit, AfterViewInit, OnDestroy
     * @param {SmartService} _smartService
     */
    constructor(
-        private _smartService: DashboardService
+        private _smartService: DashboardService,
+        public dialog: MatDialog
    )
    {
        // Set the private defaults
@@ -153,6 +156,23 @@ export class DashboardComponent implements OnInit, AfterViewInit, OnDestroy
    // @ Public methods
    // -----------------------------------------------------------------------------------------------------

+    openDialog() {
+        const dialogRef = this.dialog.open(DashboardSettingsComponent);
+
+        dialogRef.afterClosed().subscribe(result => {
+            console.log(`Dialog result: ${result}`);
+        });
+    }
+
+    deviceTitle(disk){
+        var title = [`/dev/${disk.device_name}`]
+        if (disk.device_type && disk.device_type != 'scsi' && disk.device_type != 'ata'){
+            title.push(disk.device_type)
+        }
+        title.push(disk.model_name)
+        return title.join(' - ')
+    }
+
    /**
     * Track by function for ngFor loops
     *
@@ -1,8 +1,8 @@
 import { NgModule } from '@angular/core';
 import { RouterModule } from '@angular/router';
 import { SharedModule } from 'app/shared/shared.module';
-import { DashboardComponent } from 'app/modules/admin/dashboard/dashboard.component';
-import { dashboardRoutes } from 'app/modules/admin/dashboard/dashboard.routing';
+import { DashboardComponent } from 'app/modules/dashboard/dashboard.component';
+import { dashboardRoutes } from 'app/modules/dashboard/dashboard.routing';
 import { MatButtonModule } from '@angular/material/button';
 import { MatDividerModule } from '@angular/material/divider';
 import { MatIconModule } from '@angular/material/icon';
@@ -12,6 +12,7 @@ import { MatSortModule } from '@angular/material/sort';
 import { MatTableModule } from '@angular/material/table';
 import { NgApexchartsModule } from 'ng-apexcharts';
 import { MatTooltipModule } from '@angular/material/tooltip'
+import { DashboardSettingsModule } from "app/layout/common/dashboard-settings/dashboard-settings.module";

@NgModule({
    declarations: [
@@ -28,7 +29,8 @@ import { MatTooltipModule } from '@angular/material/tooltip'
        MatSortModule,
        MatTableModule,
        NgApexchartsModule,
-        SharedModule
+        SharedModule,
+        DashboardSettingsModule
    ]
 })
 export class DashboardModule
@@ -1,7 +1,7 @@
 import { Injectable } from '@angular/core';
 import { ActivatedRouteSnapshot, Resolve, RouterStateSnapshot } from '@angular/router';
 import { Observable } from 'rxjs';
-import { DashboardService } from 'app/modules/admin/dashboard/dashboard.service';
+import { DashboardService } from 'app/modules/dashboard/dashboard.service';

@Injectable({
    providedIn: 'root'
@@ -1,5 +1,5 @@
 import { Route } from '@angular/router';
-import { DashboardComponent } from 'app/modules/admin/dashboard/dashboard.component';
+import { DashboardComponent } from 'app/modules/dashboard/dashboard.component';
 import {DashboardResolver} from "./dashboard.resolvers";

 export const dashboardRoutes: Route[] = [
@@ -17,7 +17,7 @@
                    <span class="ml-2">Export</span>
                </button>
                <button class="ml-2 xs:hidden"
-                        matTooltip="not yet implemented"
+                        (click)="openDialog()"
                        mat-stroked-button>
                    <mat-icon class="icon-size-20 rotate-90 mirror"
                              [svgIcon]="'tune'"></mat-icon>
@@ -38,7 +38,7 @@
                            <span class="ml-2">Export</span>
                        </button>
                        <button mat-menu-item
-                                matTooltip="not yet implemented">
+                                (click)="openDialog()">
                            <mat-icon class="icon-size-20 rotate-90 mirror"
                                      [svgIcon]="'tune'"></mat-icon>
                            <span class="ml-2">Settings</span>
@@ -56,6 +56,10 @@
                    <div class="text-2xl font-semibold leading-tight">/dev/{{data.data.device_name}}</div>
                </div>
                <div class="flex flex-col my-2 grid grid-cols-2">
+                    <div *ngIf="data.data.device_type && data.data.device_type != 'ata' && data.data.device_type != 'scsi'" class="my-2 col-span-2 lt-md:col-span-1">
+                        <div>{{data.data.device_type}}</div>
+                        <div class="text-secondary text-md">Device Type</div>
+                    </div>
                    <div *ngIf="data.data.manufacturer" class="my-2 col-span-2 lt-md:col-span-1">
                        <div>{{data.data.manufacturer}}</div>
                        <div class="text-secondary text-md">Model Family</div>
@@ -3,9 +3,11 @@ import {ApexOptions} from "ng-apexcharts";
 import {MatTableDataSource} from "@angular/material/table";
 import {MatSort} from "@angular/material/sort";
 import {Subject} from "rxjs";
-import {DetailService} from "../detail/detail.service";
+import {DetailService} from "./detail.service";
 import {takeUntil} from "rxjs/operators";
-import {fadeOut} from "../../../../@treo/animations/fade";
+import {fadeOut} from "../../../@treo/animations/fade";
+import {DetailSettingsComponent} from "app/layout/common/detail-settings/detail-settings.component";
+import {MatDialog} from "@angular/material/dialog";

@Component({
  selector: 'detail',
@@ -34,7 +36,9 @@ export class DetailComponent implements OnInit, AfterViewInit, OnDestroy {
     * @param {DetailService} _detailService
     */
    constructor(
-        private _detailService: DetailService
+        private _detailService: DetailService,
+        public dialog: MatDialog
+
    )
    {
        // Set the private defaults
@@ -277,6 +281,15 @@ export class DetailComponent implements OnInit, AfterViewInit, OnDestroy {
        this.smartAttributeDataSource.data = this._generateSmartAttributeTableDataSource(this.data.data.smart_results);

    }
+
+    openDialog() {
+        const dialogRef = this.dialog.open(DetailSettingsComponent);
+
+        dialogRef.afterClosed().subscribe(result => {
+            console.log(`Dialog result: ${result}`);
+        });
+    }
+
    /**
     * Track by function for ngFor loops
     *
@@ -1,8 +1,8 @@
 import { NgModule } from '@angular/core';
 import { RouterModule } from '@angular/router';
 import { SharedModule } from 'app/shared/shared.module';
-import { DetailComponent } from 'app/modules/admin/detail/detail.component';
-import { detailRoutes } from 'app/modules/admin/detail/detail.routing';
+import { DetailComponent } from 'app/modules/detail/detail.component';
+import { detailRoutes } from 'app/modules/detail/detail.routing';
 import { MatButtonModule } from '@angular/material/button';
 import { MatDividerModule } from '@angular/material/divider';
 import { MatIconModule } from '@angular/material/icon';
@@ -13,6 +13,7 @@ import { MatTableModule } from '@angular/material/table';
 import { MatTooltipModule } from '@angular/material/tooltip'
 import { NgApexchartsModule } from 'ng-apexcharts';
 import { TreoCardModule } from '@treo/components/card';
+import {DetailSettingsModule} from "app/layout/common/detail-settings/detail-settings.module";

@NgModule({
    declarations: [
@@ -31,7 +32,7 @@ import { TreoCardModule } from '@treo/components/card';
        NgApexchartsModule,
        TreoCardModule,
        SharedModule,
-
+        DetailSettingsModule,
    ]
 })
 export class DetailModule
@@ -1,7 +1,7 @@
 import { Injectable } from '@angular/core';
 import { ActivatedRouteSnapshot, Resolve, RouterStateSnapshot } from '@angular/router';
 import { Observable } from 'rxjs';
-import { DetailService } from 'app/modules/admin/detail/detail.service';
+import { DetailService } from 'app/modules/detail/detail.service';

@Injectable({
    providedIn: 'root'
@@ -1,5 +1,5 @@
 import { Route } from '@angular/router';
-import { DetailComponent } from 'app/modules/admin/detail/detail.component';
+import { DetailComponent } from 'app/modules/detail/detail.component';
 import {DetailResolver} from "./detail.resolvers";

 export const detailRoutes: Route[] = [
Author	SHA1	Message	Date
Jason Kulatunga	75a5f7dfb6	(0.2.1) Automated packaging of release by Packagr Signed-off-by: Jason Kulatunga <jason@thesparktree.com>	2020-09-24 03:26:38 +00:00
Jason Kulatunga	2c2ecbd9f7	fix link	2020-09-23 20:51:13 -06:00
Jason Kulatunga	9bd8aec315	update getting started & documentation to remove -v /dev/ mount and --privileged requirement. Uses --cap-add and --device instead close #26 close #18	2020-09-23 20:49:56 -06:00
Jason Kulatunga	e44864e64b	fixes.	2020-09-23 11:01:53 -06:00
Jason Kulatunga	8a336bf5c6	wwn should always be lowercase for consistency. It's used in the URL for pushing smart data.	2020-09-23 10:37:59 -06:00
Jason Kulatunga	6a20228262	adding error handling for all DB calls. Returning StatusInternalServerError whenever an error occurs. Adding additional logging to server handlers. Make sure we "return" after a c.JSON call.	2020-09-23 09:54:33 -06:00
Jason Kulatunga	531fea76b2	keep example config file in sync with config defaults. fixes #3 Adding an example config file for local development to CONTRIBUTING.md.	2020-09-22 18:18:57 -06:00
Jason Kulatunga	5127399e94	conditionally log request body.	2020-09-22 10:26:33 -06:00
Jason Kulatunga	8a975e2164	log request body.	2020-09-22 10:09:29 -06:00
Jason Kulatunga	7cdacbaffc	add information about how to generate debug logs	2020-09-21 22:27:59 -06:00
Jason Kulatunga	2fee2bf906	fix tests.	2020-09-21 21:22:07 -06:00
Jason Kulatunga	1c59b3c245	fix tests.	2020-09-21 19:24:39 -06:00
Jason Kulatunga	119e24f6ec	Merge pull request #41 from AnalogJ/ext_logging adding new environmental variables for added debugging: COLLECTOR_LOG…	2020-09-21 19:04:29 -06:00
Jason Kulatunga	a57120d600	adding new environmental variables for added debugging: COLLECTOR_LOG_FILE, COLLECTOR_DEBUG, DEBUG, SCRUTINY_LOG_FILE, SCRUTINY_DEBUG	2020-09-21 18:41:52 -06:00
Jason Kulatunga	f2dd87cf82	dont use a go-routine -- disable concurrency.	2020-09-21 09:59:52 -06:00
Jason Kulatunga	8b139ad157	remove concurrency for collector, it causes issues on systems with lots of devices. Just retrieve the data in order for now (eventually we may do it in batches).	2020-09-21 08:44:50 -06:00
Jason Kulatunga	286ec25a94	(0.2.0) Automated packaging of release by Packagr Signed-off-by: Jason Kulatunga <jason@thesparktree.com>	2020-09-21 03:15:05 +00:00
Jason Kulatunga	211fe7843a	Update issue templates	2020-09-20 21:00:52 -06:00
Jason Kulatunga	d6fad90fa4	Merge pull request #38 from AnalogJ/smartctl_scan Smartctl scan	2020-09-20 20:42:26 -06:00
Jason Kulatunga	fd82e9a4da	print error message if publish does not finish successfully.	2020-09-20 17:37:43 -06:00
Jason Kulatunga	ad3f8480d9	added some anonymized nvme test data to ensure that NVMe drives are correctly processed by new device detection. thanks @Roxedus. All db testing is done with files created in a temp directory.	2020-09-20 17:37:43 -06:00
Jason Kulatunga	297f0a51c5	adding ability to write a log file with all output from collector. Executing commands will now log be logged (and when debug is enabled, their output's are also logged).	2020-09-20 17:37:43 -06:00
Jason Kulatunga	67d1c592a5	include the device type in the title, if it's non-standard.	2020-09-20 17:37:43 -06:00
Jason Kulatunga	24262f7c8c	pass deviceType when running smartctl data collection (for megaraid type disks). Make DevicePrefix a public function available outside the detect module. if device type is detected as "ata" or "scsi", dont pass in via -d parameter, can cause issues with missing data.	2020-09-20 17:37:43 -06:00
Jason Kulatunga	66122778a3	only the firmware can really change between scans. Lets slim down the metadata update.	2020-09-20 17:37:43 -06:00
Jason Kulatunga	3deca46851	fix.	2020-09-20 17:37:43 -06:00
Jason Kulatunga	23d5b86b1b	new device detection engine (OS aware). Uses smartctl --scan to detect drives (and conditionally uses jaypipes/ghw). WWN is calculated from smartctl data, then retrieved from GHW, and fallsback to serial number. WWN calcuation code is based on IEEE spec, for "Registered" IEEE format - NAA5. TODO: support NAA6 and other formats?	2020-09-20 17:37:43 -06:00
Jason Kulatunga	f53833d617	add codecov badge.	2020-09-20 16:51:36 -06:00
Jason Kulatunga	fab1e3c624	fix the coverage file path.	2020-09-20 16:28:38 -06:00
Jason Kulatunga	a55f3acacf	generate a coverage.txt file, and upload to codecov.io.	2020-09-20 16:24:30 -06:00
Jason Kulatunga	853b1a7249	Merge pull request #28 from AnalogJ/notify_failures Adding UI for settings (and eventually notifications).	2020-09-18 14:44:24 -06:00
Jason Kulatunga	a7cd912318	dont ignore NVME storage controllers.	2020-09-18 10:43:14 -06:00
Jason Kulatunga	e6eeaf7e31	adding mocked detail data for frontend viewing. placeholders for settings panels. Add dialog panel for Details setings.	2020-09-18 10:14:19 -06:00
Jason Kulatunga	5101a37964	adding device protocl and type to the. Adding class for parsing `smartctl --scan` json output, for device detection. added an example/test file for `smartctl -x -j` added a placeholder settings panel. moved dashboard & details compoonent out of "Admin" directory.	2020-09-18 10:14:19 -06:00
Jason Kulatunga	98415e625d	fix import. added simle test for notify test endpoint.	2020-09-18 10:14:19 -06:00
Jason Kulatunga	78a619b09d	moved middleware into more relevant location. Adding send test notifications handler. making sure that config is available from web handler functions.	2020-09-18 10:13:06 -06:00
Jason Kulatunga	c913cf39b9	adding new nottification validation erorr. Added a notification class containing webhook, script and shoutrrr notification logic. Adding "Test notification endpoint".	2020-09-18 10:13:06 -06:00
Jason Kulatunga	6bee0d1489	don't build a docker image and attach to the release. Scrutiny is public now, images hosted on docker-hub.	2020-09-18 10:06:33 -06:00
Jason Kulatunga	cadf055105	add license and logo link.	2020-09-18 09:58:57 -06:00
Jason Kulatunga	1acd7e0d47	badges	2020-09-18 09:45:11 -06:00
Jason Kulatunga	266c95f857	fix #3 Make sure that the collector attempts to correctly communicate with webapp container, even when running in dedicated container (and triggered manually). fixes cron schedule to run daily. added instructions for dedicated containers.	2020-09-18 09:23:39 -06:00
Jason Kulatunga	458eaf2726	Merge pull request #1 from RealOrangeOne/patch-1 Fix scrutiny binary location	2020-09-17 08:45:03 -07:00
Jake Howard	5c6e1b5ce2	Fix scrutiny binary location	2020-09-17 13:28:21 +01:00