Skip to content

slapos
slapos
Commit 9e5f148c authored by Jérome Perrin's avatar Jérome Perrin
Browse files
Options

Grafana Loki 

https://github.com/grafana/loki/blob/master/docs/README.md

> Grafana Loki is a set of components that can be composed into a fully featured logging stack.

The plan is not to really use this, just to see what interesting ideas are in there.

/reviewed-on nexedi/slapos!658
parents 17360d98 5afbacd1
Pipeline #7822 failed with stage
    Show whitespace changes
    Inline Side-by-side
    Showing with 607 additions and 869 deletions
    component/caddy/buildout.cfg
    View file @ 9e5f148c
    ......@@ -8,6 +8,8 @@ parts =
    [gowork]
    # Caddy 1.x+ uses go modules, for which gowork does not work yet
    golang = ${golang1.12:location}
    gcc-bin-directory = ${golang1.12:gcc-bin-directory}
    install =
    [gowork.goinstall]
    ......
    component/golang/buildout.cfg
    View file @ 9e5f148c
    ......@@ -17,10 +17,12 @@ configure-command = :
    location = @@LOCATION@@
    make-binary =
    make-targets= cd src && ./all.bash && cp -alf .. ${:location}
    # gcc version to use
    gcc-bin-directory = ${gcc-8.2:location}/bin
    # some testdata files have an issue with slapos.extension.strip.
    post-install = ${findutils:location}/bin/find ${:location}/src -type d -name testdata -exec rm -rf {} \; || true
    environment =
    PATH=${gcc-8.2:location}/bin:%(PATH)s
    PATH=${:gcc-bin-directory}:%(PATH)s
    GOROOT_FINAL=${:location}
    ${:environment-extra}
    ......@@ -104,6 +106,9 @@ depends = ${gowork.goinstall:recipe}
    # go version used for the workspace (possible to override in applications)
    golang = ${golang1.10:location}
    # gcc version must be compatible with go version selected
    gcc-bin-directory = ${golang1.10:gcc-bin-directory}
    # no special build flags by default
    buildflags =
    ......@@ -123,7 +128,7 @@ recipe = slapos.recipe.template
    url = ${:_profile_base_location_}/goenv.sh.in
    output = ${gowork:directory}/env.sh
    depends = ${gowork.mkdir:recipe}
    md5sum = 7a067a3974c446c3eaa0e82818ba1adb
    md5sum = 7eaad1f9aabd3cfad554975098c5d4c3
    [gowork.mkdir]
    # NOTE do not use slapos.cookbook:mkdirectory here - if anything in software (not instance)
    ......
    component/golang/goenv.sh.in
    View file @ 9e5f148c
    ......@@ -3,7 +3,7 @@
    # ---- 8< ---- (buildout substitution here)
    # PATH so that go & friends work out of the box
    export PATH=${gowork:golang}/bin:${git:location}/bin:${pkgconfig:location}/bin:${buildout:bin-directory}:$PATH
    export PATH=${gowork:golang}/bin:${git:location}/bin:${pkgconfig:location}/bin:${buildout:bin-directory}:${gowork:gcc-bin-directory}:$PATH
    X=${gowork:directory}
    export PKG_CONFIG_PATH=$(echo -n "${gowork:cpkgpath}" |tr '\n' ':'):$PKG_CONFIG_PATH
    ......
    component/helloweb/buildout.cfg
    View file @ 9e5f148c
    ......@@ -26,7 +26,7 @@ install =
    lab.nexedi.com/nexedi/helloweb/go/...
    golang = ${golang1.12:location}
    gcc-bin-directory = ${golang1.12:gcc-bin-directory}
    # -*- go -*-
    [helloweb-go]
    ......
    software/grafana/README.md
    View file @ 9e5f148c
    # grafana / telegraf / influxdb
    This is an experimental integration, mainly to evaluate these solutions.
    ## Custom telegraf plugins
    ......@@ -7,24 +8,20 @@
    See https://github.com/influxdata/telegraf to learn about plugins.
    Useful plugins in this context are probably
    [exec](https://github.com/influxdata/telegraf/tree/1.5.1/plugins/inputs/exec)
    [exec](https://github.com/influxdata/telegraf/tree/1.11.1/plugins/inputs/exec),
    [logparser](https://github.com/influxdata/telegraf/tree/1.11.1/plugins/inputs/logparser)
    or
    [httpjson](https://github.com/influxdata/telegraf/tree/1.5.1/plugins/inputs/httpjson).
    [http](https://github.com/influxdata/telegraf/tree/1.11.1/plugins/inputs/http).
    Telegraf will save in the `telegraf` database from the embedded influxdb server.
    ## Grafana
    You'll have to add yourself the influxdb data source in grafana, using the
    parameters published by the slapos instance.
    A default user is created, username and password are published as connection
    parameters. You can add more users in grafana interface.
    http://docs.grafana.org/features/datasources/influxdb/
    When adding datasource, use *proxy* option, otherwise Grafana makes your
    browser query influxdb directly, which also uses a self signed certificate.
    One workaround is to configure your browser to also accept influxdb certificate
    before using grafana, but using proxy seems easier.
    Datasources should be automatically added.
    ## Influxdb
    ......@@ -34,8 +31,32 @@ One important thing to notice is that the backup protocol is enabled on ipv4
    provided by slapos, so make sure this ip is not reachable from untrusted
    sources.
    ## TODO
    # Ingesting/Visualizing logs
    Eventhough main feature is visualizing metrics, Grafana has a feature called "Explore" to view logs for a time frame.
    The following backend can be used:
    ## Loki
    See `TestLoki` in test for an example.
    ## Influxdb
    Influxdb logs only have tags and there does not seem to be a way to search (except than tag and time frame).
    To inject log files containing:
    ```
    INFO the message
    WARN another message
    ```
    use config like:
    ```
    [[inputs.logparser]]
    files = ["/tmp/x*.log", "/tmp/aaa.log"]
    * influxdb and telegraf runs with very low priority, this could become an option
    * make one partition for each service and use switch software type
    * make it easier to add custom configuration (how ?)
    [inputs.logparser.grok]
    measurement = "logs"
    patterns = ['^%{WORD:level:tag} %{GREEDYDATA:message:string}']
    ```
    software/grafana/buildout.hash.cfg
    View file @ 9e5f148c
    ......@@ -15,7 +15,7 @@
    [instance-profile]
    filename = instance.cfg.in
    md5sum = 461d515da03de5e422e6f75189d09184
    md5sum = 214a756a6a9f73455411760b2daa25b7
    [influxdb-config-file]
    filename = influxdb-config-file.cfg.in
    ......@@ -28,3 +28,15 @@ md5sum = a1a9c22c2a7829c66a49fc2504604d21
    [grafana-config-file]
    filename = grafana-config-file.cfg.in
    md5sum = 8244d430905b968795c7946049bed9e3
    [grafana-provisioning-config-file]
    filename = grafana-provisioning-config-file.cfg.in
    md5sum = 3aa0f1ed752b2a59ea2b5e7c1733daf3
    [loki-config-file]
    filename = loki-config-file.cfg.in
    md5sum = 02ba5acf23fcf88f5594919f46838533
    [promtail-config-file]
    filename = promtail-config-file.cfg.in
    md5sum = c77788d0a3cc654ad9393eb4b1f31e94
    software/grafana/gowork.cfg
    View file @ 9e5f148c
    ......@@ -3,844 +3,39 @@
    # list of go git repositories to fetch
    [gowork.goinstall]
    depends_gitfetch =
    ${go_collectd.org:recipe}
    ${go_github.com_BurntSushi_toml:recipe}
    ${go_github.com_Microsoft_go-winio:recipe}
    ${go_github.com_RoaringBitmap_roaring:recipe}
    ${go_github.com_Shopify_sarama:recipe}
    ${go_github.com_Sirupsen_logrus:recipe}
    ${go_github.com_StackExchange_wmi:recipe}
    ${go_github.com_aerospike_aerospike-client-go:recipe}
    ${go_github.com_amir_raidman:recipe}
    ${go_github.com_apache_thrift:recipe}
    ${go_github.com_aws_aws-sdk-go:recipe}
    ${go_github.com_beorn7_perks:recipe}
    ${go_github.com_bmizerany_pat:recipe}
    ${go_github.com_boltdb_bolt:recipe}
    ${go_github.com_bsm_sarama-cluster:recipe}
    ${go_github.com_cenkalti_backoff:recipe}
    ${go_github.com_cespare_xxhash:recipe}
    ${go_github.com_couchbase_go-couchbase:recipe}
    ${go_github.com_couchbase_gomemcached:recipe}
    ${go_github.com_couchbase_goutils:recipe}
    ${go_github.com_davecgh_go-spew:recipe}
    ${go_github.com_dgrijalva_jwt-go:recipe}
    ${go_github.com_dgryski_go-bitstream:recipe}
    ${go_github.com_docker_docker:recipe}
    ${go_github.com_docker_go-connections:recipe}
    ${go_github.com_eapache_go-resiliency:recipe}
    ${go_github.com_eapache_go-xerial-snappy:recipe}
    ${go_github.com_eapache_queue:recipe}
    ${go_github.com_eclipse_paho.mqtt.golang:recipe}
    ${go_github.com_glycerine_go-unsnap-stream:recipe}
    ${go_github.com_go-ini_ini:recipe}
    ${go_github.com_go-logfmt_logfmt:recipe}
    ${go_github.com_go-ole_go-ole:recipe}
    ${go_github.com_go-sql-driver_mysql:recipe}
    ${go_github.com_gobwas_glob:recipe}
    ${go_github.com_gogo_protobuf:recipe}
    ${go_github.com_golang_protobuf:recipe}
    ${go_github.com_golang_snappy:recipe}
    ${go_github.com_google_go-cmp:recipe}
    ${go_github.com_gorilla_mux:recipe}
    ${go_github.com_golang_dep:recipe}
    ${go_github.com_grafana_grafana:recipe}
    ${go_github.com_hailocab_go-hostpool:recipe}
    ${go_github.com_hashicorp_consul:recipe}
    ${go_github.com_grafana_loki:recipe}
    ${go_github.com_influxdata_influxdb:recipe}
    ${go_github.com_influxdata_influxql:recipe}
    ${go_github.com_influxdata_tail:recipe}
    ${go_github.com_influxdata_telegraf:recipe}
    ${go_github.com_influxdata_toml:recipe}
    ${go_github.com_influxdata_usage-client:recipe}
    ${go_github.com_influxdata_wlog:recipe}
    ${go_github.com_influxdata_yamux:recipe}
    ${go_github.com_influxdata_yarpc:recipe}
    ${go_github.com_jackc_pgx:recipe}
    ${go_github.com_jmespath_go-jmespath:recipe}
    ${go_github.com_jwilder_encoding:recipe}
    ${go_github.com_kardianos_govendor:recipe}
    ${go_github.com_kardianos_osext:recipe}
    ${go_github.com_kardianos_service:recipe}
    ${go_github.com_kballard_go-shellquote:recipe}
    ${go_github.com_matttproud_golang_protobuf_extensions:recipe}
    ${go_github.com_miekg_dns:recipe}
    ${go_github.com_mitchellh_mapstructure:recipe}
    ${go_github.com_multiplay_go-ts3:recipe}
    ${go_github.com_naoina_go-stringutil:recipe}
    ${go_github.com_nats-io_go-nats:recipe}
    ${go_github.com_nats-io_nats:recipe}
    ${go_github.com_nats-io_nuid:recipe}
    ${go_github.com_nsqio_go-nsq:recipe}
    ${go_github.com_opencontainers_runc:recipe}
    ${go_github.com_opentracing-contrib_go-observer:recipe}
    ${go_github.com_opentracing_opentracing-go:recipe}
    ${go_github.com_openzipkin_zipkin-go-opentracing:recipe}
    ${go_github.com_peterh_liner:recipe}
    ${go_github.com_philhofer_fwd:recipe}
    ${go_github.com_pierrec_lz4:recipe}
    ${go_github.com_pierrec_xxHash:recipe}
    ${go_github.com_pkg_errors:recipe}
    ${go_github.com_pmezard_go-difflib:recipe}
    ${go_github.com_prometheus_client_golang:recipe}
    ${go_github.com_prometheus_client_model:recipe}
    ${go_github.com_prometheus_common:recipe}
    ${go_github.com_prometheus_procfs:recipe}
    ${go_github.com_rcrowley_go-metrics:recipe}
    ${go_github.com_retailnext_hllpp:recipe}
    ${go_github.com_samuel_go-zookeeper:recipe}
    ${go_github.com_satori_go.uuid:recipe}
    ${go_github.com_shirou_gopsutil:recipe}
    ${go_github.com_shirou_w32:recipe}
    ${go_github.com_soniah_gosnmp:recipe}
    ${go_github.com_sparrc_gdm:recipe}
    ${go_github.com_streadway_amqp:recipe}
    ${go_github.com_stretchr_objx:recipe}
    ${go_github.com_stretchr_testify:recipe}
    ${go_github.com_tidwall_gjson:recipe}
    ${go_github.com_tidwall_match:recipe}
    ${go_github.com_tinylib_msgp:recipe}
    ${go_github.com_vjeantet_grok:recipe}
    ${go_github.com_wvanbergen_kafka:recipe}
    ${go_github.com_wvanbergen_kazoo-go:recipe}
    ${go_github.com_xlab_treeprint:recipe}
    ${go_github.com_yuin_gopher-lua:recipe}
    ${go_github.com_zensqlmonitor_go-mssqldb:recipe}
    ${go_go.uber.org_atomic:recipe}
    ${go_go.uber.org_multierr:recipe}
    ${go_go.uber.org_zap:recipe}
    ${go_golang.org_x_crypto:recipe}
    ${go_golang.org_x_net:recipe}
    ${go_golang.org_x_sys:recipe}
    ${go_golang.org_x_text:recipe}
    ${go_golang.org_x_time:recipe}
    ${go_golang.org_x_tools:recipe}
    ${go_gopkg.in_asn1-ber.v1:recipe}
    ${go_gopkg.in_fatih_pool.v2:recipe}
    ${go_gopkg.in_fsnotify.v1:recipe}
    ${go_gopkg.in_gorethink_gorethink.v3:recipe}
    ${go_gopkg.in_ldap.v2:recipe}
    ${go_gopkg.in_mgo.v2:recipe}
    ${go_gopkg.in_olivere_elastic.v5:recipe}
    ${go_gopkg.in_tomb.v1:recipe}
    ${go_gopkg.in_yaml.v2:recipe}
    [go_collectd.org]
    [go_github.com_golang_dep]
    <= go-git-package
    go.importpath = collectd.org
    repository = https://github.com/collectd/go-collectd
    revision = v0.3.0-17-g606bd390f3
    [go_github.com_BurntSushi_toml]
    <= go-git-package
    go.importpath = github.com/BurntSushi/toml
    repository = https://github.com/BurntSushi/toml
    revision = v0.2.0-45-ga368813c5e
    [go_github.com_Microsoft_go-winio]
    <= go-git-package
    go.importpath = github.com/Microsoft/go-winio
    repository = https://github.com/Microsoft/go-winio
    revision = ce2922f643
    [go_github.com_RoaringBitmap_roaring]
    <= go-git-package
    go.importpath = github.com/RoaringBitmap/roaring
    repository = https://github.com/RoaringBitmap/roaring
    revision = v0.2.8-174-g0a6691af7c
    [go_github.com_Shopify_sarama]
    <= go-git-package
    go.importpath = github.com/Shopify/sarama
    repository = https://github.com/Shopify/sarama
    revision = 3b1b38866a
    [go_github.com_Sirupsen_logrus]
    <= go-git-package
    go.importpath = github.com/Sirupsen/logrus
    repository = https://github.com/Sirupsen/logrus
    revision = 61e43dc76f
    [go_github.com_StackExchange_wmi]
    <= go-git-package
    go.importpath = github.com/StackExchange/wmi
    repository = https://github.com/StackExchange/wmi
    revision = f3e2bae1e0
    [go_github.com_aerospike_aerospike-client-go]
    <= go-git-package
    go.importpath = github.com/aerospike/aerospike-client-go
    repository = https://github.com/aerospike/aerospike-client-go
    revision = v1.6.4-277-g95e1ad7791
    [go_github.com_amir_raidman]
    <= go-git-package
    go.importpath = github.com/amir/raidman
    repository = https://github.com/amir/raidman
    revision = c74861fe6a
    [go_github.com_apache_thrift]
    <= go-git-package
    go.importpath = github.com/apache/thrift
    repository = https://github.com/apache/thrift
    revision = 4aaa92ece8
    [go_github.com_aws_aws-sdk-go]
    <= go-git-package
    go.importpath = github.com/aws/aws-sdk-go
    repository = https://github.com/aws/aws-sdk-go
    revision = c861d27d03
    [go_github.com_beorn7_perks]
    <= go-git-package
    go.importpath = github.com/beorn7/perks
    repository = https://github.com/beorn7/perks
    revision = 4c0e84591b
    [go_github.com_bmizerany_pat]
    <= go-git-package
    go.importpath = github.com/bmizerany/pat
    repository = https://github.com/bmizerany/pat
    revision = 6226ea591a
    [go_github.com_boltdb_bolt]
    <= go-git-package
    go.importpath = github.com/boltdb/bolt
    repository = https://github.com/boltdb/bolt
    revision = 9da3174536
    [go_github.com_bsm_sarama-cluster]
    <= go-git-package
    go.importpath = github.com/bsm/sarama-cluster
    repository = https://github.com/bsm/sarama-cluster
    revision = v1.0.0-164-gabf039439f
    [go_github.com_cenkalti_backoff]
    <= go-git-package
    go.importpath = github.com/cenkalti/backoff
    repository = https://github.com/cenkalti/backoff
    revision = b02f2bbce1
    [go_github.com_cespare_xxhash]
    <= go-git-package
    go.importpath = github.com/cespare/xxhash
    repository = https://github.com/cespare/xxhash
    revision = e4e2bd419c
    [go_github.com_couchbase_go-couchbase]
    <= go-git-package
    go.importpath = github.com/couchbase/go-couchbase
    repository = https://github.com/couchbase/go-couchbase
    revision = bfe555a140
    [go_github.com_couchbase_gomemcached]
    <= go-git-package
    go.importpath = github.com/couchbase/gomemcached
    repository = https://github.com/couchbase/gomemcached
    revision = 4a25d2f4e1
    [go_github.com_couchbase_goutils]
    <= go-git-package
    go.importpath = github.com/couchbase/goutils
    repository = https://github.com/couchbase/goutils
    revision = 5823a0cbaa
    [go_github.com_davecgh_go-spew]
    <= go-git-package
    go.importpath = github.com/davecgh/go-spew
    repository = https://github.com/davecgh/go-spew
    revision = v1.1.0-9-gecdeabc654
    [go_github.com_dgrijalva_jwt-go]
    <= go-git-package
    go.importpath = github.com/dgrijalva/jwt-go
    repository = https://github.com/dgrijalva/jwt-go
    revision = dbeaa9332f
    [go_github.com_dgryski_go-bitstream]
    <= go-git-package
    go.importpath = github.com/dgryski/go-bitstream
    repository = https://github.com/dgryski/go-bitstream
    revision = 7d46cd22db
    [go_github.com_docker_docker]
    <= go-git-package
    go.importpath = github.com/docker/docker
    repository = https://github.com/docker/docker
    revision = v17.03.2-ce-0-gf5ec1e2936
    [go_github.com_docker_go-connections]
    <= go-git-package
    go.importpath = github.com/docker/go-connections
    repository = https://github.com/docker/go-connections
    revision = 990a1a1a70
    [go_github.com_eapache_go-resiliency]
    <= go-git-package
    go.importpath = github.com/eapache/go-resiliency
    repository = https://github.com/eapache/go-resiliency
    revision = b86b1ec0dd
    [go_github.com_eapache_go-xerial-snappy]
    <= go-git-package
    go.importpath = github.com/eapache/go-xerial-snappy
    repository = https://github.com/eapache/go-xerial-snappy
    revision = bb955e01b9
    [go_github.com_eapache_queue]
    <= go-git-package
    go.importpath = github.com/eapache/queue
    repository = https://github.com/eapache/queue
    revision = 44cc805cf1
    [go_github.com_eclipse_paho.mqtt.golang]
    <= go-git-package
    go.importpath = github.com/eclipse/paho.mqtt.golang
    repository = https://github.com/eclipse/paho.mqtt.golang
    revision = d4f545eb10
    [go_github.com_glycerine_go-unsnap-stream]
    <= go-git-package
    go.importpath = github.com/glycerine/go-unsnap-stream
    repository = https://github.com/glycerine/go-unsnap-stream
    revision = 62a9a9eb44
    [go_github.com_go-ini_ini]
    <= go-git-package
    go.importpath = github.com/go-ini/ini
    repository = https://github.com/go-ini/ini
    revision = 9144852efb
    [go_github.com_go-logfmt_logfmt]
    <= go-git-package
    go.importpath = github.com/go-logfmt/logfmt
    repository = https://github.com/go-logfmt/logfmt
    revision = v0.3.0-0-g390ab7935e
    [go_github.com_go-ole_go-ole]
    <= go-git-package
    go.importpath = github.com/go-ole/go-ole
    repository = https://github.com/go-ole/go-ole
    revision = be49f7c077
    [go_github.com_go-sql-driver_mysql]
    <= go-git-package
    go.importpath = github.com/go-sql-driver/mysql
    repository = https://github.com/go-sql-driver/mysql
    revision = v1.0-470-g2e00b5cd70
    [go_github.com_gobwas_glob]
    <= go-git-package
    go.importpath = github.com/gobwas/glob
    repository = https://github.com/gobwas/glob
    revision = v0.2.2-0-gbea32b9cd2
    [go_github.com_gogo_protobuf]
    <= go-git-package
    go.importpath = github.com/gogo/protobuf
    repository = https://github.com/gogo/protobuf
    revision = 160de10b25
    [go_github.com_golang_protobuf]
    <= go-git-package
    go.importpath = github.com/golang/protobuf
    repository = https://github.com/golang/protobuf
    revision = 1e59b77b52
    [go_github.com_golang_snappy]
    <= go-git-package
    go.importpath = github.com/golang/snappy
    repository = https://github.com/golang/snappy
    revision = 553a641470
    [go_github.com_google_go-cmp]
    <= go-git-package
    go.importpath = github.com/google/go-cmp
    repository = https://github.com/google/go-cmp
    revision = f94e52cad9
    [go_github.com_gorilla_mux]
    <= go-git-package
    go.importpath = github.com/gorilla/mux
    repository = https://github.com/gorilla/mux
    revision = 392c28fe23
    go.importpath = github.com/golang/dep
    repository = https://github.com/golang/dep
    revision = 1f7c19e5f5
    [go_github.com_grafana_grafana]
    <= go-git-package
    go.importpath = github.com/grafana/grafana
    repository = https://github.com/grafana/grafana
    revision = v4.6.0-beta1-1360-g9606a34e0a
    [go_github.com_hailocab_go-hostpool]
    <= go-git-package
    go.importpath = github.com/hailocab/go-hostpool
    repository = https://github.com/hailocab/go-hostpool
    revision = e80d13ce29
    revision = v6.3.0-0-g830da0fda0
    [go_github.com_hashicorp_consul]
    [go_github.com_grafana_loki]
    <= go-git-package
    go.importpath = github.com/hashicorp/consul
    repository = https://github.com/hashicorp/consul
    revision = v0.7.3-35-g63d2fc6823
    go.importpath = github.com/grafana/loki
    repository = https://github.com/grafana/loki
    revision = v0.3.0-0-gc673380bb3
    [go_github.com_influxdata_influxdb]
    <= go-git-package
    go.importpath = github.com/influxdata/influxdb
    repository = https://github.com/influxdata/influxdb
    revision = v1.4.0rc0-328-g938db68198
    [go_github.com_influxdata_influxql]
    <= go-git-package
    go.importpath = github.com/influxdata/influxql
    repository = https://github.com/influxdata/influxql
    revision = 851636b092
    [go_github.com_influxdata_tail]
    <= go-git-package
    go.importpath = github.com/influxdata/tail
    repository = https://github.com/influxdata/tail
    revision = v0-95-ga395bf99fe
    revision = v1.7.8rc1-0-ga7afa2ecf1
    [go_github.com_influxdata_telegraf]
    <= go-git-package
    go.importpath = github.com/influxdata/telegraf
    repository = https://github.com/influxdata/telegraf
    revision = 1.5.0-rc1-73-g90b6b760d1
    [go_github.com_influxdata_toml]
    <= go-git-package
    go.importpath = github.com/influxdata/toml
    repository = https://github.com/influxdata/toml
    revision = 5d1d907f22
    [go_github.com_influxdata_usage-client]
    <= go-git-package
    go.importpath = github.com/influxdata/usage-client
    repository = https://github.com/influxdata/usage-client
    revision = 6d38953763
    [go_github.com_influxdata_wlog]
    <= go-git-package
    go.importpath = github.com/influxdata/wlog
    repository = https://github.com/influxdata/wlog
    revision = 7c63b0a71e
    [go_github.com_influxdata_yamux]
    <= go-git-package
    go.importpath = github.com/influxdata/yamux
    repository = https://github.com/influxdata/yamux
    revision = 1f58ded512
    [go_github.com_influxdata_yarpc]
    <= go-git-package
    go.importpath = github.com/influxdata/yarpc
    repository = https://github.com/influxdata/yarpc
    revision = fdd7e84bf3
    [go_github.com_jackc_pgx]
    <= go-git-package
    go.importpath = github.com/jackc/pgx
    repository = https://github.com/jackc/pgx
    revision = 63f58fd32e
    [go_github.com_jmespath_go-jmespath]
    <= go-git-package
    go.importpath = github.com/jmespath/go-jmespath
    repository = https://github.com/jmespath/go-jmespath
    revision = 0.2.2-14-gbd40a432e4
    [go_github.com_jwilder_encoding]
    <= go-git-package
    go.importpath = github.com/jwilder/encoding
    repository = https://github.com/jwilder/encoding
    revision = b4e1701a28
    [go_github.com_kardianos_govendor]
    <= go-git-package
    go.importpath = github.com/kardianos/govendor
    repository = https://github.com/kardianos/govendor
    revision = 274337c49c
    [go_github.com_kardianos_osext]
    <= go-git-package
    go.importpath = github.com/kardianos/osext
    repository = https://github.com/kardianos/osext
    revision = c2c54e542f
    [go_github.com_kardianos_service]
    <= go-git-package
    go.importpath = github.com/kardianos/service
    repository = https://github.com/kardianos/service
    revision = 6d3a0ee7d3
    [go_github.com_kballard_go-shellquote]
    <= go-git-package
    go.importpath = github.com/kballard/go-shellquote
    repository = https://github.com/kballard/go-shellquote
    revision = d8ec1a69a2
    [go_github.com_matttproud_golang_protobuf_extensions]
    <= go-git-package
    go.importpath = github.com/matttproud/golang_protobuf_extensions
    repository = https://github.com/matttproud/golang_protobuf_extensions
    revision = v1.0.0-2-gc12348ce28
    [go_github.com_miekg_dns]
    <= go-git-package
    go.importpath = github.com/miekg/dns
    repository = https://github.com/miekg/dns
    revision = 99f84ae56e
    [go_github.com_mitchellh_mapstructure]
    <= go-git-package
    go.importpath = github.com/mitchellh/mapstructure
    repository = https://github.com/mitchellh/mapstructure
    revision = d0303fe809
    [go_github.com_multiplay_go-ts3]
    <= go-git-package
    go.importpath = github.com/multiplay/go-ts3
    repository = https://github.com/multiplay/go-ts3
    revision = 07477f49b8
    [go_github.com_naoina_go-stringutil]
    <= go-git-package
    go.importpath = github.com/naoina/go-stringutil
    repository = https://github.com/naoina/go-stringutil
    revision = 6b638e95a3
    [go_github.com_nats-io_go-nats]
    <= go-git-package
    go.importpath = github.com/nats-io/go-nats
    repository = https://github.com/nats-io/go-nats
    revision = v1.2.0-73-gea9585611a
    [go_github.com_nats-io_nats]
    <= go-git-package
    go.importpath = github.com/nats-io/nats
    repository = https://github.com/nats-io/nats
    revision = v1.2.0-73-gea9585611a
    [go_github.com_nats-io_nuid]
    <= go-git-package
    go.importpath = github.com/nats-io/nuid
    repository = https://github.com/nats-io/nuid
    revision = 289cccf02c
    [go_github.com_nsqio_go-nsq]
    <= go-git-package
    go.importpath = github.com/nsqio/go-nsq
    repository = https://github.com/nsqio/go-nsq
    revision = v1.0.7-0-geee57a3ac4
    [go_github.com_opencontainers_runc]
    <= go-git-package
    go.importpath = github.com/opencontainers/runc
    repository = https://github.com/opencontainers/runc
    revision = v0.0.5-426-g89ab7f2ccc
    [go_github.com_opentracing-contrib_go-observer]
    <= go-git-package
    go.importpath = github.com/opentracing-contrib/go-observer
    repository = https://github.com/opentracing-contrib/go-observer
    revision = a52f234244
    [go_github.com_opentracing_opentracing-go]
    <= go-git-package
    go.importpath = github.com/opentracing/opentracing-go
    repository = https://github.com/opentracing/opentracing-go
    revision = 1361b9cd60
    [go_github.com_openzipkin_zipkin-go-opentracing]
    <= go-git-package
    go.importpath = github.com/openzipkin/zipkin-go-opentracing
    repository = https://github.com/openzipkin/zipkin-go-opentracing
    revision = 1cafbdfde9
    [go_github.com_peterh_liner]
    <= go-git-package
    go.importpath = github.com/peterh/liner
    repository = https://github.com/peterh/liner
    revision = 3681c2a912
    [go_github.com_philhofer_fwd]
    <= go-git-package
    go.importpath = github.com/philhofer/fwd
    repository = https://github.com/philhofer/fwd
    revision = bb6d471dc9
    [go_github.com_pierrec_lz4]
    <= go-git-package
    go.importpath = github.com/pierrec/lz4
    repository = https://github.com/pierrec/lz4
    revision = v1.0-0-g5c9560bfa9
    [go_github.com_pierrec_xxHash]
    <= go-git-package
    go.importpath = github.com/pierrec/xxHash
    repository = https://github.com/pierrec/xxHash
    revision = 5a004441f8
    [go_github.com_pkg_errors]
    <= go-git-package
    go.importpath = github.com/pkg/errors
    repository = https://github.com/pkg/errors
    revision = v0.8.0-0-g645ef00459
    [go_github.com_pmezard_go-difflib]
    <= go-git-package
    go.importpath = github.com/pmezard/go-difflib
    repository = https://github.com/pmezard/go-difflib
    revision = v1.0.0-0-g792786c740
    [go_github.com_prometheus_client_golang]
    <= go-git-package
    go.importpath = github.com/prometheus/client_golang
    repository = https://github.com/prometheus/client_golang
    revision = v0.9.0-pre1-21-g180b8fdc22
    [go_github.com_prometheus_client_model]
    <= go-git-package
    go.importpath = github.com/prometheus/client_model
    repository = https://github.com/prometheus/client_model
    revision = model-0.0.2-16-g99fa1f4be8
    [go_github.com_prometheus_common]
    <= go-git-package
    go.importpath = github.com/prometheus/common
    repository = https://github.com/prometheus/common
    revision = 89604d1970
    [go_github.com_prometheus_procfs]
    <= go-git-package
    go.importpath = github.com/prometheus/procfs
    repository = https://github.com/prometheus/procfs
    revision = b15cd069a8
    [go_github.com_rcrowley_go-metrics]
    <= go-git-package
    go.importpath = github.com/rcrowley/go-metrics
    repository = https://github.com/rcrowley/go-metrics
    revision = 1f30fe9094
    [go_github.com_retailnext_hllpp]
    <= go-git-package
    go.importpath = github.com/retailnext/hllpp
    repository = https://github.com/retailnext/hllpp
    revision = v1.0.0-2-g6e8b6d3944
    [go_github.com_samuel_go-zookeeper]
    <= go-git-package
    go.importpath = github.com/samuel/go-zookeeper
    repository = https://github.com/samuel/go-zookeeper
    revision = 1d7be4effb
    [go_github.com_satori_go.uuid]
    <= go-git-package
    go.importpath = github.com/satori/go.uuid
    repository = https://github.com/satori/go.uuid
    revision = v1.1.0-8-g5bf94b69c6
    [go_github.com_shirou_gopsutil]
    <= go-git-package
    go.importpath = github.com/shirou/gopsutil
    repository = https://github.com/shirou/gopsutil
    revision = v2.16.10-243-g384a55110a
    [go_github.com_shirou_w32]
    <= go-git-package
    go.importpath = github.com/shirou/w32
    repository = https://github.com/shirou/w32
    revision = 3c9377fc67
    [go_github.com_soniah_gosnmp]
    <= go-git-package
    go.importpath = github.com/soniah/gosnmp
    repository = https://github.com/soniah/gosnmp
    revision = v1.0-204-g5ad50dc75a
    [go_github.com_sparrc_gdm]
    <= go-git-package
    go.importpath = github.com/sparrc/gdm
    repository = https://github.com/sparrc/gdm
    revision = 1.4-4-g81089dabfa
    [go_github.com_streadway_amqp]
    <= go-git-package
    go.importpath = github.com/streadway/amqp
    repository = https://github.com/streadway/amqp
    revision = 63795daa9a
    [go_github.com_stretchr_objx]
    <= go-git-package
    go.importpath = github.com/stretchr/objx
    repository = https://github.com/stretchr/objx
    revision = 1a9d0bb9f5
    [go_github.com_stretchr_testify]
    <= go-git-package
    go.importpath = github.com/stretchr/testify
    repository = https://github.com/stretchr/testify
    revision = v1.0-187-g4d4bfba8f1
    [go_github.com_tidwall_gjson]
    <= go-git-package
    go.importpath = github.com/tidwall/gjson
    repository = https://github.com/tidwall/gjson
    revision = 0623bd8fbd
    [go_github.com_tidwall_match]
    <= go-git-package
    go.importpath = github.com/tidwall/match
    repository = https://github.com/tidwall/match
    revision = 173748da73
    [go_github.com_tinylib_msgp]
    <= go-git-package
    go.importpath = github.com/tinylib/msgp
    repository = https://github.com/tinylib/msgp
    revision = 428e467e72
    [go_github.com_vjeantet_grok]
    <= go-git-package
    go.importpath = github.com/vjeantet/grok
    repository = https://github.com/vjeantet/grok
    revision = d73e972b60
    [go_github.com_wvanbergen_kafka]
    <= go-git-package
    go.importpath = github.com/wvanbergen/kafka
    repository = https://github.com/wvanbergen/kafka
    revision = bc265fedb9
    [go_github.com_wvanbergen_kazoo-go]
    <= go-git-package
    go.importpath = github.com/wvanbergen/kazoo-go
    repository = https://github.com/wvanbergen/kazoo-go
    revision = 9689573521
    [go_github.com_xlab_treeprint]
    <= go-git-package
    go.importpath = github.com/xlab/treeprint
    repository = https://github.com/xlab/treeprint
    revision = 06dfc6fa17
    [go_github.com_yuin_gopher-lua]
    <= go-git-package
    go.importpath = github.com/yuin/gopher-lua
    repository = https://github.com/yuin/gopher-lua
    revision = 66c871e454
    [go_github.com_zensqlmonitor_go-mssqldb]
    <= go-git-package
    go.importpath = github.com/zensqlmonitor/go-mssqldb
    repository = https://github.com/zensqlmonitor/go-mssqldb
    revision = ffe5510c6f
    [go_go.uber.org_atomic]
    <= go-git-package
    go.importpath = go.uber.org/atomic
    repository = https://github.com/uber-go/atomic
    revision = v1.3.1-0-g8474b86a5a
    [go_go.uber.org_multierr]
    <= go-git-package
    go.importpath = go.uber.org/multierr
    repository = https://github.com/uber-go/multierr
    revision = v1.1.0-1-gfb7d312c2c
    [go_go.uber.org_zap]
    <= go-git-package
    go.importpath = go.uber.org/zap
    repository = https://github.com/uber-go/zap
    revision = v1.7.1-11-gf85c78b1dd
    [go_golang.org_x_crypto]
    <= go-git-package
    go.importpath = golang.org/x/crypto
    repository = https://go.googlesource.com/crypto
    revision = b3c9a1d25c
    [go_golang.org_x_net]
    <= go-git-package
    go.importpath = golang.org/x/net
    repository = https://go.googlesource.com/net
    revision = ab555f366c
    [go_golang.org_x_sys]
    <= go-git-package
    go.importpath = golang.org/x/sys
    repository = https://go.googlesource.com/sys
    revision = 810d700034
    [go_golang.org_x_text]
    <= go-git-package
    go.importpath = golang.org/x/text
    repository = https://go.googlesource.com/text
    revision = e19ae14969
    [go_golang.org_x_time]
    <= go-git-package
    go.importpath = golang.org/x/time
    repository = https://go.googlesource.com/time
    revision = 6dc17368e0
    [go_golang.org_x_tools]
    <= go-git-package
    go.importpath = golang.org/x/tools
    repository = https://go.googlesource.com/tools
    revision = fbec762f83
    [go_gopkg.in_asn1-ber.v1]
    <= go-git-package
    go.importpath = gopkg.in/asn1-ber.v1
    repository = https://gopkg.in/asn1-ber.v1
    revision = 4e86f43671
    [go_gopkg.in_fatih_pool.v2]
    <= go-git-package
    go.importpath = gopkg.in/fatih/pool.v2
    repository = https://gopkg.in/fatih/pool.v2
    revision = 6e328e6789
    [go_gopkg.in_fsnotify.v1]
    <= go-git-package
    go.importpath = gopkg.in/fsnotify.v1
    repository = https://gopkg.in/fsnotify.v1
    revision = a8a77c9133
    [go_gopkg.in_gorethink_gorethink.v3]
    <= go-git-package
    go.importpath = gopkg.in/gorethink/gorethink.v3
    repository = https://gopkg.in/gorethink/gorethink.v3
    revision = v2.2.1-35-g7ab832f7b6
    [go_gopkg.in_ldap.v2]
    <= go-git-package
    go.importpath = gopkg.in/ldap.v2
    repository = https://gopkg.in/ldap.v2
    revision = 8168ee085e
    [go_gopkg.in_mgo.v2]
    <= go-git-package
    go.importpath = gopkg.in/mgo.v2
    repository = https://gopkg.in/mgo.v2
    revision = 3f83fa5005
    [go_gopkg.in_olivere_elastic.v5]
    <= go-git-package
    go.importpath = gopkg.in/olivere/elastic.v5
    repository = https://gopkg.in/olivere/elastic.v5
    revision = v5.0.41-0-g3113f9b9ad
    [go_gopkg.in_tomb.v1]
    <= go-git-package
    go.importpath = gopkg.in/tomb.v1
    repository = https://gopkg.in/tomb.v1
    revision = dd632973f1
    [go_gopkg.in_yaml.v2]
    <= go-git-package
    go.importpath = gopkg.in/yaml.v2
    repository = https://gopkg.in/yaml.v2
    revision = 4c78c975fe
    revision = 1.11.1-0-gf91de60d37
    software/grafana/grafana-provisioning-config-file.cfg.in 0 → 100644
    View file @ 9e5f148c
    # https://grafana.com/docs/administration/provisioning/#example-datasource-config-file
    apiVersion: 1
    datasources:
    - name: telegraf
    type: influxdb
    access: proxy
    url: {{ influxdb['url'] }}
    user: {{ influxdb['auth-username'] }}
    database: telegraf
    isDefault: true
    jsonData:
    tlsSkipVerify: true
    secureJsonData:
    password: {{ influxdb['auth-password'] }}
    version: 1
    editable: false
    - name: loki
    type: loki
    access: proxy
    url: {{ loki['url'] }}
    version: 1
    editable: false
    software/grafana/instance-input-schema.json
    View file @ 9e5f148c
    ......@@ -28,6 +28,11 @@
    "description": "Name used in From: header of emails",
    "default": "Grafana",
    "type": "string"
    },
    "promtail-extra-scrape-config": {
    "description": "Raw promtail config (experimental parameter, see https://github.com/grafana/loki/blob/v0.3.0/docs/promtail.md#scrape-configs for detail)",
    "default": "",
    "type": "string"
    }
    }
    }
    software/grafana/instance.cfg.in
    View file @ 9e5f148c
    ......@@ -3,8 +3,6 @@ parts =
    promises
    publish-connection-parameter
    extends = {{ monitor_template }}
    eggs-directory = {{ buildout['eggs-directory'] }}
    develop-eggs-directory = {{ buildout['develop-eggs-directory'] }}
    offline = true
    ......@@ -35,17 +33,21 @@ etc = ${:home}/etc
    var = ${:home}/var
    srv = ${:home}/srv
    service = ${:etc}/service
    promise = ${:etc}/promise
    influxdb-data-dir = ${:srv}/influxdb
    grafana-dir = ${:srv}/grafana
    grafana-data-dir = ${:grafana-dir}/data
    grafana-logs-dir = ${:var}/log
    grafana-plugins-dir = ${:grafana-dir}/plugins
    grafana-provisioning-config-dir = ${:grafana-dir}/provisioning-config
    grafana-provisioning-datasources = ${:grafana-provisioning-config-dir}/datasources
    grafana-provisioning-dashboards = ${:grafana-provisioning-config-dir}/dashboards
    grafana-provisioning-datasources-dir = ${:grafana-provisioning-config-dir}/datasources
    grafana-provisioning-dashboards-dir = ${:grafana-provisioning-config-dir}/dashboards
    telegraf-dir = ${:srv}/telegraf
    telegraf-extra-config-dir = ${:telegraf-dir}/extra-config
    loki-dir = ${:srv}/loki
    loki-storage-boltdb-dir = ${:loki-dir}/index/
    loki-storage-filesystem-dir = ${:loki-dir}/chunks/
    promtail-dir = ${:srv}/promtail
    # macros
    [generate-certificate]
    ......@@ -71,11 +73,14 @@ mode = 0644
    extensions = jinja2.ext.do
    [check-port-listening-promise]
    <= monitor-base-promise
    module = check_port_listening
    name = ${:_buildout_section_name_}.py
    recipe = slapos.cookbook:check_port_listening
    path = ${directory:promise}/${:_buildout_section_name_}
    [check-url-available-promise]
    recipe = slapos.cookbook:check_url_available
    path = ${directory:promise}/${:_buildout_section_name_}
    dash_path = {{ dash_bin }}
    curl_path = {{ curl_bin }}
    [influxdb]
    ipv6 = ${instance-parameter:ipv6-random}
    ......@@ -112,27 +117,27 @@ username = influxdb
    [influxdb-listen-promise]
    <= check-port-listening-promise
    config-hostname = ${influxdb:ipv6}
    config-port = ${influxdb:http-port}
    hostname = ${influxdb:ipv6}
    port = ${influxdb:http-port}
    [influxdb-password-promise]
    <= monitor-base-promise
    module = check_command_execute
    name = ${:_buildout_section_name_}.py
    config-command =
    recipe = slapos.cookbook:wrapper
    command-line =
    {{ influx_bin }} -username ${influxdb:auth-username} -password ${influxdb:auth-password} -socket ${influxdb:unix-socket} -execute "CREATE USER ${influxdb:auth-username} WITH PASSWORD '${influxdb:auth-password}' WITH ALL PRIVILEGES"
    wrapper-path = ${directory:promise}/${:_buildout_section_name_}
    [grafana]
    ipv6 = ${instance-parameter:ipv6-random}
    port = 8080
    port = 8180
    url = https://[${:ipv6}]:${:port}
    data-dir = ${directory:grafana-data-dir}
    logs-dir = ${directory:grafana-logs-dir}
    plugins-dir = ${directory:grafana-plugins-dir}
    provisioning-config-dir = ${directory:grafana-provisioning-config-dir}
    provisioning-datasources-dir = ${directory:grafana-provisioning-datasources-dir}
    admin-user = ${grafana-password:username}
    admin-password = ${grafana-password:passwd}
    secret-key = ${grafana-secret-key:passwd}
    ......@@ -160,11 +165,20 @@ context =
    section grafana grafana
    section apache_frontend apache-frontend
    key slapparameter_dict slap-configuration:configuration
    depends =
    ${grafana-provisioning-config-file:rendered}
    [grafana-provisioning-config-file]
    <= config-file
    rendered = ${grafana:provisioning-datasources-dir}/datasource.yaml
    context =
    section influxdb influxdb
    section loki loki
    [grafana-listen-promise]
    <= check-port-listening-promise
    config-hostname= ${grafana:ipv6}
    config-port = ${grafana:port}
    hostname= ${grafana:ipv6}
    port = ${grafana:port}
    ......@@ -183,6 +197,53 @@ context =
    section telegraf telegraf
    [loki]
    recipe = slapos.cookbook:wrapper
    command-line =
    bash -c 'nice -19 chrt --idle 0 ionice -c3 {{ loki_bin }} -config.file=${loki-config-file:rendered}'
    wrapper-path = ${directory:service}/loki
    storage-boltdb-dir = ${directory:loki-storage-boltdb-dir}
    storage-filesystem-dir = ${directory:loki-storage-filesystem-dir}
    ip = ${instance-parameter:ipv4-random}
    port = 3100
    url = http://${:ip}:${:port}
    [loki-config-file]
    <= config-file
    context =
    section loki loki
    [loki-listen-promise]
    <= check-url-available-promise
    url = ${loki:url}/ready
    [promtail]
    recipe = slapos.cookbook:wrapper
    command-line =
    bash -c 'nice -19 chrt --idle 0 ionice -c3 {{ promtail_bin }} -config.file=${promtail-config-file:rendered}'
    wrapper-path = ${directory:service}/promtail
    dir = ${directory:promtail-dir}
    http_port = 19080
    ip = ${instance-parameter:ipv4-random}
    url = http://${:ip}:${:http_port}
    [promtail-config-file]
    <= config-file
    context =
    section promtail promtail
    section loki loki
    key slapparameter_dict slap-configuration:configuration
    [promtail-listen-promise]
    <= check-port-listening-promise
    hostname= ${promtail:ip}
    port = ${promtail:http_port}
    [apache-frontend]
    <= slap-connection
    recipe = slapos.cookbook:requestoptional
    ......@@ -201,6 +262,8 @@ instance-promises =
    ${influxdb-listen-promise:path}
    ${influxdb-password-promise:wrapper-path}
    ${grafana-listen-promise:path}
    ${loki-listen-promise:path}
    ${promtail-listen-promise:path}
    [publish-connection-parameter]
    ......@@ -213,4 +276,6 @@ telegraf-extra-config-dir = ${telegraf:extra-config-dir}
    grafana-url = ${grafana:url}
    grafana-username = ${grafana:admin-user}
    grafana-password = ${grafana:admin-password}
    loki-url = ${loki:url}
    promtail-url = ${promtail:url}
    url = ${apache-frontend:connection-secure_access}
    software/grafana/loki-config-file.cfg.in 0 → 100644
    View file @ 9e5f148c
    auth_enabled: false
    server:
    http_listen_port: {{ loki['port'] }}
    ingester:
    lifecycler:
    address: {{ loki['ip'] }}
    ring:
    kvstore:
    store: inmemory
    replication_factor: 1
    chunk_idle_period: 15m
    schema_config:
    configs:
    - from: 2018-04-15
    store: boltdb
    object_store: filesystem
    schema: v9
    index:
    prefix: index_
    period: 168h
    storage_config:
    boltdb:
    directory: {{ loki['storage-boltdb-dir'] }}
    filesystem:
    directory: {{ loki['storage-filesystem-dir'] }}
    limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 168h
    chunk_store_config:
    max_look_back_period: 0
    table_manager:
    chunk_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
    index_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
    retention_deletes_enabled: false
    retention_period: 0
    software/grafana/promtail-config-file.cfg.in 0 → 100644
    View file @ 9e5f148c
    # https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md
    server:
    http_listen_port: {{ promtail['http_port'] }}
    # XXX this external_url does not work ... promtail still listen on all IPs
    external_url: {{ promtail['url'] }}
    grpc_listen_port: 0
    positions:
    filename: {{ promtail['dir'] }}/positions.yaml
    clients:
    - url: {{ loki['url'] }}/api/prom/push
    scrape_configs:
    - job_name: test
    static_configs:
    - targets:
    - localhost
    labels:
    job: grafanalogs
    __path__: ./var/log/*log
    {{ slapparameter_dict.get('promtail-extra-scrape-config', '') }}
    software/grafana/software.cfg
    View file @ 9e5f148c
    ......@@ -2,10 +2,11 @@
    extends =
    ../../stack/slapos.cfg
    ../../stack/nodejs.cfg
    ../../stack/monitor/buildout.cfg
    ../../component/make/buildout.cfg
    ../../component/golang/buildout.cfg
    ../../component/openssl/buildout.cfg
    ../../component/curl/buildout.cfg
    ../../component/dash/buildout.cfg
    buildout.hash.cfg
    gowork.cfg
    ......@@ -17,21 +18,24 @@ parts =
    influxdb-config-file
    telegraf-config-file
    grafana-config-file
    grafana-provisioning-config-file
    loki-config-file
    promtail-config-file
    [nodejs]
    <= nodejs-8.6.0
    <= nodejs-10.6.0
    [yarn]
    # this could become a component, but it needs to be invoked from nodejs explicitly,
    # otherwise it uses system's nodejs
    recipe = slapos.recipe.build:download-unpacked
    url = https://github.com/yarnpkg/yarn/releases/download/v1.3.2/yarn-v1.3.2.tar.gz
    md5sum = db82fa09c996e9318f2f1d2ab99228f9
    url = https://github.com/yarnpkg/yarn/releases/download/v1.16.0/yarn-v1.16.0.tar.gz
    md5sum = 46790033c23803387890f545e4040690
    [gowork]
    # All the softwares installed in the go work have "non standard" installation
    # All the softwares installed in the go workspace have "non standard" installation
    # methods, so we install them in specific parts with custom commands.
    # They will be installed because they are dependencies of ${gowork.goinstall}
    install =
    ......@@ -41,6 +45,12 @@ influx-bin = ${:bin}/influx
    influxd-bin = ${:bin}/influxd
    grafana-bin = ${:bin}/grafana-server
    grafana-homepath = ${go_github.com_grafana_grafana:location}
    loki-bin = ${:bin}/loki
    promtail-bin = ${:bin}/promtail
    # use recent go
    golang = ${golang1.12:location}
    gcc-bin-directory = ${golang1.12:gcc-bin-directory}
    [gowork.goinstall]
    command = :
    ......@@ -48,17 +58,22 @@ depends =
    ${influxdb-install:recipe}
    ${telegraf-install:recipe}
    ${grafana-install:recipe}
    ${loki-install:recipe}
    ${promtail-install:recipe}
    [influxdb-install]
    <= gowork.goinstall
    command = bash -c ". ${gowork:env.sh} && \
    go install -v github.com/golang/dep/cmd/dep && \
    cd ${gowork:directory}/src/github.com/influxdata/influxdb && \
    dep ensure && \
    go install ./cmd/..."
    update-command =
    [telegraf-install]
    <= gowork.goinstall
    command = bash -c ". ${gowork:env.sh} && \
    go install -v github.com/golang/dep/cmd/dep && \
    cd ${gowork:directory}/src/github.com/influxdata/telegraf && \
    ${make:location}/bin/make &&
    cp telegraf ${gowork:bin}"
    ......@@ -70,12 +85,32 @@ update-command =
    command = bash -c "export PATH=${nodejs:location}/bin/:$PATH && \
    . ${gowork:env.sh} && \
    cd ${gowork:directory}/src/github.com/grafana/grafana && \
    ${gowork:golang}/bin/go run build.go setup && \
    ${gowork:golang}/bin/go run build.go build && \
    go run build.go setup && \
    go run build.go build && \
    ${yarn:location}/bin/yarn install --pure-lockfile && \
    ${nodejs:location}/bin/npm run build"
    update-command =
    [loki-install]
    <= gowork.goinstall
    # loki also uses nodejs
    command = bash -c "export PATH=${nodejs:location}/bin/:$PATH && \
    . ${gowork:env.sh} && \
    go install -v github.com/golang/dep/cmd/dep && \
    cd ${gowork:directory}/src/github.com/grafana/loki && \
    go install ./cmd/loki"
    update-command =
    [promtail-install]
    <= gowork.goinstall
    # CGO_ENABLED is to disable systemd support (did not compile in my case)
    command = bash -c "export CGO_ENABLED=0 && \
    . ${gowork:env.sh} && \
    go install -v github.com/golang/dep/cmd/dep && \
    cd ${gowork:directory}/src/github.com/grafana/loki && \
    go install ./cmd/promtail"
    update-command =
    [download-file-base]
    recipe = slapos.recipe.build:download
    ......@@ -92,6 +127,15 @@ mode = 0644
    [grafana-config-file]
    <= download-file-base
    [grafana-provisioning-config-file]
    <= download-file-base
    [loki-config-file]
    <= download-file-base
    [promtail-config-file]
    <= download-file-base
    [instance-profile]
    recipe = slapos.recipe.template:jinja2
    template = ${:_profile_base_location_}/${:filename}
    ......@@ -106,10 +150,13 @@ context =
    key influx_bin gowork:influx-bin
    key grafana_bin gowork:grafana-bin
    key grafana_homepath gowork:grafana-homepath
    key monitor_template monitor2-template:rendered
    key loki_bin gowork:loki-bin
    key promtail_bin gowork:promtail-bin
    key curl_bin :curl-bin
    key dash_bin :dash-bin
    curl-bin = ${curl:location}/bin/curl
    dash-bin = ${dash:location}/bin/dash
    [versions]
    slapos.recipe.template = 4.2
    inotifyx = 0.2.2
    software/grafana/test/README.md 0 → 100644
    View file @ 9e5f148c
    Tests for Grafana software release
    software/grafana/test/setup.py 0 → 100644
    View file @ 9e5f148c
    ##############################################################################
    #
    # Copyright (c) 2018 Nexedi SA and Contributors. All Rights Reserved.
    #
    # WARNING: This program as such is intended to be used by professional
    # programmers who take the whole responsibility of assessing all potential
    # consequences resulting from its eventual inadequacies and bugs
    # End users who are looking for a ready-to-use solution with commercial
    # guarantees and support are strongly adviced to contract a Free Software
    # Service Company
    #
    # This program is Free Software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 3
    # of the License, or (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
    #
    ##############################################################################
    from setuptools import setup, find_packages
    version = '0.0.1.dev0'
    name = 'slapos.test.grafana'
    long_description = open("README.md").read()
    setup(
    name=name,
    version=version,
    description="Test for SlapOS' Grafana",
    long_description=long_description,
    long_description_content_type='text/markdown',
    maintainer="Nexedi",
    maintainer_email="info@nexedi.com",
    url="https://lab.nexedi.com/nexedi/slapos",
    packages=find_packages(),
    install_requires=[
    'slapos.core',
    'slapos.libnetworkcache',
    'erp5.util',
    'requests',
    'supervisor',
    'psutil',
    ],
    zip_safe=True,
    test_suite='test',
    )
    software/grafana/test/test.py 0 → 100644
    View file @ 9e5f148c
    ##############################################################################
    #
    # Copyright (c) 2019 Nexedi SA and Contributors. All Rights Reserved.
    #
    # WARNING: This program as such is intended to be used by professional
    # programmers who take the whole responsibility of assessing all potential
    # consequences resulting from its eventual inadequacies and bugs
    # End users who are looking for a ready-to-use solution with commercial
    # guarantees and support are strongly adviced to contract a Free Software
    # Service Company
    #
    # This program is Free Software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 3
    # of the License, or (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
    #
    ##############################################################################
    import os
    import textwrap
    import logging
    import tempfile
    import time
    import requests
    from slapos.testing.testcase import makeModuleSetUpAndTestCaseClass
    setUpModule, SlapOSInstanceTestCase = makeModuleSetUpAndTestCaseClass(
    os.path.abspath(
    os.path.join(os.path.dirname(__file__), '..', 'software.cfg')))
    class GrafanaTestCase(SlapOSInstanceTestCase):
    """Base test case for grafana.
    Since the instances takes timte to start and stop,
    we increate as lot the number of retries.
    """
    report_max_retry = 30
    instance_max_retry = 30
    class TestGrafana(GrafanaTestCase):
    def setUp(self):
    self.grafana_url = self.computer_partition.getConnectionParameterDict(
    )['grafana-url']
    def test_grafana_available(self):
    resp = requests.get(self.grafana_url, verify=False)
    self.assertEqual(requests.codes.ok, resp.status_code)
    def test_grafana_api(self):
    # check API is usable
    api_org_url = '{self.grafana_url}/api/org'.format(**locals())
    resp = requests.get(api_org_url, verify=False)
    self.assertEqual(requests.codes.unauthorized, resp.status_code)
    connection_params = self.computer_partition.getConnectionParameterDict()
    resp = requests.get(
    api_org_url,
    verify=False,
    auth=requests.auth.HTTPBasicAuth(
    connection_params['grafana-username'],
    connection_params['grafana-password'],
    ))
    self.assertEqual(requests.codes.ok, resp.status_code)
    self.assertEqual(1, resp.json()['id'])
    def test_grafana_datasource_povisinonned(self):
    # data sources are provisionned
    connection_params = self.computer_partition.getConnectionParameterDict()
    resp = requests.get(
    '{self.grafana_url}/api/datasources'.format(**locals()),
    verify=False,
    auth=requests.auth.HTTPBasicAuth(
    connection_params['grafana-username'],
    connection_params['grafana-password'],
    ))
    self.assertEqual(requests.codes.ok, resp.status_code)
    self.assertEqual(
    sorted(['influxdb', 'loki']),
    sorted([ds['type'] for ds in resp.json()]))
    class TestInfluxDb(GrafanaTestCase):
    def setUp(self):
    self.influxdb_url = self.computer_partition.getConnectionParameterDict(
    )['influxdb-url']
    def test_influxdb_available(self):
    ping_url = '{self.influxdb_url}/ping'.format(**locals())
    resp = requests.get(ping_url, verify=False)
    self.assertEqual(requests.codes.no_content, resp.status_code)
    def test_influxdb_api(self):
    query_url = '{self.influxdb_url}/query'.format(**locals())
    connection_params = self.computer_partition.getConnectionParameterDict()
    for i in range(10):
    # retry, as it may take a little delay to create databases
    resp = requests.get(
    query_url,
    verify=False,
    params=dict(
    q='SHOW DATABASES',
    u=connection_params['influxdb-username'],
    p=connection_params['influxdb-password']))
    self.assertEqual(requests.codes.ok, resp.status_code)
    result, = resp.json()['results']
    if result['series'] and 'values' in result['series'][0]:
    break
    time.sleep(0.5 * i)
    self.assertIn(
    [connection_params['influxdb-database']], result['series'][0]['values'])
    class TestTelegraf(GrafanaTestCase):
    def test_telegraf_running(self):
    with self.slap.instance_supervisor_rpc as supervisor:
    all_process_info = supervisor.getAllProcessInfo()
    process_info, = [p for p in all_process_info if 'telegraf' in p['name']]
    self.assertEqual('RUNNING', process_info['statename'])
    class TestLoki(GrafanaTestCase):
    @classmethod
    def getInstanceParameterDict(cls):
    cls._logfile = tempfile.NamedTemporaryFile(suffix='log')
    return {
    'promtail-extra-scrape-config':
    textwrap.dedent(
    r'''
    - job_name: {cls.__name__}
    pipeline_stages:
    - regex:
    expression: "^(?P<timestamp>.*) - (?P<name>\\S+) - (?P<level>\\S+) - (?P<message>.*)"
    - timestamp:
    format: 2006-01-02T15:04:05Z00:00
    source: timestamp
    - labels:
    level:
    name:
    static_configs:
    - targets:
    - localhost
    labels:
    job: {cls.__name__}
    __path__: {cls._logfile.name}
    ''').format(**locals())
    }
    @classmethod
    def tearDownClass(cls):
    cls._logfile.close()
    super(TestLoki, cls).tearDownClass()
    def setUp(self):
    self.loki_url = self.computer_partition.getConnectionParameterDict(
    )['loki-url']
    def test_loki_available(self):
    self.assertEqual(
    requests.codes.ok,
    requests.get('{self.loki_url}/ready'.format(**locals()),
    verify=False).status_code)
    def test_log_ingested(self):
    # create a logger logging to the file that we have
    # configured in instance parameter.
    test_logger = logging.getLogger(self.id())
    test_logger.setLevel(logging.INFO)
    test_handler = logging.FileHandler(filename=self._logfile.name)
    test_handler.setFormatter(
    logging.Formatter(
    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
    test_logger.addHandler(test_handler)
    test_logger.info("testing message")
    test_logger.info("testing another message")
    test_logger.warning("testing warn")
    # Check our messages have been ingested
    # we retry a few times, because there's a short delay until messages are
    # ingested and returned.
    for i in range(10):
    resp = requests.get(
    '{self.loki_url}/api/prom/query?query={{job="TestLoki"}}'.format(
    **locals()),
    verify=False).json()
    if not resp:
    time.sleep(0.5 * i)
    continue
    warn_stream_list = [stream for stream in resp['streams'] if 'level="WARNING"' in stream['labels']]
    self.assertEqual(1, len(warn_stream_list), resp['streams'])
    warn_stream, = warn_stream_list
    self.assertIn("testing warn", warn_stream['entries'][0]['line'])
    info_stream_list = [stream for stream in resp['streams'] if 'level="INFO"' in stream['labels']]
    self.assertEqual(1, len(info_stream_list), resp['streams'])
    info_stream, = info_stream_list
    self.assertTrue(
    [
    line for line in info_stream['entries']
    if "testing message" in line['line']
    ])
    self.assertTrue(
    [
    line for line in info_stream['entries']
    if "testing another message" in line['line']
    ])
    # The labels we have configued are also available
    resp = requests.get(
    '{self.loki_url}/api/prom/label'.format(**locals()),
    verify=False).json()
    self.assertIn('level', resp['values'])
    self.assertIn('name', resp['values'])
    software/slapos-sr-testing/buildout.hash.cfg
    View file @ 9e5f148c
    ......@@ -15,4 +15,4 @@
    [template]
    filename = instance.cfg
    md5sum = 4664f7dae66d3f582e34cec2ca627501
    md5sum = 1cbab58e896ff63575f6a67db530d183
    software/slapos-sr-testing/instance.cfg
    View file @ 9e5f148c
    ......@@ -28,7 +28,7 @@ bin = $${buildout:directory}/bin
    working-dir = $${buildout:directory}/tmp
    [test-list]
    path_list = ${slapos.cookbook-setup:setup},${slapos.test.caddy-frontend-setup:setup},${slapos.test.erp5-setup:setup},${slapos.test.slapos-master-setup:setup},${slapos.test.kvm-setup:setup},${slapos.test.monitor-setup:setup},${slapos.test.plantuml-setup:setup},${slapos.test.powerdns-setup:setup},${slapos.test.proftpd-setup:setup},${slapos.test.re6stnet-setup:setup},${slapos.test.seleniumserver-setup:setup},${slapos.test.slaprunner-setup:setup},${slapos.test.helloworld-setup:setup},${slapos.test.jupyter-setup:setup},${slapos.test.nextcloud-setup:setup},${slapos.test.turnserver-setup:setup},${slapos.test.theia-setup:setup}
    path_list = ${slapos.cookbook-setup:setup},${slapos.test.caddy-frontend-setup:setup},${slapos.test.erp5-setup:setup},${slapos.test.slapos-master-setup:setup},${slapos.test.kvm-setup:setup},${slapos.test.monitor-setup:setup},${slapos.test.plantuml-setup:setup},${slapos.test.powerdns-setup:setup},${slapos.test.proftpd-setup:setup},${slapos.test.re6stnet-setup:setup},${slapos.test.seleniumserver-setup:setup},${slapos.test.slaprunner-setup:setup},${slapos.test.helloworld-setup:setup},${slapos.test.jupyter-setup:setup},${slapos.test.nextcloud-setup:setup},${slapos.test.turnserver-setup:setup},${slapos.test.theia-setup:setup},${slapos.test.grafana-setup:setup}
    [slapos-test-runner]
    recipe = slapos.cookbook:wrapper
    ......
    software/slapos-sr-testing/software.cfg
    View file @ 9e5f148c
    ......@@ -107,6 +107,11 @@ setup = ${slapos-repository:location}/software/turnserver/test/
    egg = slapos.test.theia
    setup = ${slapos-repository:location}/software/theia/test/
    [slapos.test.grafana-setup]
    <= setup-develop-egg
    egg = slapos.test.grafana
    setup = ${slapos-repository:location}/software/grafana/test/
    [slapos.core-repository]
    <= git-clone-repository
    repository = https://lab.nexedi.com/nexedi/slapos.core.git
    ......
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment
    GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7