# How to measure and monitor an inlets exit server on Oracle Cloud Infrastructure

>This is an update of the existing how-to article, about running an exit server on Oracles Cloud Infrastructure. -> If you missed it here is the [link](https://blog.ediri.io/how-to-run-an-inlets-exit-server-on-oracle-cloud-infrastructure)

That is the tweet, le raison d'etre for this how-to! 

%[https://twitter.com/alexellisuk/status/1429009127846469638?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1429009127846469638%7Ctwgr%5E%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fpublish.twitter.com%2F%3Fquery%3Dhttps3A2F2Ftwitter.com2Falexellisuk2Fstatus2F1429009127846469638widget%3DTweet]

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749567753/buGodMknu.png)

I am a "crazy" for all things monitoring related. I just could not miss this opportunity.

Lets us change the `terraform` code from the last time, to get ready for the upcoming major update of Inlets v0.9.0. 

BTW: [Alex Ellis](https://twitter.com/alexellisuk) covered all the upcoming new functionalities in his own blog [post](https://inlets.dev/blog/2021/08/18/measure-and-monitor.html). So go and check this out.

What did I changed in the code? Nothing on the infrastructure part. 

We still run on OCI, the only difference is that we now deploy in the cloud-init the Grafana agent to pump our data into the [Grafana Cloud](https://grafana.com/products/cloud/).

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749841411/tGpJwqDlf.png)

Here is the small addition in the `startup` skript. Check the [github repo](https://github.com/dirien/inlets-oci-terraform) for in-depth details.

```bash
mkdir /grafana
tee /grafana/agent.yaml <<EOF
server:
  log_level: info
  http_listen_port: 12345
...
EOF

tee /etc/systemd/system/grafana-agent.service <<EOF
...
EOF
...
apt install -y unzip
curl -SLsf https://github.com/grafana/agent/releases/download/v0.18.2/agent-linux-amd64.zip -o /tmp/agent-linux-amd64.zip && \
  unzip /tmp/agent-linux-amd64.zip -d /tmp && \
  chmod a+x /tmp/agent-linux-amd64 && \
  mv /tmp/agent-linux-amd64 /usr/local/bin/agent-linux-amd64

systemctl restart grafana-agent.service
systemctl enable grafana-agent.service
```

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749849406/KFRHaZh_s.png)

## Grafana Cloud

_What is the Grafana Cloud?_

> Grafana Cloud is a composable observability platform, integrating metrics, traces and logs with Grafana. Leverage the best open source observability software – including Prometheus, Loki, and Tempo – without the overhead of installing, maintaining, and scaling your observability stack.

And they offer a free tier, which is absolutely perfect for us to use with our inlets exit nodes. The free tier includes:

- 10,000 series for Prometheus or Graphite metrics
- 50 GB of logs
- 50 GB of traces
- 14 days retention for metrics, logs and traces
- Access for up to 3 team members

So go and create an account. I wait here.

![waiting-simpsons.gif](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749148246/O9Dk2sRsW.gif)

### Grafana Agent

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749819797/SzhslKRXU.png)

Ok thats the last picture of agent Smith, we got the joke.

The Grafana agent collects observability data and sends it to Grafana Cloud. Once the agent is deployed to your hosts, it collects and sends Prometheus-style metrics and log data using a pared-down Prometheus collector.

The Grafana agent is designed for easy installation and updates. It uses a subset of Prometheus code features to interact with hosted metrics, specifically:

- Service discovery
- Scraping
- Write ahead log (WAL)
- Remote writing

Along with this, the agent typically uses less memory than scraping metrics using Prometheus. Yeah!

### API Key

After you created an account, we need to get the values for the three new variables the updated `terraform` script needs:

````terraform
prom-id = xxx
prom-pw = xxx
prom-url = xxx
````

Just follow this instruction  https://grafana.com/docs/grafana-cloud/reference/create-api-key/ to get the `API key` we going to use as `prom-pw`

The `prom-id` and `prom-url`, we find both in the same spot in your account details page int the prometheus details:

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749269213/sP2DiQuDK.png)

Click on `Details`:

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749394808/0RLbW-cxm.png)

Grab the Remote Write Endpoint and Instance ID.

### Agent Config.

Here is the full agent config, where are going to use. I will go more into the details, for the special points.

```yaml
server:
  log_level: info
  http_listen_port: 12345
prometheus:
  wal_directory: /tmp/wal
  global:
    scrape_interval: 60s
  configs:
    - name: agent
      scrape_configs:
        - job_name: 'http-tunnel'
          static_configs:
            - targets: [ 'localhost:8123' ]
          scheme: https
          authorization:
            type: Bearer
            credentials: ${authToken}
          tls_config:
            insecure_skip_verify: true
      remote_write:
        - url: ${promUrl}
          basic_auth:
            username: ${promId}
            password: ${promPW}

integrations:
  agent:
    enabled: true

  prometheus_remote_write:
    - url: ${promUrl}
      basic_auth:
        username: ${promId}
        password: ${promPW}
```

##### Inlets scraping config

```yaml
...
scrape_configs:
  - job_name: 'http-tunnel'
    static_configs:
      - targets: [ 'localhost:8123' ]
    scheme: https
    authorization:
      type: Bearer
      credentials: ${authToken}
    tls_config:
      insecure_skip_verify: true
...
```

This section is directly from the how-to of [Alex](https://inlets.dev/blog/2021/08/18/measure-and-monitor.html) I just substituted the credentials with a template variable.

##### remote-write config

```yaml
...
remote_write:
  - url: ${promUrl}
    basic_auth:
      username: ${promId}
      password: ${promPW}
...
prometheus_remote_write:
  - url: ${promUrl}
    basic_auth:
      username: ${promId}
      password: ${promPW}
```

We're going to use the remote write functionality. That means the Grafana agent, will push any collected metrics, to our central `Prometheus` in the Grafana cloud. Keep this in mind, as this is a major difference to a classic prometheus installation you may have worked with.

I highly recommend you watch following YouTube video [The Future is Bright, the Future is Prometheus Remote Write - Tom Wilkie, Grafana Labs](https://www.youtube.com/watch?v=vMeCyX3Y3HY). 

%[https://www.youtube.com/watch?v=vMeCyX3Y3HY]

That's it, with the new bits. Now we can deploy the whole stack via:

```bash
terraform apply --auto-approve 
...
terraform output inlets-connection-string 
```

When you log into Grafana, you can now start to display your inlets metrics:

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629749509961/XNeNZ1Zqr.png)

Use the how-to of [Alex](https://inlets.dev/blog/2021/08/18/measure-and-monitor.html) for all the available metrics.


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1629750011035/FExzoXnPA.png)


- Check out the github repo -> https://github.com/dirien/inlets-oci-terraform
- Join the inlets [slack](openfaas.slack.com) channel. 
- The official documentation -> https://inlets.dev/
- Drop me DM in Twitter -> https://twitter.com/_ediri
