1. Check Ssh Server Status
2. Check Ssh Status

1. No, really, how to check if ssh-agent is already running in bash? Answers so far don't appear to answer the original question.
2. Best practice is to use the Icinga agent as secure execution bridge (checknt and checknrpe are considered insecure) and query the NSClient service locally. You can use the checknt plugin from the Monitoring Plugins project to query NSClient. Icinga 2 provides the nscp check command for this.
3. I'm looking to lock down our Solaris 10, RHEL 5, and SLES 11.2 servers. It has come to my attention that some of the users have no passphrase for their SSH keys. Is there a way I can check for this.

For each SSH session, enable agent and add the key by running.

Many webservices generate ssh keys to access their service. With the amount of services the number of SSH keys grows. To avoid dealing with the keys in the command line and the ssh_config(5), you can simply add the ssh-key to the local ssh-agent(1) to manage them.

The ssh-agent(1) is a program used to hold ssh private keys used to authenticate to remote systems via ssh public key authentication. ssh clients use it to authenticate to the remote via the keys in the ssh-agent.

When the ssh-agent is started, the output does not show output meant to be read by humans

The output are shell commands (-s for bash shell) that should be executed. To do this in one command, the eval command can be used.

When the ssh-agent is started, it prints out the environment variables that need to be set. These environment variables are required so the client programs, like ssh, know how to connect to the ssh-agent. To directly set these environment variables, the output of the ssh-agent is passed as parameter to the eval command. The “eval” command will execute the parameters passed to it as if they were directly entered into the shell. In this case, the output of the ssh-agent is directly executed.

With the ssh-agent now running, there are not yet any keys added to the agent. Adding an ssh-key to the ssh-agent is done via the ssh-add(1) command. As a parameter, ssh-add takes the filename of the ssh private-key. If the default location and file name is used for the key (~/.ssh/id_rsa), the filename parameter can be omitted.

When the private key is secured by a password (which is suggested), ssh-add will ask for it to add the key to the agent.

With the ssh-agent running and the key added to the agent, any ssh session can now be authenticated with that key. Additional ssh-keys, if needed, can be added using ssh-add as described above. Instead of entering the ssh-key password each time, the agent manages the keys and only asks once for the password of the keys.

To list the ssh-keys currently available in the ssh-agent, the -l option can be used. The fingerprints of the keys in the agent are shown, like in the example below.

With the -L option, the keys in the ssh-agent can be listed as well. While the -l option shows the fingerprint for each key, the -L option shows the public key part of each key. This can be especially helpful if the ssh-key should be added to a remote server for authentication. The output lines can directly be used to add the key to the authorized_keys file.

(The above output has been shortened as the public key sections are quite long and really relevant to this example).

To remove a key from the ssh-agent, the -d option can be used as the example below shows. The parameter given is the file name as it is shown in the list of keys. This identifies the key to be removed.

Removing all private keys from the ssh-agent can be achieved with the -D option as shown below. In this case it is not necessary to add identifiers for the individual keys.

Keeping the private keys secure is important. It is suggested that the ssh-agent is terminated or locked when it is not needed anymore. To terminate the ssh-agent the following command can be executed.

This command – like starting the agent – uses the eval command. In this case not to set environment variables but to unset them as the agent has been stopped.

Automate the ssh-agent start

With all the above in mind, it seems to be a lot of commands to remember and enter every time. There has to be a way to automate at least some of this, and there is.

The ssh-agent should be started when a Shell session is started and the default key should be added to the ssh-agent, but with this approach, every shell opened and every session started gets its own ssh-agent and the password is asked for adding the key to the ssh-agent. To share the ssh-agent between the sessions, the following can be added to the “~/.bashrc”.

The command in the if statement is constructed in a way to list all ssh-agents running in the context of the user shells/session’s user. If the user has no ssh-agent running yet, the ssh-agent is started. For the purpose of reusing the ssh-agent for any parallel sessions, the output of the ssh-agent is not directly passed to the “eval” command but stored into a file. This file is then loaded in the second step. As final step the keys are loaded via ssh-add. The ssh-add command will cause the session to immediately ask for the ssh-key passphrases.

If the user already has an ssh-agent running (from a parallel session/shell) the else part is executed which simply loads the environment variables needed. Those where stored in a file when the agent was started.

This way, the agent is started with the first login and used for every further session started.

Automate the ssh-agent termination

To secure the ssh-keys after all sessions for a user are ended, the ssh-agent should be stopped when the user exits the sessions.

bash(1) supports a script to be executed on logout from a session. The file that is executed on logout is called ~/.bash_logout.

The same way as the section in the .bashrc, the script in the .bash_logout needs to be aware of the logged in sessions as the ssh-agent is shared between the sessions.

The script above placed in the .bash_logout file will be executed when the user will logout from a session. The command in the if statement will list all logged in sessions (who(1)) and reduces them to only sessions of the current user (grep). The “wc -l” as last part of this command will count the resulting lines which makes it easier to compare. The comparison checks if there are less or equal to one session (at this point, the current session is still active so “1” session active will be considered the last session. When this condition is met, all ssh-keys are removed from (ssh-add -D) the and the agent is stopped (ssh-agent -s -k). To cleanup properly, the file where the ssh-agent output was stored is deleted as well.

Terminate ssh-agent when no logout was performed

The ssh session might be disrupted by a timeout or when the network connection drops. In such cases the shell does not receive the logout command and the ssh-agent is not terminated by the script in the .bash_logout script. To ensure that the ssh-agent is ended and the ssh-keys are secure, the following can be done.

Create a script which can be executed as a cron job. This script looks similar to the script in the .bash_logout. It differs only by the amount of sessions that should be active before it terminates the ssh-agent. The cron script starts its action when no session is active any more for that user.

The script (in this example “~/.cron_ssh-agent”) can now be added to the cron jobs. To do so, open the crontab in edit mode (-e).

The cron job can now be added. Depending on the configuration of your system, the crontab is opened in vim or whatever editor is configured for it. Add the following line to execute the script every 5 minutes.

With the .bash_logout script in place and the cron job configured, the ssh-agent will be immediately stopped when you logout. In case of a connection drop, the ssh-agent will be automatically stopped at the latest 5 minutes after the connection was lost.

Read more of my posts on my blog at https://blog.tinned-software.net/.

If the remote services are not directly accessible through the network, alocal agent installation exposing the results to check queries canbecome handy.

Prior to installing and configuration an agent service, evaluate possibleoptions based on these requirements:

• Security (authentication, TLS certificates, secure connection handling, etc.)
• Connection direction
  • Master/satellite can execute commands directly or
  • Agent sends back passive/external check results
• Availability on specific OS types and versions
  • Packages available
• Configuration and initial setup
• Updates and maintenance, compatibility

Available agent types:

• Icinga Agent on Linux/Unix and Windows
• SSH on Linux/Unix
• SNMP on Linux/Unix and hardware
• SNMP Traps as passive check results
• REST API for passive external check results
• NSClient++ and WMI on Windows

Icinga Agent ¶

For the most common setups on Linux/Unix and Windows, we recommendto setup the Icinga agent in a distributed environment.

Key benefits:

• Directly integrated into the distributed monitoring stack of Icinga
• Works on Linux/Unix and Windows
• Secure communication with TLS
• Connection can be established from both sides. Once connected, command execution and check results are exchanged.
  • Master/satellite connects to agent
  • Agent connects to parent satellite/master
• Same configuration language and binaries
• Troubleshooting docs and community best practices

Follow the setup and configuration instructions here.

On Windows hosts, the Icinga agent can query a local NSClient++ servicefor additional checks in case there are no plugins available. The NSCPinstaller is bundled with Icinga and can be installed with the setup wizard.

SSH ¶

Tip

This is the recommended way for systems where the Icinga agent is not availableBe it specific hardware architectures, old systems or forbidden to install an additional software.

This method uses the SSH service on the remote host to executean arbitrary plugin command line. The output and exit code isreturned and used by the core.

The check_by_ssh plugin takes care of this. It is available in theMonitoring Plugins package.For your convenience, the Icinga template library provides the by_sshCheckCommand already.

SSH: Preparations ¶

SSH key pair for the Icinga daemon user. In case the user has no shell, temporarily enable this.When asked for a passphrase, do not set it and press enter.

On the remote agent, create the icinga user and generate a temporary password.

Copy the public key from the Icinga server to the remote agent, e.g. with ssh-copy-idor manually into /home/icinga/.ssh/authorized_keys.This will ask for the password once.

After the SSH key is copied, test at the connection at least once andaccept the host key verification. If you forget about this step, checks willbecome UNKNOWN later.

After the SSH key login works, disable the previously enabled logins.

• Remote agent user’s password with passwd -l icinga
• Local icinga user terminal

Also, ensure that the permissions are correct for the .ssh directoryas otherwise logins will fail.

Check Ssh Server Status

• .ssh directory: 700
• .ssh/id_rsa.pub public key file: 644
• .ssh/id_rsa private key file: 600

SSH: Configuration ¶

First, create a host object which has SSH configured and enabled.Mark this e.g. with the custom variable agent_type to lateruse this for service apply rule matches. Best practice is tostore that in a specific template, either in the static configurationor inside the Director.

Example for monitoring the remote users:

A more advanced example with better arguments is shown in this blogpost.

SNMP ¶

The SNMP daemon runs on the remote system and answers SNMP queries by plugin scripts.The Monitoring Plugins package providesthe check_snmp plugin binary, but there are plenty of existing pluginsfor specific use cases already around, for example monitoring Cisco routers.

The following example uses the SNMP ITLCheckCommand and sets the snmp_oid custom variable. A service is created for all hosts whichhave the snmp-community custom variable.

If no snmp_miblist is specified, the plugin will default to ALL. As the number of available MIB fileson the system increases so will the load generated by this plugin if no MIB is specified.As such, it is recommended to always specify at least one MIB.

Additional SNMP plugins are available using the Manubulon SNMP Plugins.

For network monitoring, community members advise to use nwc_healthfor example.

SNMP Traps and Passive Check Results ¶

SNMP Traps can be received and filtered by using SNMPTTand specific trap handlers passing the check results to Icinga 2.

Following the SNMPTT Formatdocumentation and the Icinga external command syntax found herewe can create generic services that can accommodate any number of hosts for a given scenario.

Simple SNMP Traps ¶

A simple example might be monitoring host reboots indicated by an SNMP agent reset.Building the event to auto reset after dispatching a notification is important.Setup the manual check parameters to reset the event from an initial unhandledstate or from a missed reset event.

Add a directive in snmptt.conf

1. Define the EVENT as per your need.
2. Construct the EXEC statement with the service name matching your templateapplied to your n hosts. The host address inferred by SNMPTT will be thecorrelating factor. You can have snmptt provide host names or ip addresses tomatch your Icinga convention.

Note

Replace the deprecated command pipe EXEC statement with a curl callto the REST API action process-check-result.

Add an EventCommand configuration object for the passive service auto reset event.

Create the coldstart_reset_event.sh shell script to pass the expanded variabledata in. The $service.state_id$ is important in order to prevent an endless loopof event firing after the service has been reset.

Note

Replace the deprecated command pipe EXEC statement with a curl callto the REST API action process-check-result.

Finally create the Service and assign it:

Complex SNMP Traps ¶

A more complex example might be passing dynamic data from a traps varbind listfor a backup scenario where the backup software dispatches status updates. Byutilizing active and passive checks, the older freshness concept can be leveraged.

By defining the active check as a hard failed state, a missed backup can be reported.As long as the most recent passive update has occurred, the active check is bypassed.

Add a directive in snmptt.conf

1. Define the EVENT as per your need using your actual oid.
2. The service name, state and text are extracted from the first three varbinds.This has the advantage of accommodating an unlimited set of use cases.

Note

Check Ssh Status

Replace the deprecated command pipe EXEC statement with a curl callto the REST API action process-check-result.

Create a Service for the specific use case associated to the host. If the hostmatches and the first varbind value is Backup, SNMPTT will submit the correspondingpassive update with the state and text from the second and third varbind:

Agents sending Check Results via REST API ¶

Whenever the remote agent cannot run the Icinga agent, or a backup scriptshould just send its current state after finishing, you can use the REST APIas secure transport and send passive external check results.

Use the process-check-result API action to send the external passive check result.You can either use curl or implement the HTTP requests in your preferred programminglanguage. Examples for API clients are available in this chapter.

Feeding check results from remote hosts requires the host/serviceobjects configured on the master/satellite instance.

NSClient++ on Windows ¶

NSClient++ works on both Windows and Linux platforms and is wellknown for its magnificent Windows support. There are alternatives like the WMI interface,but using NSClient++ will allow you to run local scripts similar to check plugins fetchingthe required output and performance counters.

Tip

Best practice is to use the Icinga agent as secure executionbridge (check_nt and check_nrpe are considered insecure)and query the NSClient++ service locally.

You can use the check_nt plugin from the Monitoring Plugins project to query NSClient++.Icinga 2 provides the nscp check command for this:

Example:

For details on the NSClient++ configuration please refer to the official documentation.

WMI on Windows ¶

The most popular plugin is check_wmi_plus.

Check WMI Plus uses the Windows Management Interface (WMI) to check for common services (cpu, disk, sevices, eventlog…) on Windows machines. It requires the open source wmi client for Linux.

Community examples: