This is the second post in the Azure Serial Console blog series, which offers information to help defenders be better equipped to look into the activities of Azure SQL Console on virtual Azure Linux machines. While the first blog post covered a variety of tracing techniques, such as using Sysmon logs and Azure activity on Windows virtual machines to track serial console activity, this one explains how to enable logging for Azure Linux virtual computers by capturing these events and sending them to an analytics workspace. Additionally, it discusses Microsoft Defender’s serial console activity for Endpoint logs. The objective goes beyond finding serial console activity on Linux virtual machines to using these artifacts to generate new hunting queries for spotting potential abuse by attackers.
Contents
Enable logging on a virtual Linux machine
Please be aware that this step might not be required depending on how things are set up right now. Feel free to skip this step if Microsoft Sentinel or another security monitoring tool, such as SIEM ( security information and event management ), is already monitoring your cloud computing workloads. In order to capture the Azure serial console activity artifacts, this section of the blog explains how to log Azure Linux virtual machine events using Microsoft Defender for Endpoint ( for Linux ) agents and Sysmon for Linux.
Part 1 of the blog discusses how to create a log analytics workspace and enable Azure activity monitoring. Install the Log Analytics Linux Agent from the Agents tab in the log analytics workspace to enable logging for a virtual Linux machine.
Syslog
A common event logging protocol used in Linux is called syslog. Local events are gathered and sent to a Log Analytics workspace by the Syslog daemon on Linux devices as specified. Applications send messages to a Syslog collector or for local storage. When the local Syslog daemon is set up to forward messages to the Azure Monitor agent when it is enabled by data collection rules (DCRs ), this is done when the agent is installed for Linux. These messages are then sent to an Azure Monitor or Log Analytics workspace, where they are stored in a Syslog table as corresponding records. Using the guide here, gather Syslog events with Azure Monitor Agent.
Linux version of Sysmon
a prerequisite for installing Sysmon for Linux
- Make sure the Linux virtual machine is connected to the workspace for log analytics.
- This guide will help you use Azure Monitor Agent to collect Syslog events.
Use this guide to install Sysmon on a virtual Linux machine. You can find the Sysmon for Linux configuration here, and you can install the service with the configuration below.
<Sysmonschemaversion="4.70"><EventFiltering><!--EventID1==ProcessCreate.Logallnewlycreatedprocesses--><RuleGroupname=""groupRelation="or"><ProcessCreateonmatch="exclude"/></RuleGroup><!--EventID3==NetworkConnectDetected.Logallnetworkconnections--><RuleGroupname=""groupRelation="or"><NetworkConnectonmatch="exclude"/></RuleGroup><!--EventID5==ProcessTerminate.Logallprocessesterminated--><RuleGroupname=""groupRelation="or"><ProcessTerminateonmatch="exclude"/></RuleGroup><!--EventID9==RawAccessRead.Logallrawaccessread--><RuleGroupname=""groupRelation="or"><RawAccessReadonmatch="exclude"/></RuleGroup><!--EventID10==ProcessAccess.Logallopenprocessoperations--><RuleGroupname=""groupRelation="or"><ProcessAccessonmatch="exclude"/></RuleGroup><!--EventID11==FileCreate.Logeveryfilecreation--><RuleGroupname=""groupRelation="or"><FileCreateonmatch="exclude"/></RuleGroup><!--EventID23==FileDelete.Logallfilesbeingdeleted--><RuleGroupname=""groupRelation="or"><FileDeleteonmatch="exclude"/></RuleGroup></EventFiltering></Sysmon>
Artifacts from the serial console on a virtual Linux machine
The evidence of accessing a virtual machine using serial console can be found in native Syslog andLinux version of Sysmonlogs on virtual machine. The Serial console connection happens over /dev/ttyS0 on Linux virtual machines.
Activity tracing using Syslog
Identify launching of serial console to access a Linux virtual machine, it will initiate connection on ttyS0 and show an authentication prompt with username and password to login into the virtual machine. The following query can be used to identify the same.
Syslog|whereProcessID==1|whereFacility=="daemon"|whereSyslogMessagehas"Found device /dev/ttyS0"
To identify the start of a virtual ttyS0 terminal upon the /dev/ttyS0 is enabled on Linux virtual machine, the following query can be used.
Syslog|whereProcessID=="1"|whereFacility=~"daemon"|whereSeverityLevel=~"info"|whereSyslogMessagehas"Started Serial Getty on ttyS0"
Use the Serial console in Syslog native logs to find successful virtual machine access.
Syslog|whereProcessID==1|whereProcessName=~"systemd"|whereSyslogMessagehas"[email protected]: Succeeded"
Find failed login attempts using the serial console to connect to a virtual machine. Additionally, the number of unsuccessful attempts will be recorded in syslog logs for the relevant user account over time.
Syslog|whereProcessName=~"login"|whereSyslogMessagehas"FAILED LOGIN"andSyslogMessagehas"/dev/ttyS0"
The use of ‘sudo’ after a successful authentication using serial console will be captured in syslog authpriv logs with the command being executed with ‘sudo’ and the commands with ‘sudo’ will also be captured.
Syslog|whereProcessName=~"sudo"|whereSyslogMessagehas"TTY=ttyS0"andSyslogMessagehas"USER=root"
Tracing activity usingLinux version of Sysmon
OnceLinux version of Sysmonis successfully installed, the log analytics workspace begins to capture Sysmon events within syslog messages that contain “Linux-Sysmon.” The query below can be used to identify all process execution events on a Linux virtual machine accessed via the serial console.Linux version of Sysmonlogs all ProcessCreate events on the Linux virtual machine under Event ID 1.
This Kusto query can parse allLinux version of Sysmonlogs for all events mentioned in the current configuration file. This query can be saved as a function (for example, Sysmon_for_linux ) to search for different events on a Linux virtual machine using event ID field.Every process on a Linux virtual machine using the serial console has a unique parent process & command line. The parent process IDs will change depending on the serial console sessions. The parent process will be “/usr/bin/bash” and parent process commandline will be “-bash.” The Kusto query below can parseLinux version of Sysmonlogs for Event ID 1 (ProcessCreate) to identify the processes executed using serial console Special Administration Console (SAC). However, there is a caveat to this query as it only projects immediate child processes executed from the SAC console, not the processes spawned from a shell script.
Sysmon_for_Linux|whereEventID==1|whereParentCommandLine=="-bash"orParentCommandLine=="bash"|extendParentProcessId=tostring(ParentProcessId)|joinkind=inner(Sysmon_for_Linux|whereEventID==1|extendProcessId=tostring(ProcessId)|whereParentImagehas"/bin/login")on$left.ParentProcessId==$right.ProcessId|projectUtcTime,Computer,User,CurrentDirectory,Image,CommandLine,ParentImage,ParentCommandLine,InitiatingProcessParentImage=ParentImage1,InitiatingProcessParentCommandLine=CommandLine1,LogonId,ProcessId,ParentProcessId,InitiatingProcessParentId=ParentProcessId1
For instance, the following query can use a serial console to identify all public network connections from the processes carried out ( within the context of the logged in user ) on an imaginary machine.
Sysmon_for_Linux|whereEventID==3// Network Connect|wherenot(ipv4_is_private(tostring(DestinationIp)))|wherenot(Userin~("omsagent","syslog","systemd-resolve"))|wherenot(Imagehas_any("/opt/microsoft/azuremonitoragent/","/opt/microsoft/omsagent/"))|whereDestinationIp!in("127.0.0.1")|projectUtcTime,EventID,Image,tostring(ProcessId),tostring(User),Computer,Protocol,SourceIp,DestinationIp,SourcePort,DestinationPort,Operation|joinkind=inner(Sysmon_for_Linux|whereEventID==1//Process Create|whereParentCommandLine=="-bash"orParentCommandLine=="bash"|extendParentProcessId=tostring(ParentProcessId)|joinkind=inner(Sysmon_for_Linux|whereEventID==1|extendProcessId=tostring(ProcessId)|whereParentImagehas"/bin/login")on$left.ParentProcessId==$right.ProcessId|projectUtcTime,Computer,tostring(User),CurrentDirectory,Image,CommandLine,ParentImage,ParentCommandLine,InitiatingProcessParentImage=ParentImage1,InitiatingProcessParentCommandLine=CommandLine1,LogonId,tostring(ProcessId),ParentProcessId,InitiatingProcessParentId=ParentProcessId1)onProcessId,User,Computer|projectUtcTime,Computer,User,Image,CommandLine,SourcePort,DestinationIp,DestinationPort
Microsoft Defender for Endpoint ( MDE ) activity tracking
Microsoft Defender for Endpoint captures Azure serial console activity for a Linux virtual machine. This activity includes logging into virtual machines via the SAC console, executing all processes, and establishing network connections using the logged-in serial console. Serial console connections to the Linux virtual machine are established on virtual console terminals with device /dev/vc/. In general scenarios, direct serial console access to any device happens over /dev/ttyS0. The Kusto queries below for MDE will provide information related to logged-in users, executed processes, and established network connections using serial console.
The following Kusto query for MDE will use a serial console to find users who have logged in.
DeviceLogonEvents|extendTerminal=tostring(parse_json(AdditionalFields).["Terminal"])|whereTerminalhas"ttyS0"|projectTimestamp,DeviceName,ActionType,LogonType,AccountDomain,AccountName,InitiatingProcessCommandLine,Terminal
All processes carried out during the session connected via the serial console are identified by the Kusto query below.
DeviceProcessEvents|extendInitiatingProcessPosixAttachedTerminal=tostring(parse_json(AdditionalFields).InitiatingProcessPosixAttachedTerminal)|whereInitiatingProcessPosixAttachedTerminal!=""|whereInitiatingProcessPosixAttachedTerminalhas"/dev/vc/"//virtual console devices for Serial Console |projectTimestamp,DeviceName,FileName,ProcessCommandLine,InitiatingProcessFileName,InitiatingProcessCommandLine,AccountName,InitiatingProcessPosixAttachedTerminal|sortbyTimestampdesc
The Kusto query below will find every network connection based on the serial console processes that were run.
DeviceNetworkEvents|extendInitiatingProcessPosixAttachedTerminal=tostring(parse_json(AdditionalFields).InitiatingProcessPosixAttachedTerminal)|whereInitiatingProcessPosixAttachedTerminalhas"/dev/vc/"//virtual console devices for Serial Console |projectTimestamp,DeviceName,ActionType,RemoteIP,RemoteIPType,InitiatingProcessCommandLine,InitiatingProcessAccountName,InitiatingProcessPosixAttachedTerminal
searching for shady activities
The examples below show how a defender can spot suspicious activity near an Azure serial console.
Hunting suspicious operations usingLinux version of Sysmonevent logs
Through the Azure serial console, an attacker can gain initial access, and any activity that follows this access can be tracked using the process lineage generated by the console. The following Kusto query can be especially helpful for hunting the most popular techniques, even though there are many KQL ( Kusto Query Language ) hunting queries available to detect suspicious operations on a Linux virtual machine. Be aware that not all Linux attack techniques are covered by the following query. It does, however, outline the process hierarchy that security analysts can use to track down activity carried out on a linux machine through the serial console.
Sysmon_for_Linux|whereEventID==1|whereParentCommandLine=="-bash"orParentCommandLine=="bash"|extendParentProcessId=tostring(ParentProcessId)|joinkind=inner(Sysmon_for_Linux|whereEventID==1|extendProcessId=tostring(ProcessId)|whereParentImagehas"/bin/login")on$left.ParentProcessId==$right.ProcessId|whereImagehas_any("ssh","netcat","telnet","rsh","rcp","rsync","ftp","smbclient","smbmount","smbget","curl","wget","/nc","ncat","socat","chisel","pivoting","proxy","tunneling","forwarding","sshuttle","htran","iodine","dnscat2","dnscat3","ptunnel","udptunnel","httptunnel","proxytunnel","stunnel","sslh","sslwrap","proxychains")orImagehas_any("whoami","history","uname","/ps","passwd","groups","smbclient","systemctl","ifconfig","/id","ping","traceroute","whois","nslookup","dig","host","useradd")orImagehas_any("/var/tmp/")orCommandLinehas_any("dpkg -i","dpkg-deb",".deb")or(Imagehas"atd"andCommandLinehas"atd")or(Imagehas"cron"andCommandLinehas"cron")or(Imagehas"find"andCommandLinehas"-perm")or(CommandLinehas_any("chmod","chown","chattr"))orCommandLinehas_any("stop","disable","off","pkill","kill","killall")or(Imagehas"find"and(CommandLinehas_any("password","login","token","key")))or(Imagehas"find"and(CommandLinehas_any("pem","pfx","pgp","gpg","ppk","p12","cer","p7b","asc")))|projectUtcTime,Computer,User,CurrentDirectory,Image,CommandLine,ParentImage,ParentCommandLine,InitiatingProcessParentImage=ParentImage1,InitiatingProcessParentCommandLine=CommandLine1,LogonId,ProcessId,ParentProcessId,InitiatingProcessParentId=ParentProcessId1using Microsoft Defender for Endpoint to search suspicious operations
It is possible to use access to the serial console to perform reconnaissance, download files, look for secrets, and perform lateral movement by using well-known native system images. The execution of the most popular native images using a serial console is found in the query below.
letPotentialAbusedImages=pack_array("ssh","netcat","telnet","rsh","rcp","rsync","ftp","smbclient","smbmount","smbget","curl","wget","nc","ncat","socat","chisel","pivoting","proxy","tunneling","forwarding","sshuttle","htran","iodine","dnscat2","dnscat3","ptunnel","udptunnel","httptunnel","proxytunnel","stunnel","sslh","sslwrap","proxychains","whoami","history","uname","ps","passwd","groups","smbclient","systemctl","ifconfig","/id","ping","traceroute","whois","nslookup","dig","host","useradd","chmod","chown","chattr","atd","cron","pkill","kill","killall");DeviceProcessEvents|extendInitiatingProcessPosixAttachedTerminal=tostring(parse_json(AdditionalFields).InitiatingProcessPosixAttachedTerminal)|whereInitiatingProcessPosixAttachedTerminal!=""|whereInitiatingProcessPosixAttachedTerminalhas"/dev/vc/"//virtual console devices for Serial Console |whereInitiatingProcessCommandLinehas"bash"|whereFileNamehas_any(PotentialAbusedImages)|projectTimestamp,DeviceName,FileName,ProcessCommandLine,InitiatingProcessFileName,InitiatingProcessCommandLine,AccountName,InitiatingProcessPosixAttachedTerminalusing Azure Activity logs to look for suspicious operations
Anomalyptic serial console connect activity can be detected using the following query in addition to the hunting methods mentioned in Part 1.
AzureActivity|whereOperationNameValue=="MICROSOFT.SERIALCONSOLE/SERIALPORTS/CONNECT/ACTION"|make-seriesperCount=count()onTimeGeneratedfromago(15d)tonow()step1dbyCallerIpAddress,Caller|rendertimechart
using Boot diagnostics logs to look for suspicious operations
We covered how a foe can misuse boot diagnostic logs to leak sensitive information in Part 1. Boot diagnostics logs, however, continue to be a useful source of activity identification. Boot diagnostics logs can offer important insights when log files like Syslog are not backed up to a central location like SIEM. Due to the potentially sensitive nature of the data they may contain, these logs are not backed up to a SIEM. However, in situations where the logs are missing, they have important forensic value.
The following is an example of using Netcat to execute LinPEAS, a well-known privilege escalation tool, and reverse shell connectivity. Please be aware that Microsoft does not support the use of these tools, and it is the reader’s duty to do so in a moral and responsible manner. Before using these tools against Azure or other Microsoft infrastructure, kindly review our terms of service.
Best Practices
Here is a partial list of the best practices we advise for maintaining Azure Serial Console security:
- For all users with” Virtual machine contributor” access, enforce the use of MFA ( multifactor authentication ).
- To make sure the list of privileged users is current, regularly audit the RBAC ( role-based access control ) permissions of users.
- Use cloud native SIEM ( such as Microsoft Sentinel ) to implement continuous monitoring of the Serial Console activities for Azure virtual machines. Set up suspicious activity alerts.
- Create a clear and well-documented procedure for Serial Console emergency access. Make sure that authorized personnel are aware of these procedures and that they are strictly followed in emergency situations.
Conclusion
Although Azure Serial Console is a useful tool for administrators and developers to troubleshoot during difficult times, if it is n’t properly monitored and secured, it could end up being liable for security breaches. This blog highlighted a number of opportunities for spotting malicious activity involving the Azure serial console. The article’s discussion of logging opportunities, detection strategies, and hunting questions made it simple to protect Azure Serial console.