Security Tips: Your Server Got Hacked?

Your OpenVMS or Mac OS X or Mac OS X Server or other server box got hacked. Now what?

Preserving the Evidence

If there is the possibility of any legal action arising from the security breach, check with the appropriate legal representatives and authorities; a pristine copy of the disks with the details of the hack(s) left in place may well be needed. This copy can also be useful for offline examination of the attack; if you've figured out the immediate hole, but not the details of the attack.

Offline Investigation and Reinstallation

After a server gets hacked, the first step is to go offline. To keep the box on a private LAN.

And you then need to figure out how you got hacked.

Decontamination involves determining the source of the breach, and looking for what back-doors might have been left for later. Simply reinstalling an operating system does not address a backdoor left elsewhere on the server. Nor does reinstallation ensure the exposure was addressed. Sanitizing a complete system is a big project.

The re-installation of the software from known good distributions is a necessary step. But it is far from sufficient.

Unlike the “just reboot it” approach to troubleshooting bugs, a successful security attack is very likely one that will be repeated. Attackers do quite commonly try to replay the same and previously successful attack.

As part of the electronic archeology, review the system logs. Additional steps include the review of the web server logs and the logs of anything that has its ports exposed through the firewall. On Mac OS X and Mac OS X Server, review the system console logs. On OpenVMS, review the system accounting and security auditing logs and OPCOM. Look for odd users or odd login times, or patterns of network probes.

Review the directories for modified files by date and (if needed) by contents. Some attacks will be visible by file modification dates, though other attacks will specifically reset these to the original or to a more innocuous value.

Don't assume that because there are no IP ports open when you checked, that there is no back-door lurking.

Web Services

Determine what php or Perl or Python or Ruby code is running. Ensure this code is current, and that the code is known clear of vulnerabilities and XSS and SQL injection attacks. If you don't know what XSS or SQL attacks look like, you'll want to learn about the attacks and review your web server logs; these attacks are quite common.

Confirm that the web server does not have write access to its web directories, and ensure that the web server does not own its web directories. This would be the ls -al and chown commands in a typical bash shell, or the DIRECTORY /SECURITY and SET SECURITY commands on OpenVMS.

You might pick -rw-r--r-- with root:www ownership for the files and directories, for instance. This would be akin to chmod u=rw,go=r or mayhap chmod -R u=rwx,go=r and chown -R root:www, or some such. It's also easily feasible to add www to an ACL, and allow it only specific read access. OpenVMS has analogous protection masks and ownership.

A web server that can write into the web-facing directories is a central configuration issue here; this can provide part of a foothold or can allow an existing breach to be expanded by an attacker. This is a Really Bad Idea. The web server can and should have read access via protection mask or via ACL. The web server should not have ownership nor write access save to specified and potentially protected subdirectories, and then only as required.

The usual trigger with content management system (CMS) vulnerabilities is down-revision software; a CMS package that is insecure. This either due to long-standing bugs that have been found in the target CMS, or due to a failure to maintain a current version of the software. Most any CMS system requires regular updates and review. HoffmanLabs prefers additional and out-board defenses for CMS systems, as well. Techniques that seek to block common attack types before the attack reaches the CMS environment.

All web-facing CMS code needs to validate its input. Attackers do try to jam unexpected data into the input fields, seeking to cause the CMS package or whatever code is accepting and processing the input to perform untoward acts. If you review your web server and CMS server logs, you'll almost certainly find evidence of cross-site scripting (XSS) attacks, too.

Mail Services

Determining the source of problems with a mail server — often spam being injected into the mail server — means finding the source of the spam. Where it's being injected into the mail server. This can be a so-called open relay configuration, or an authorized mail client that's been breached, or a weak password that's been identified. In various cases, web services including content management systems and bug-tracking systems can have weaknesses, and these can be used to send mail.

You will want and need to look for the way the spam is arriving via the SMTP server log files and potentially also the POP or IMAP log files, and the web server logs, and other tools and packages that might be authorized to send email through your mail server. This is the most direct means of identifying the path through your server, and thus the source, and closing the hole.

Reinstalling a mail server can be suggested, but if the breach is a weakness in web-facing software, or in a mail client, or if there's a backdoor left, then the effort involved in reinstalling the system will be wasted.

Within Postfix, the following are some key settings, and these can (if mis-set) allow an open relay:

smtpd_recipient_restrictions = permit_sasl_authenticated permit_mynetworks reject_non_fqdn_recipient 
reject_unknown_recipient_domain reject_unauth_pipelining reject_unauth_destination check_policy_service unix:private/policy permit
 
smtpd_sender_restrictions = permit_sasl_authenticated reject_non_fqdn_sender reject_unknown_sender_domain permit
 
smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated reject_rbl_client zen.spamhaus.org permit

The above settings do enable greylisting.

Related documentation for folks learning and using and troubleshooting Postfix settings, including OS X Server systems and many Unix and Linux servers: Address Verification, relay access control, backscatter spam, and the full list of Postfix parameters. Here are the SpamCop recommendations for smtpd_recipient_restrictions settings.

Network Attacks

Network-level and network infrastructure attacks can breach a server indirectly. More than a few routers and firewalls have been breached.

Check DNS settings. There are malware attacks that target DNS and that IP routing via BGP. Check the DNS translations of your core services.

If you use mDNS, check that.

Check your designated IP router setting, and check your arp cache.

Check your IP firewalls. You do have firewalls?

Check your web proxy server, and the proxy settings in your web browser.

Confirm secure protocols are in use. A VPN connection or ssh or such is (usually) secure. Avoid insecure protocols and servers, as authentication credentials and passwords can be compromised when telnet or ftp are used.

And do determine if you have any nodes with unusual network activity. This could be a printer or a storage controller; yes, there are attacks against these.

If you're running Postfix, the output of postconf -n can point to problems in the configuration. Other SMTP mail servers have similar configuration and diagnostic tools. These mail server vulnerabilities can include relay spam (which can get your host mail traffic black-listed) up to (if your server is down-revision or your passwords are weak) a full server breach.

Restoration and Recovery

Ensure that all internet-facing software on the server is current.

Embedded or downloaded malware is common on various platforms, and possibility that should be considered on all platforms. Ensure that the users on the system have not downloaded trojans or other malware; there is reportedly a Trojan lurking within one of the recent Torrent downloads for Apple's iWork software, for instance. This is centrally why restoring to known-good copies of the installed software is a valuable approach.

How good are your data archives?

Before you get hacked (or before you get hacked again), ensure you have an archival strategy.

An archival strategy is key to the recovery processing; it's a central step toward system recovery.

This could use Time Machine and Time Capsule, out-board storage, tapes, the OpenVMS BACKUP utility, or whatever is deemed locally appropriate.

Network Defenses

HoffmanLabs recommends using an external network firewall appliance.

Avoid using a host for a firewall as with having services active on the firewall itself can increase the potential for exposures and breaches.

As a side benefit, a network firewall can also simplify IP network routing on smaller LANs. Getting a host with two or more NICs working can be interesting given how IP routing operates. It is feasible to set up default routes for each NIC present, though the default operations of a dual-NIC firewall host are seldom what a user would want.

The firewall can also provide an end-point for in-bound VPN connections.

A firewall is not a panacea (and modern firewalls need to watch traffic headed inbound and outbound), but it can help reduce exposure.

Auditing and Accounting

These mechanisms need to be enabled and in use prior to the breach. And monitored.

Breaches are more difficult to mask in the presence of a reasonable degree of security auditing.

If the details don't get logged, then the breach might not be detected, or the specific path used might not be obvious.

So you think your platform is immune?

Think again.

No site and no software platform should be presumed immune to attack. Including OpenVMS. Including Mac OS X.

These attacks and these techniques are evolving quickly, and some of the security attacks can be fiendishly clever.

Prepare.

Know what your network looks like; having effective change control and profiles can be key to fast(er) recovery.

Audits. Monitoring. Updates. Firewalls. Passive network monitors.

Need help with security defenses or with recovery from an attack? HoffmanLabs can assist.

Comments

OS X and iOS Security Checklist

US National Security Agency (NSA) Security Guidelines for various configurations, including for OS X and iOS devices, Red Hat Enterprise Linux (RHEL) and other platforms.

Apple has (older) security guides available, as well.

OS X malware persistence

From Shakacon, a presentation by Patrick Wardle of SynAck on how malware can become persistent on OS X, for those interested in the topic:

https://s3.amazonaws.com/s3.synack.com/Synack_Shakacon_OSX_Malware_Persistence.pdf

This presentation also includes a quick overview of how OS X Gatekeeper, sandboxing and app code signing works, among other details.

Detecting added LaunchAgent and LaunchDaemon plist files

Courtesy of Cnet, Monitor OS X LaunchAgents folders to help prevent malware attacks

This does work but of course throws up alerts when installing trusted applications. The alert dialogs aren't elegant either...

No, I haven't tried their further suggestion of locking the Launch folders, so cannot say what problems this may cause.

The '90s called and want their static signature AV product back

See slide 8 "XProtect" in the OSX Malware Persistence presentation.

From the 2007* paper Omniunpack: Fast, generic, and safe unpacking of malware (PDF):

"Malicious software (or malware) has become a growing threat as malware writers have learned that signature-based detectors can be easily evaded by “packing” the malicious payload in layers of compression or encryption."

From 2009*, Automatic Generation of String Signatures for Malware Detection (PDF):

"Scanning files for signatures is a proven technology, but exponential growth in unique malware programs has caused an explosion in signature database sizes. One solution to this problem is to use string signatures, each of which is a contiguous byte sequence that potentially can match many variants of a malware family."

* for reasons unknown (but assumed to be something to do with the academic publishing process) these papers do not contain publication dates. The dates given above were derived from PDF metadata and source URLs.

More...

Also from an earlier era here is ever-evolving malware, and there's a Symantec VP that thinks AV is dead, too.

Though I can't locate it right now, there's a recent article detailing the speed at which the malware changes — if you ever want to name something that'll be near-impossible to find with a web search, name it virus, infected or by and large or some variation and searches become vastly more difficult — and the speed of malware evolution was quite striking. Most of the malware is seen once or twice in the scanners and for a few hours at most, and then changes. That means you're dragging around ever-larger lists of signatures, as you'll want to be protected against old malware that reappears, but new malware arrives and then changes before many folks have even loaded the signature updates from their providers.

For completeness, OpenVMS has seen malware, and I would not assume your OpenVMS server or your SCADA system or your application is safe against attack. You're just a different sort of target than Windows and OS X and other systems, so the attacks and the risks will be very different than what the Windows systems are usually incurring. If there's money around your OpenVMS server, you're a target for targeted attacks and for spearphishing, for instance.

PRISM-Proof Security Considerations

PRISM-Proof Security Considerations, an IETF working draft:

Abstract

PRISM is reputed to be a classified US government that involves
covert interception of a substantial proportion of global Internet
traffic. This document describe the security concerns such a program
raises for Internet users and security controls that may be employed
to mitigate the risk of pervasive intercept capabilities regardless
of source.

If you're interested in this area of security and encryption, also read On the NSA, by Matthew Green.

OS X Auditing

Mentioned here more for completeness: OS X and OS X Server include the OpenBSM security auditing tools, and here is one write-up.

The auditing configuration is found in the /etc/security/audit_control file, and the default setting is lo and the ill-documented aa; the former is for auditing logins and logouts, and the latter is “A new audit event class 'aa', for post-login authentication and authorization events”. This is quite minimal auditing coverage, and (if not reset prior to a breach) quite possibly not sufficient to figure out what really happened. (And this assumes that the attacker doesn't disable and remove these logs.)

And for more details on OpenBSM and auditing, the OpenBSD Security Auditing and the Apple Common Criteria materials.

ServerFault

From ServerFault via HN, a detailed answer to the generic My server's been hacked EMERGENCY question.

An example of a printer vulnerability

Per HPSBPI02398 SSRT080166 rev.1 - Certain HP LaserJet Printers, HP Color LaserJet Printers, and HP Digital Senders, Remote Unauthorized Access to Files and per CVE-2008-4419, the HP LaserJet 2410, LaserJet 2420, LaserJet 2430, LaserJet 4250, LaserJet 4350, LaserJet 9040, and LaserJet 9050 and HP Color LaserJet 4730mfp, HP Color LaserJet 9500mfp, and HP 9200C Digital Sender printer devices all have a security vulnerability.