The Office of Information Security has created several checklists to guide Penn local IT support in handling large-scale computer security incidents. While these checklists are primarily designed to help prepare for and respond to an incident, they also serve as best practice for many areas as well. We encourage you to review them and see how they may be helpful in improving the security posture in your own area.
- Keep host and host contact information up-to-date in Assignments
- Information in Assignments is critical for determining the escalation path for problems about Penn hosts.
- Periodically (quarterly, semi-annually, programmatically) adjust the registration information for the hosts whose entries you maintain in Assignments, and delete entries for decommissioned hosts and any CNAME records that are out of date.
- Please pay particular attention to the fields:
- Hostname
- IP address
- University Budget code
- Building
- Room
- Administrative contacts for host
- The primary end user (if applicable)
- Be able to map an external IP address to the host “using” it at any given time
- This is particularly important for hosts residing behind a NAT gateway.
- Make sure you are logging and retaining NAT gateway information for a reasonable length of time (we suggest 60 days).
- Develop an inventory of the hosts you operate and are operated on your behalf
- For each service you provide, enumerate all of the application, database, and network devices used by that service
- Note the listening services on each host
- Note the kinds of data and their sensitivity [1] residing on each host
- Diagram the connections among your hosts and their major network connections.
- Include high-level features such as physical location, local firewalls, building routers, and network services such as DHCP, DNS, NTP.
- For each host containing Confidential University Data
- Are its OS- and Application-layer logs being forwarded to a centralized repository, off-box from the originating host? ISC provides Splunk as such a repository through the Data Analytics Service [2].
- Looking at the hosts’ logs forwarding to a central repository, can you reliably tell when:
- A user has authenticated to the hosts’ OS?
- A user has taken a privileged action on the hosts’ OS?
- A user has authenticated to the applications running on the hosts?
- A user has taken a privileged action within the applications running on the hosts?
- A new user account has been created on the hosts’ OS or the applications they serve?
- A web server is handling a request?
- Develop a baseline of expected system behavior
- At the application layer
- At the platform layer
- At the network layer
- Brainstorm: what does suspicious behavior look like on your host or application?
- Is it normal for admins to be doing work at 3 a.m. on a Saturday?
- Is it normal for users to be authenticating from overseas?
- What privileged actions are seldom used?
- Refer to ISC Office of Information Security’s Security Logging Guidelines for more use cases [3]
- Do you have alerts configured on your central repository to notify you of suspicious behavior?
- Do you have tools to periodically review logs in your central repository?
- Summary statistics on different kinds of events (e.g. percent failures).
- Graphs of different kinds of events over time.
Backup Creation
- Backup servers should require authentication of backup clients
- Backup servers should require authentication of backup administrators
- Employ Role-Based Access Control (RBAC) for all backups to determine who can make backups and who can restore from backups
- Backups should be encrypted at rest
- When sending backups, use encryption in transit (SSH or SFTP/FTPS for example)
- Create “golden master” (final version of a software compilation or configuration prior to release into production) prior to host deployment on the network
- Create a snapshot after initial system build to hasten recovery time
- Create snapshots after major configuration changes to hasten recovery time
- Ensure that some backups exist in offline media to help recover from data corruption/ransomware
Testing
- Test backups regularly
- Spot check file system backups
- For large/critical applications, consider documenting the length of time it takes to recover from backup for disaster recovery and planning purposes
Recovery
- Employ role-based access control lists for all recoveries
- Ensure backup date/time precedes date/time of first known exploitation
- Create an isolated environment to test the recovery of backups that may still contain malicious code
- When recovering from an incident, attain a list of IoC’s from Incident Commander and test backup for the existence of IoC’s prior to recovery on production/live environments
- Virtual machines may be restored to an isolated “host only” network
OIS has a forensics checklist available for reference, but due to the risk of data loss or corruption that may occur if the procedures are performed incorrectly, we ask that you contact our office to obtain it and receive individual guidance before attempting any forensic actions.
[1] https://www.isc.upenn.edu/how-to/network-time-protocol-ntp(link is external)
[2]https://www.isc.upenn.edu/security/log/service(link is external)
[3] https://www.isc.upenn.edu/security/log/guideline(link is external)