Layer Seven Security

Top Five Tips for System Recommendations

Layer Seven on June 12, 2018

System Recommendations in SAP Solution Manager connects directly to SAP Support for real-time patch updates. It also connects to each system within SAP landscapes to monitor patch levels. SysRec downloads corrections for security vulnerabilities from SAP Support to each system and integrates with other areas in Solution Manager for change impact analysis, change management, and test management. SAP customers can therefore discover unapplied patches, bundle patches into change requests, and plan and execute test plans for patch cycles from a single integrated platform.

This article provides suggestions for optimizing System Recommendations to improve the performance of the application and the user experience. The tips will enable you to minimize false positives, identify and troubleshoot errors, and personalize the user interface.

System Recommendations reads the Landscape Management Database (LMDB) to determine the version and support pack levels for installed software components in each system. Therefore, the LMDB should be configured correctly, regularly updated and synchronized with the System Landscape Directory (SLD). This will reduce the likelihood of false positives such as the display of notes for irrelevant components, databases and operating systems. Kernel registration in the SLD will also help to minimize false positives. Alternatively, irrelevant components can be set to inactive in the customizing table AGSSR_OSDB to exclude them from the results returned by SysRec.

The background job SM:SYSTEM RECOMMENDATIONS periodically updates System Recommendations by connecting to SAP support and to managed systems to calculate unapplied notes. Processing errors for the object ASG_SR should be monitored using the Application Log (transaction SLG1). Alerts for job errors including automatic email notifications should be configured using Business Process Monitoring (BPMon) in Solution Manager.

System Recommendations excludes notes that are irrelevant, postponed or discontinued.  Therefore, it displays results for notes that have the implementation status New or New version available. Since the available status options don’t include options for notes with manual corrections that have been implemented, a custom status option for such notes should be configured by maintaining table AGSSR_STATUS. This can be performed using transaction SM30. Customers can also create custom status options to group notes by patch cycle, project or other criteria. In the example below, we’ve assigned a group of notes to the custom status group Q3 2018 and filtered the results to list the notes assigned to the group.

Status changes performed by users for notes are logged by System Recommendations. The changes are tracked in the details section for each note.  This section also tracks comments entered by users for notes. Comments are useful for tracking discussions between users that could impact implementation decisions including the approach, rationale, and timeline for applying security patches. Changes and comments entered by users can be viewed in table AGSSR_SYSNOTEC.

Finally, Fiori tiles can be configured in SysRec to create shortcuts for notes for specific systems, groups, and other variables. The tiles are accessed from the Fiori Launchpad and can be assigned to custom or standard groups. Once saved to the Launchpad, the results for each tile are automatically updated by System Recommendations.

Monitoring the SAProuter with SAP Solution Manager

Layer Seven on May 18, 2018

The SAProuter performs a pivotal role in SAP landscapes by filtering SAP traffic using a more granular approach than is possible with conventional network-level firewalls. As a stand-alone program, it is commonly installed in DMZ servers that support network services rather than SAP applications.

The SAProuter is often targeted by attackers given it’s function as the gateway to SAP systems. There are several attack vectors targeting known vulnerabilities in earlier versions of the program. Therefore, it’s important to regularly update the SAProuter to the latest release and patch level. You can refer to note 1897597 for release information and note 1921693 for instructions for updating the program. Other recommendations include changing the well-known default port and blocking remote access to the SAProuter. This could be abused to control the SAProuter from external clients or hosts. It can also be exploited to modify the route permission table.

The route permission table is maintained in the saprouttab file stored in the working directory of the SAProuter and controls route strings between hosts.  It applies an access control list to permit or reject connections between source and target systems through the SAProuter. Standard entries in the route permission table have the syntax P (Permit) /S (Secure) /D (Deny) <source-host> <destination-host> <destination-port or service> <password>. The password option for permitted connections is optional.

The access control list should be as restrictive as possible and only permit the necessary connections. Wildcards (*) should not be used in the destination host and port fields. The rule D * * * * should be included as the last entry in the list to explicitly deny all connections that are not defined in the route permission table.

Lastly, the access list should be configured to support only authenticated and encrypted connections using the K prefix for positive entries. This requires the configuration of Secure Network Communications (SNC) for the SAProuter. For detailed instructions, refer to the SAP guide for SAProuter SNC Configuration.

The SAProuter can be monitored with SAP Solution Manager. The Solution Manager Diagnostics (SMD) agent should be installed on the server hosting the SAProuter. The Remote OS Script Collector (ROSCC) is also required to run OS commands through the Monitoring and Alerting Infrastructure (MAI) of Solution Manager. The next steps are the registration of the SAProuter in Solution Manager and the execution of the steps for managed system setup. Once completed, the SAProuter is available for monitoring.

The route permission table can be monitored by Solution Manager to automatically detect insecure entries including unauthenticated and unencrypted connections and entries with wildcards in the destination and port fields. An example is provided below.

 

The release and patch level of the SAProuter can be checked using the ROSCC. The port used by the SAProuter and whether the program accepts commands from remote hosts can also be monitored with the ROSCC.

The SAProuter log can be read to detect connections rejected by the SAProuter based on the route permission table. An example of an alert is provided below. Click on the image to enlarge.

Email notifications are automatically triggered by Solution Manager for alerts. See below.

 

Analysts can execute guided procedures in Solution Manager to investigate alerts and document findings. An example is provided below for Securing the Route Permission Table.

The guided procedure provides a framework for discovering insecure entries in the saprouttab file, identifying required entries, maintaining the route permission table and finally, monitoring the SAProuter log for rejected connections.

Detailed reference documentation is included for each step in the procedure.

GDPR Compliance with SAP Solution Manager 7.2

Layer Seven on April 23, 2018

The General Data Protection Regulation (GDPR) will be enforceable throughout the European Union in less than a month. The regulation specifies how personal data should be managed and applies to organizations that collect data on EU citizens, regardless of whether or not they are located within the EU. GDPR requirements include data protection measures to secure systems that store or process personal data (privacy by design). They also include breach notification requirements that oblige organizations to notify customers within 72 hours of any compromise of their personal data.  Organizations may be fined up to €20 million or 4% of global turnover for non-compliance, whichever is greater. Therefore, GDPR is regarded as the most stringent data security framework to date.

In SAP environments, the requirement for privacy by design can be met by hardening and patching systems using Service Level Reporting, Security Dashboards, and System Recommendations, available in SAP Solution Manager. This will support automated monitoring for security vulnerabilities and missing security patches that could be exploited to target SAP systems and compromise personal data. Interface Monitoring and the Monitoring and Alerting Infrastructure (MAI) in Solution Manager can be used to perform event monitoring to detect potential attacks in near-time.

The requirement for breach notification can be met through systematic monitoring of access to personal data, supported by incident response procedures aligned to GDPR disclosure standards. This can also be performed using SAP Solution Manager. The flowchart below summarizes the process for logging access to personal data in SAP systems, automatically monitoring logs for access violations and finally, responding to potential breaches using guided procedures.

The logging for access to personal data is performed using SAP Read Access Logging (RAL). RAL supports logging for access and changes to sensitive data fields in SAP systems, including custom fields. It can also be used to log access to SAP tables containing bulk records of sensitive data. Data fields and tables are tagged for logging using RAL scenarios. Once configured, RAL configurations can be transported across systems within a landscape. User exclusion lists can be maintained to exclude authorized users from logging. The RAL recording below captures access to banking information accessed through vendor maintenance in SAP ERP.

Access violations are logged in RAL tables. The raw log can be viewed using SRALMONITOR. In the example below, RAL has logged access to banking data belonging to vendor number 1752249 by the user ATTACKER on April 24, 2018 at 1.39 PM. Click on each image to enlarge.

The next step involves automated monitoring of RAL logs and triggering alerts and notifications for suspected breaches to personal data. This is performed using the MAI in SAP Solution Manager. MAI continuously monitors the contents of RAL tables as often as every minute. The MAI Event Calculation Engine alerts for each incident recorded in RAL tables and triggers email notifications to security analysts.

The alert details include the date and time of the event, the impacted system and client, and the name of the user that accessed the personal data.

The final step involves incident response. Analysts can execute guided procedures to investigate suspected breaches of personal data directly from each alert. Once executed, guided procedures provide standard operating procedures including automated steps for security investigations.

Guided procedures provide a framework for ensuring incident response procedures are aligned with GDPR requirements including disclosures to impacted customers, employees or other parties within the reporting window mandated by the regulation. Completed guided procedures are archived to provide an audit trail for compliance purposes.

The comprehensive coverage provided by SAP Solution Manager for the privacy by design and monitoring and disclosure requirements of GDPR presents a powerful alternative to third party software platforms.  Contact Layer Seven Security to discuss how Solution Manager can help your organization become and stay GDPR-compliant.

Monitor Dangerous Function Module Calls with SAP Solution Manager

Layer Seven on March 28, 2018

SAP systems operate in highly interconnected landscapes integrated by numerous interfacing technologies.  The most common interface technology is the RFC protocol. The RFC protocol enables remote-enabled function modules (RFMs) to be called in remote systems. Some RFMs can be exploited to perform dangerous, administrative commands in target systems. For example, the function module BAPI_USER_CREATE can be used to create or maintain users. RFC_ABAP_INSTALL_AND_RUN can be used to register and execute arbitrary code. External commands including operating system commands can be executed using SXPG_CALL_SYSTEM and SXPG_COMMAND_EXECUTE. Therefore, monitoring for the execution of dangerous RFMs is critical for detecting potential attacks against SAP systems.

This article discusses how SAP Solution Manager detects and triggers alerts for dangerous RFM calls using Interface and Connection Monitoring (ICMon) and the Monitoring and Alerting Infrastructure (MAI). The article also discusses how the Guided Procedure Framework in Solution Manager can be used to create automated workflows for alert handling and forensic investigations.

ICMon provides a centralized platform for monitoring communications between systems within and across SAP landscapes. The application is accessed from the System and Application Monitoring group in the Fiori Launchpad.

Monitoring scenarios must be configured before using ICMon. The scenarios define the target systems and interface channels for monitoring. They also define the direction of the communications traffic. ICMon supports monitoring for both internal and external systems. It also supports several communication protocols including not just synchronous, transactional, queued, and background RFCs but Web Services, Gateway (OData) connections, HTTP, IDoc, CRM, PI and Cloud services.

Once configured, Solution Manager starts to collect usage data for each scenario at regular intervals through background jobs. It also generates dynamic topologies for each scenario to visualize connections. Channels are color coded based on performance, availability, and configuration issues or exceptions detected by Solution Manager.

Monitoring for specific function modules can be performed by maintaining blacklisted RFMs for RFC interface channels in each scenario. The Number of RFC Executions metric should then be enabled to automatically trigger alerts for the execution of any of the RFMs.

The channel will be colored red in the topology if a dangerous RFC function module call is performed.

The Alert Ticker displays open alerts in the Overview screen.

 

Alerts can be managed from the Alert Inbox of the MAI.

The Alert Details specify the function module and the RFC destination used to call the RFM, as well as details of the calling system, called system, and the timestamp of the event.

The details are also included in attachments appended to email notifications sent by Solution Manager.

 

The Guided Procedure Framework (GPF) in Solution Manager can be used to create standard operating procedures for investigating dangerous RFM executions. The procedures can be started by selecting the option to Start Guided Procedure in each alert. Once initiated, the guided procedure will provide investigators with detailed instructions for performing forensic investigations and log the progress of each step in the procedure.

 

SAP Solution Manager is ITIL-Certified for Information Security Management

Layer Seven on January 8, 2018

The SAP Integration and Certification Center (ICC) has been validating and certifying solutions from partners and software vendors for over twenty years. The certifications provided by the ICC are based on rigorous testing and enable customers to invest with confidence in technologies that integrate with SAP solutions. This includes technologies that support security scenarios such as automated vulnerability management, code scanning and threat detection.

The ICC cannot certify SAP’s own product offerings since self-certification does not provide the same level of assurance as independent certification. However, SAP platforms are often certified by recognized certification authorities. SAP Solution Manager, for example, is certified by organizations such as SERVIEW. In fact, Solution Manager is one of the most awarded service management platforms in the market and certified for all 18 certifiable processes of the ITIL framework, including Information Security Management.

ITIL is the Information Technology Infrastructure Library and provides best practices to support the design, management and monitoring of IT infrastructure and optimization of service levels for end users. The framework consists of five distinct lifecycle phases for service strategy, design, transition, operations, and continuous improvement. It includes key performance indicators to identify problems, measure performance, and track progress.

IT Security Management is a process within the Service Design lifecycle of the most recent version of the ITIL framework. It includes four sub-processes for the design of security controls, the performance of regular security reviews, and the management of security incidents. The sub-processes are targeted at preventing, detecting and containing security intrusions and breaches. The chart below maps each sub-process to relevant applications available in SAP Solution Manager.

ITIL v3 – IT Security Management

Applications such as Configuration Validation, Service Level Reporting and the Dashboard Builder enable customers to enforce security baselines for SAP landscapes and monitor compliance against security KPIs. System Recommendations automatically detects missing security patches through a direct connection to SAP support. Interface Monitoring detects potential breaches of cross-system connections. Finally, the Monitoring and Alerting Infrastructure and Guided Procedures provide an advanced framework for detecting and responding to security incidents and suspected breaches. Overall, Solution Manager provides a powerful ITIL-compliant platform for defining, implementing and sustaining secure SAP system landscapes.

 

5 Common Myths for Security Monitoring with SAP Solution Manager

Layer Seven on December 5, 2017

Does Solution Manager have a complex installation process? Is it difficult to maintain? Does it create dangerous connections with SAP systems? Is it a high value target for attackers? Does it provide no support for zero-day vulnerabilities?

This article tackles the five most common myths about SAP Solution Manager and reveals the truth behind the fiction.

The first and most common myth is that SAP Solution Manager is complex to install and difficult to maintain. In fact, the installation procedures for Solution Manager are relatively simple and standardized, especially in comparison to other SAP platforms such as ECC. Once installed, guided procedures in Solution Manager track the progress of the setup process across three major areas: System Preparation, Basic Configuration, and Managed System Configuration. Performing the configuration steps in Technical or Application Monitoring is recommended to enable the monitoring capabilities of Solution Manager.

Once configured, security-relevant applications such as System Recommendations, Dashboards, Interface Monitoring and the Monitoring and Alerting Infrastructure are enabled and ready to use. Therefore, the standard setup procedures automatically activate most of the requirements for security monitoring using Solution Manager. Since security applications use existing connections with SAP systems, there is no need to install and configure additional agents in target systems.

Maintenance is relatively straightforward. Support packs for functional enhancements and bug fixes are released at regular intervals and are applied using the Maintenance Optimizer. The guided procedures for SOLMAN_SETUP will flag any configuration issues that need to be tackled after an SP upgrade.

The second myth is that SAP Solution Manager creates dangerous RFC connections with managed systems. The RFC connections created by Solution Manager are no more or less dangerous than similar connections between other systems in SAP landscapes. Also, the risk is not removed if you decide not to perform security monitoring using SAP Solution Manager since the connections will remain in place.

The third myth is that SAP Solution Manager is a high-value target for attackers. In fact, all SAP systems are valuable targets for attackers. Since Solution Manager does not typically store or process sensitive business data, it may be a less valuable target than systems such as ECC, CRM and SRM. Also, Solution Manager performs self-monitoring to detect security vulnerabilities including misconfigurations and missing patches, and potential security breaches captured in SAP logs. In dual landscapes, Solution Manager systems can monitor each other.

Fourthly, it’s often emphasized that Solution Manager is not certified by SAP. SAP certifies third party solutions developed by independent software vendors for integration with platforms including SAP NetWeaver. SAP does not certify it’s own software platforms such as Solution Manager. However, Solution Manager is ITIL-certified by organizations such as SERVIEW for Information Security Management.

The final myth is that Solution Manager does not provide any coverage for zero-day vulnerabilities that are unpatched by SAP. Security researchers choose to deliver virtual patches for zero-day vulnerabilities through third party tools in order to induce SAP customers to subscribe to expensive licenses for such tools. This is a business decision and not due to any technical limitation in Solution Manager. Also, all zero-day vulnerabilities do not pose a critical risk to SAP systems. The fact that patches for vulnerabilities are often released many months after the weaknesses are disclosed by security researchers to SAP does not necessarily mean that SAP systems are at serious risk. SAP’s response to such disclosures depends on an assessment of the risk posed by reported vulnerabilities. This includes factors such as the complexity and range of related exploits and the impact to data confidentiality, integrity and availability.

Featured in SAPinsider: Secure Your SAP Landscapes with SAP Solution Manager 7.2

Layer Seven on November 21, 2017

Firewalls, intrusion detection systems, and antivirus solutions may not protect SAP systems against advanced cyberattacks. However, this does not necessarily mean that SAP customers have to license third-party vulnerability scanning or threat detection solutions to deal with the risk. The answer to their security questions may be closer than they realize. Bundled with standard and enterprise SAP support agreements, SAP Solution Manager 7.2 includes five integrated applications to safeguard SAP systems against cyber threats:

Service Level Reporting (SLR)
Dashboard Builder
System Recommendations
Interface and Connection Monitoring (ICMon)
and the Monitoring and Alerting Infrastructure (MAI)

Read the full article

Monitor Table Access with SAP Solution Manager

Layer Seven on October 28, 2017

There has never been a greater need to monitor access to sensitive data in SAP systems. SAP data is increasingly accessible from access points outside network perimeters. Data in SAP systems is also targeted by attackers for cybercrime and corporate espionage. This article demonstrates how you can use SAP Solution Manager to detect and contain potential information leaks in your SAP systems before they lead to a full-blown data breach. The demonstration leverages the advanced diagnostics capabilities of the Monitoring and Alerting Infrastructure (MAI) in Solution Manager. MAI connects directly to SAP systems to monitor event data in SAP log files and tables.

The specific scenario that will be used to demonstrate the monitoring capabilities of MAI is access to SAP logon data in table USR02 using data browsing transactions such as SE16. However, the scenario can be adapted for other sensitive data including financial, employee and product-related information in SAP tables.

The first step is to configure a logging scenario to log access to table USR02 through SE16. This can be transported from a source system or configured directly in a target system using Read Access Logging (RAL). For the configuration option, we will define a log purpose and domain.

The next step is to create a recording to capture the data fields and values using SAP GUI. Read Access Logging also supports logging scenarios for data accessed through web browsers, web services, remote function calls, and OData services via the SAP Gateway.

Once the recording is completed, we will define the log contexts, groups and conditions in the RAL configuration.

Finally, we will maintain User Exclusion Lists for users that should be excluded from logging and activate the scenario.

The activation of the scenario will trigger logging for access to table USR02 through SE16. The log records can be read using the RAL Monitor.

Although RAL logs access to sensitive data with timestamps and usernames, it does not trigger an alert or notification for logged events. Therefore, the next step is to configure metrics, alerts and notifications using MAI in SAP Solution Manager.

Custom metrics and alerts are defined in the Template Maintenance section of System Monitoring within Solution Manager Configuration. Metrics and alerts can be at the database, host, system or instance level and are contained in monitoring templates. For custom metrics, we need to specify a metric name, data type and unit of measure. We also have to specify options for data collection including collector types and intervals. For the RAL scenario, we will use the RFC option for a table connector with a collection interval of 5 minutes. We will also specify the RAL table and configuration ID in the metric input parameters. Based on the configuration, MAI will connect to the RAL table in each system every 5 minutes and search for the configuration ID of the logging scenario.

For the alert, we will define a custom name and description, select the category and severity, and maintain the notification settings to automatically generate an email and/ or SMS for the alert. We will also maintain recipient lists for the notifications. To avoid alert flooding, we can adjust the interval for follow-up notifications based on number of minutes, hours or days. We can also group multiple alerts into a single notification. To activate the alert, we need to assign the metric to the alert and then assign the template containing the alert to the target systems in the landscape.

Below is the alert and email notification generated by Solution Manager for the RAL event. The alert details include the username and source IP address of the user that accessed table USR02 using transaction SE16. This is displayed in the Text Value.

 

 

 

 

 

The third step is the configuration of a guided procedure to support the investigation of the alert. This can be performed using the Guided Procedure Authoring Tool in Solution Manager. In the example below, we have created a 2-step guided procedure to firstly, access the RAL Monitor in order to review the event and secondly, investigate other actions performed by the user logged in the Security Audit Log. The Guided Procedure includes automatic transaction jumps to the required screens and reports in the target system.

Log settings, monitoring templates and guided procedures can be licensed and transported directly into managed systems and SAP Solution Manager to accelerate the implementation of threat detection using MAI. Contact Layer Seven Security to learn more.

 

Equifax Data Breach: Attackers Exploited an Unapplied Security Patch, not a Zero-Day Vulnerability

Layer Seven on September 16, 2017

On September 15, Equifax released a statement to confirm the initial attack vector that led to the compromise of personal information relating to 143 million consumers in the US, UK and Canada targeted an Apache Struts vulnerability within a web application that supports the organization’s online dispute portal. The patch for the vulnerability had been available since March but had not been applied by Equifax at the time the breach was detected on July 29. The patch was subsequently applied by Equifax but it was too late – the damage had been done.

Predictably, Equifax’s patching procedures have been cast into doubt with many questioning why the organization took four months to patch an external-facing web application that accessed large-volumes of sensitive information.  The doubts were evidently shared by the Board of Directors at Equifax: both the Chief Information Officer and the Chief Security Officer were forced out last week.

Fortunately, few SAP applications are impacted by the Apache Struts vulnerability addressed by CVE-2017-5638. Although many SAP products including Banking, BusinessObjects, and Sybase use the Apache framework, very few products use the Struts library within the framework.

However, SAP customers are strongly advised to review and revise their patching efforts in light of the breach. Despite concerns related to zero-day vulnerabilities, the root cause of the vast majority of breaches remains poor security practices rather than zero-day attacks. This includes ineffective patching procedures that open a wide window of opportunity for attackers to exploit known vulnerabilities before they are patched by organizations. This point was emphasized by a statement from Fortinet with the recent release of the company’s Global Threat Landscape Report. According to Fortinet, “Cybercriminals aren’t breaking into systems using new zero day attacks, they are primarily exploiting already discovered vulnerabilities”.

SAP customers can discover and apply security patches for SAP products using System Recommendations (SysRec). SysRec is an application within SAP Solution Manager that connects directly to SAP Support for real-time patch updates. It also connects directly to each system within SAP landscapes to monitor patch levels. SysRec downloads corrections for security vulnerabilities from SAP Support to each system. It also integrates with other areas in Solution Manager including Usage Logging and Solution Documentation for change impact analysis, Change Request Management (ChaRM) for managing changes, and Test Management for testing and deployment.

Discover Vulnerable System Connections with Interface Monitoring

Layer Seven on August 23, 2017

Interface Monitoring provides the answer to one of the most vexing questions in SAP security: where are our vulnerable cross-system connections and how do we monitor them to ensure they’re not abused by attackers?

Although Interface Monitoring, also known as Interface Channel Monitoring or ICMon, has been available in SAP Solution Manager since version 7.10 SP05, the application has been completely overhauled in version 7.2, especially in SP05, which has been in general availability since June.

ICMon in SolMan 7.2 includes an SAPUI5 graphical display that automatically maps the entire landscape topology in a single screen (see below). Topologies are generated by ICMon based on so-called monitoring scenarios configured in Integration Monitoring within SolMan configuration.

During scenario creation, you specify the systems and channels to monitor in each scenario. Multiple scenarios can be created to monitor different channels, systems, environments or other variables. Scenarios can also be landscape-wide to include all available systems and even cross-landscape to monitor systems located in different SAP landscapes.

Unlike some third party security tools that focus exclusively on RFC communications, ICMon can support monitoring for any SAP-supported protocol. This includes not only RFC, but HTTP, HTTPS, IDoc and Web Services.

Once the scenarios are configured, you can select from the list of available scenarios from Scope Selection in ICMon to monitor the scenario.

ICMon’s ability to automatically generate a graphical topology of cross-system connections enables users to discover vulnerable interfaces between systems including trust RFC relationships between systems in different environments. Trust relationships and stored credentials in RFC destinations could be exploited by attackers to, for example, pivot from vulnerable development or test systems to productive systems.

However, ICMon doesn’t just generate a static topology of system interfaces. It also continuously collects metrics and usage data for each channel to monitor availability, configuration and performance errors. Errors and warnings are displayed in both the ICMon dashboard (see below) and the topology.  Connections with errors or warnings are displayed in red in the topology. Successful connections are displayed in green.

Usage data includes destinations and function modules called through each RFC channel with timestamps.

Alerts configured for metrics and thresholds including security-related scenarios can be viewed in the Alert Ticker from the ICMon home screen. The alerts can also be viewed in the Alert Inbox of SAP Solution Manager. In common with alerts for other application areas, ICMon uses the Monitoring and Alerting Infrastructure (MAI). Therefore, the Guided Procedure Framework can be used to apply standard operating procedures and best practices for incident management and alert handing.