[ Datadog ] Log Monitor

When log collection is enabled, you can create a Log Monitor that alerts when the specified log type exceeds a user-defined threshold for the specified period.

1. Define the search query

   

   1. Construct a search query using the same logic as Log Explorer search. Alarms will only be set for logs retrieved by the search query.

   2. Select to monitor Log Count, Facet, or Measure.

      • Count: When * is selected, monitoring is based on the line count of logs matching the search query.
        If a tag or facet for Group by is selected, alerts will be triggered for each tag/facet.

      • Facet (Count Unique): If a facet is selected, alerts will be triggered based on the number of facets.
        If a tag or facet for Group by is selected, alerts will be triggered for each tag/facet.
        (Example: source group by host: The number of sources is monitored per host.)

      • Measure: When a measure is selected, the monitor alerts based on numerical values of Log Facets (similar to metric monitors). You must select an aggregation and group by tags/facets (min, avg, sum, median, pc75, pc90, pc95, pc98, pc99, max).

      • Up to 4 aggregation groups can be selected.

   3. Configure alert grouping strategy (optional):

      • Simple Alert: A simple alert aggregates across all reporting sources. If the aggregated value meets the set condition, a single alert is triggered. This is best for monitoring a single host's metrics or the sum of multiple hosts' metrics. This strategy can help reduce alert noise.

      • Multi Alert: A multi-alert applies alerts per source based on grouping parameters. Alert events are generated for each group that meets the set condition. For example, you can group alerts per device to receive separate alerts when each device runs out of space.

2. Set alert conditions
   Set the conditions for triggering alerts.

   

   • An alert is triggered when the metric threshold is above, above or equal to, below, or below or equal to for the selected time period.

   • Set Alert or Warning thresholds for the metric. (Only one can be set.)

   • Configure a delay for threshold comparison calculations.

   • Set the wait time for applying alerts when a new group is added.

3. Notify your team
   Select recipients (email) to receive alerts, or choose a channel from those set up in Integrations.
   Configure the message to be sent with the alert.

   

   • Alert Title: The title of the message when an alert is triggered.
     - Example: [Warning] An exception message occurred in {{log.source}} on {{host.name}}.

   • Alert Message
     - The content of the message when an alert is triggered.
     - Example

     {{#is_alert}}
     
     Occurrence Time (KST): {{local_time 'last_triggered_at' 'Asia/Seoul'}}
     
     ## [Warning] An exception message occurred in {{log.source}} on {{host.name}}. 
     Please check.
     
     Message:
     {{log.message}} 
     
     {{/is_alert}}

   • Use Message Template Variables
     Check available templates and variables for use in the Alert title and message body.
     Reference available variables: https://docs.datadoghq.com/monitors/notify/variables/?tab=is_alert

   • Notify your services and team members settings
     Integration channels like Opsgenie, Slack, TEAMS, webhook, and email are displayed.
     Set the channels or email recipients for alert notifications.

   • Content displayed settings (Message composition settings)
     Choose whether to include automatically added content such as query/snapshot in the message.

   • Include Triggering tags in notification title
     Display tags of the affected targets in the alert message title.

   • Include a table of the top 10 breaching values
     Include up to 10 logs related to the alert in the message.

   • Aggregation settings
     If a group is selected in Select Monitor scope, Multi Alert is automatically selected.

   • Renotification settings
     Resend alerts at the selected interval if Alert (Warning) or No Data conditions persist.

   • Tags settings
     Set tags for monitoring to use in Manage Monitors and Downtime scheduling.

   • Priority settings
     Set alert severity levels (P1–P5).
     Priority settings (standardized based on the following criteria).

4. Define permissions and audit notifications
Set the edit permissions and edit notifications for the Monitor in question.