Unit

Monitor Logs Using Forwarders

Learn how to install, configure, secure, deploy, validate, and troubleshoot log forwarders for reliable centralized log collection.

A forwarder is a lightweight agent or relay that collects event data from a machine and sends it to a central logging destination. Forwarders commonly collect application files, Linux system logs, system journal data, or Windows Event Log channels.

Instead of manually copying files, analysts can search one central platform. Administrators can also limit host access because the forwarder transmits approved data without requiring every analyst to log in to every server.

What a Forwarder Does

A forwarding system usually separates several responsibilities:

  • Collection: reading a file, event channel, journal, or other source.
  • Forwarding: transmitting events over a network connection.
  • Parsing: identifying timestamps, event boundaries, encodings, and fields.
  • Indexing: storing events in a logical destination such as an index.
  • Searching: querying indexed events.
  • Alerting: evaluating searches or health signals and notifying operators.

A forwarder may perform some parsing and filtering, but it is not automatically the indexer, search engine, or alerting system. Keeping these responsibilities distinct makes capacity planning and troubleshooting easier.

Forwarder Architecture and Data Flow

The normal event path is:

source host and event source
        -> input on the forwarder
        -> optional local parser or relay
        -> receiver or collector
        -> parser and indexer
        -> destination index
        -> search and alerting users
  • Source: the originating file path, channel, stream, or event identifier.
  • Input: the configuration that reads the source.
  • Parser: the component that determines event boundaries, timestamps, encoding, and fields.
  • Output: the destination address, port, protocol, and delivery settings.
  • Receiver: the endpoint that accepts forwarded events.
  • Destination index: the logical storage partition selected for the events.
  • Metadata: context such as host, source, source type, application, environment, and index.

In a direct-forwarding design, each host sends events directly to one or more receivers. In an intermediary design, agents send to a relay or heavy forwarder, which can route traffic to different receivers or destinations. Relays are useful when sites have limited connectivity or when centralized filtering and protocol conversion are required.

Deployment or control traffic is different from event-data traffic. A forwarder may contact a deployment server to retrieve configuration while separately sending events to receiver ports. Document both paths, firewall rules, certificates, and failure behavior.

Forwarder Types

A universal forwarder is a lightweight agent intended mainly for efficient collection and transmission. A heavy forwarder is a fuller forwarding instance that can perform more extensive parsing, routing, filtering, and transformation.

Lightweight or universal forwarder — Low resource use; limited transformation; best for routine file and event collection; may not support complex routing or parsing.

Heavy forwarder — Higher CPU, memory, and storage requirements; supports richer parsing, filtering, routing, and protocol conversion; useful as a relay or integration point; adds operational complexity.

Agentless collection — Uses remote protocols or collectors instead of installing software on every host; useful where installation is prohibited; can require inbound access, elevated permissions, polling, and extra network overhead.

Choose a lightweight agent when the host needs simple, secure, low-overhead forwarding. Choose a heavier relay when data must be transformed, routed by application or site, filtered before transmission, or converted from another protocol. Agentless collection is convenient for limited cases, but it can be less reliable for high-volume or near-real-time data.

Plan the Deployment

  1. Inventory sources: operating system, hostname, application, file paths, event channels, journal units, rotation behavior, and expected event rate.
  2. Group similar hosts, such as production Linux web servers, development servers, or Windows domain controllers.
  3. Define naming conventions for host identifiers, source types, applications, environments, indexes, receiver groups, and deployment groups.
  4. Document retention, search, compliance, privacy, and destination requirements before collecting data.
  5. Plan outbound connectivity, DNS, firewall rules, receiver ports, TLS certificates, authentication, and whether the design must be outbound-only.
  6. Estimate event volume, peak bursts, bandwidth, local disk buffering, CPU, memory, queue capacity, and receiver scaling.

Use predictable metadata such as environment=production, application=web, and a stable source type such as web_access. Avoid creating a new destination for every host unless isolation is required; group data by security, retention, or search needs.

Install Forwarders on Linux and Windows

On Linux, install the vendor package using the operating system's package tools or a signed deployment package. Create a dedicated service account where supported, grant it only the permissions needed to read selected logs, and protect its configuration and certificates. Set the service to start at boot, but plan maintenance windows for upgrades.

sudo systemctl status log-forwarder
sudo systemctl start log-forwarder
sudo systemctl stop log-forwarder
sudo systemctl restart log-forwarder
sudo systemctl is-enabled log-forwarder

Use the actual service name supplied by the product. On Windows, install the signed package, select an appropriate service identity, grant that identity the required event-log and file permissions, and verify startup behavior.

Get-Service -Name LogForwarder
Get-Service -Name LogForwarder | Set-Service -StartupType Automatic
Start-Service LogForwarder
Restart-Service LogForwarder

After installation, verify the executable path, service account, startup mode, configuration directory, available disk space, and diagnostic-log location. Upgrade a pilot group first, retain a rollback package, and avoid changing binaries and configuration simultaneously when possible.

Configure Forwarding Destinations

An output defines one or more receiver hostnames, ports, transport protocols, and delivery behavior. Confirm that a receiver is listening before sending production data. A successful DNS lookup does not prove that the receiver port is reachable or accepting events.

[output:receivers]
servers = log-receiver-a.example.net:9997, log-receiver-b.example.net:9997
load_balance = true
transport = TLS
verify_certificate = true
acknowledgment = enabled
retry_interval = 10s

This is conceptual syntax; adapt the names and settings to the selected platform. Load balancing distributes traffic across receivers. Failover keeps trying an alternate receiver when one is unavailable. Acknowledgments provide confirmation that a receiver accepted data, but they do not necessarily prove that the event has been indexed or is searchable.

getent hosts log-receiver-a.example.net
nc -vz log-receiver-a.example.net 9997
openssl s_client -connect log-receiver-a.example.net:9997 -servername log-receiver-a.example.net

Use equivalent Windows connectivity tools when needed, such as Test-NetConnection log-receiver-a.example.net -Port 9997. Validate certificate trust, hostname matching, expiration, private-key protection, and receiver authentication.

Configure Log Inputs

An input identifies what to collect and how to label it. Monitor specific files where possible. Directory patterns are useful for rotation, but broad wildcards can accidentally collect unrelated or sensitive files.

[monitor:/var/log/myapp/access.log]
enabled = true
source_type = myapp_access
index = app_production
application = myapp
environment = production

[monitor:/var/log/myapp/*.log]
enabled = true
source_type = myapp_logs
index = app_production
exclude = /var/log/myapp/debug-*.log
recursive = false

For rotated logs, verify whether the forwarder tracks file identity, inode changes, renames, truncation, and compression. Do not assume that a wildcard alone handles every rotation scheme.

[monitor:/var/log/myapp/test.log]
enabled = false
source_type = myapp_test
index = discarded_test

Platform-specific inputs can collect system journal records or Windows channels:

[windows_event_channel:Security]
enabled = true
index = security
source_type = windows_security
application = operating_system
environment = production

The Windows service identity must have the required event-log read permissions. For Linux files, test read access as the forwarder's service account, not merely as an administrator. Check excluded paths, disabled configuration layers, recursive settings, and whether the file is actually receiving new lines.

Parsing and Event Boundaries

Search quality depends on correct event boundaries. A single application event may span several lines, while one line may contain multiple logical records. Timestamp format, time zone, encoding, and line-breaking rules affect event time and search results.

  • Assign one consistent source type to similar event formats.
  • Define multiline rules carefully so stack traces remain one event without joining unrelated records.
  • Confirm whether parsing occurs on the lightweight forwarder, a heavy relay, or the central processor.
  • Prefer central parsing when it reduces endpoint CPU or keeps parsing rules consistent, but parse locally when early routing or filtering requires it.
  • Test daylight-saving changes, time zones, non-UTF encodings, and malformed timestamps.

Incorrect event boundaries can cause duplicate timestamps, broken fields, oversized events, or events that appear missing even though they were received.

Centralized Deployment and Configuration Management

A deployment server distributes approved configuration packages to enrolled deployment clients. A configuration-management or automation system can perform the same role across operating systems and sites.

  1. Store configuration in version control with peer review.
  2. Package inputs, outputs, parsing rules, certificates, and tests as a reusable application.
  3. Assign packages to deployment groups such as production-linux-web or production-windows-security.
  4. Roll out to a small pilot, inspect check-in and effective configuration, then expand in stages.
  5. Record the previous version and define rollback steps.
[deployment_client]
management_server = config.example.net:8089
client_name = web-023
client_group = production-linux-web
poll_interval = 60s

Do not edit managed files manually unless the change process explicitly supports it. Configuration precedence can cause a local setting to be overridden by a package, or a broad wildcard to override a more specific input. Regularly compare enrolled hosts, package versions, and effective settings to prevent configuration drift.

Validate Incoming Data

Validation must cover the entire path, not just the running service:

  1. Confirm the forwarder service is running and the intended output is active.
  2. Append a recognizable test event to the source, or generate a controlled Windows event.
  3. Check agent logs for input discovery, connection, TLS, queue, and acknowledgment messages.
  4. Confirm the receiver accepted the connection and the indexer stored the event.
  5. Search for the test event and verify host, source, source type, application, environment, index, and timestamp.
  6. Measure event latency and compare expected source coverage with the host inventory.
  7. Check for duplicates, gaps, delayed events, and incorrect time zones.

A targeted search should include the known hostname and a distinctive test value. Also inspect receiver-side ingestion metrics, because a successful socket connection does not guarantee indexing or search visibility. The index and event fields should match the onboarding design.

Monitor Forwarder Health

Health monitoring should cover service state, last-seen time, connectivity, throughput, queue utilization, blocked outputs, ingestion delay, errors, and dropped events. Maintain an inventory containing hostname, operating system, forwarder version, group, sources, destination, last check-in, and last event time.

  • Alert when an expected host stops reporting.
  • Alert when a configured source has no events for an appropriate interval.
  • Alert on sustained queue growth, blocked outputs, receiver failures, or certificate errors.
  • Compare event rate with historical baselines to detect silent collection failures and noisy sources.
  • Monitor free disk space because persistent queues and diagnostic logs consume local storage.

Reliability, Performance, and Scale

Buffering holds events temporarily in memory or on disk. A persistent queue is disk-backed buffering that can preserve events during a temporary downstream outage. Backpressure occurs when downstream congestion slows or stops input processing.

Persistent queue — Preserves events during receiver outages; improves resilience; consumes disk and cannot protect data beyond its capacity; use when short outages must not create gaps.

Load balancing — Distributes traffic across receivers; improves capacity and availability; requires consistent receiver configuration; use for large or redundant deployments.

Failover — Selects another receiver after a connection failure; improves continuity; may delay delivery or reorder events; use when receivers have equivalent destination behavior.

Acknowledgment — Confirms receiver acceptance; reduces uncertainty about delivery; adds protocol and latency overhead and may not confirm indexing; use for important or loss-sensitive streams.

Filtering or masking — Reduces volume or removes sensitive values; saves bandwidth and limits exposure; can remove information needed for investigations; use only with documented data-governance approval.

For high-volume sources, measure peak rather than average rate. Reduce unnecessary debug logs, add receivers, use load balancing, and size queues for the expected outage window. During maintenance, drain or pause inputs when supported, confirm queues are empty or safely retained, stop the service cleanly, and verify continuity after restart. Abrupt termination can cause duplicates or gaps depending on file tracking and acknowledgment behavior.

Security Practices

  • Use encrypted transport and authenticated receivers.
  • Validate certificate chains, hostnames, expiration, and revocation policy.
  • Protect private keys, credentials, tokens, and configuration files with restrictive permissions.
  • Run the agent with least privilege and grant read access only to required files or event channels.
  • Restrict receiver ports to approved source networks and use outbound-only communication when appropriate.
  • Classify sensitive data before collection; filter or mask only under an approved policy.
  • Control which destination index receives security, privacy-sensitive, and operational data.

Never place passwords or private keys in broadly readable deployment packages. Rotate certificates and credentials before expiration and test the replacement with a pilot group.

Practical Example: Linux Web Access Logs

Install a lightweight forwarder on the web host, grant its service account read access to the access and error logs, and configure a redundant receiver group.

[monitor:/var/log/nginx/access.log]
enabled = true
source_type = web_access
index = web_production
application = nginx
environment = production

[monitor:/var/log/nginx/error.log]
enabled = true
source_type = web_error
index = web_production
application = nginx
environment = production

[output:receivers]
servers = log-receiver-a.example.net:9997, log-receiver-b.example.net:9997
load_balance = true
transport = TLS
acknowledgment = enabled

Append a test request, then search for its path. Confirm the host, source path, source type, event timestamp, and destination index. Include rotation testing before declaring the source complete.

Practical Example: Windows Security Events

Install the agent, use a service identity with the required Security channel permissions, and assign the events to a restricted security destination.

[windows_event_channel:Security]
enabled = true
source_type = windows_security
index = security
application = windows
environment = production

Generate or identify a recent controlled login event, then confirm that it is searchable with the correct host and event metadata. Avoid collecting every channel by default; onboard channels according to investigation and retention requirements.

Operational Troubleshooting

Use this sequence: service, effective configuration, input permissions, local event detection, network path, receiver acceptance, metadata, and search visibility. This separates an input failure from an output failure and from a destination indexing failure.

Service will not start — Invalid syntax, certificate or port error, permissions, dependency, or damaged installation. Check service status, agent logs, recent configuration, ownership, disk space, and dependencies; correct the error and restart.

Agent runs but no central events arrive — Wrong receiver, firewall or DNS failure, inactive listener, TLS trust failure, or wrong output scope. Test the port, inspect output logs, verify the listener, certificates, and active configuration.

File is not collected — Wrong path, wildcard, permissions, disabled input, exclusion, inactive file, or rotation behavior. Check effective inputs, test service-account read access, inspect discovery logs, and append a recognizable event.

Wrong index or source type — Missing or overridden metadata, precedence conflict, or broad wildcard. Inspect effective precedence, narrow patterns, set metadata explicitly, and perform a targeted search.

Delayed or slow events — Receiver congestion, network loss, backpressure, local resource exhaustion, or excessive volume. Review queues and throughput, check receiver and network health, add capacity, and reduce unnecessary volume.

Duplicates or gaps after outage — Unclean shutdown, limited queues, rotation timing, or retry without end-to-end acknowledgment. Compare source and central timestamps, review queue and acknowledgment settings, test rotation, and document duplicate handling.

Exam-Relevant Notes

  • A forwarder collects and transmits data; an indexer stores it; search and alerting are separate responsibilities.
  • A receiver accepts event data, while a deployment server distributes configuration.
  • A universal forwarder minimizes endpoint overhead; a heavy forwarder supports richer routing and transformation.
  • A successful service status does not prove that inputs are readable or outputs are connected.
  • Metadata errors can make correctly received events appear missing from expected searches.
  • Persistent queues improve outage tolerance but require disk capacity and have finite limits.
  • Acknowledgment confirms acceptance, not necessarily indexing or search visibility.

For adjacent administration skills, review file permissions, disk-space checks, packet inspection with tcpdump, and centralized log platforms.