Log Routing in ECS Fargate Tasks

- 4 mins read

Prologue

At times we need routing different ECS Fargate task container application logs to different logging destinations. For example, nginx container application has two important logs: /var/log/nginx/error.log & var/log/nginx/access.log. Let’s say we need to route both the logs to two different CloudWatch log groups or one to CloudWatch and another to a different logging, like Firehose.

This article delves into detailed procedure of sending different logs to different destinations using AWSFirelens container as side car.

Building Blocks

This solution will contain following pieces:

  • There will be two containers: Nginx (nginx application container is taken as example here) & log_router (fluent-bit side car).
  • A custom Firelens configuration file loaded in AWS fluent-bit image.
  • A bind mount volume in the task definition and volume mount point used in the Nginx as well as the fluent-bit side car container. Path (container path) for both the containers’ mount-point should be same to the nginx log /var/log/nginx.

NOTE: Using a mount point facilitates the fluent-bit side car container with access to the log directory on nginx container.

Steps

  1. Create a task definition by adding a “Bind Mount” type “Volume” (this article uses “logfile-directory” as the volume name).
Nginx Container
  1. Add a container (Nginx in this case) with image “nginx”. Select “awsfirelens” for Log configuration and mount point “logfile-directory” with path to the log file location. The mount point section of the container in Json format will look like:
    "mountPoints": [
        {
            "readOnly": null,
            "containerPath": "/var/log/nginx",
            "sourceVolume": "logfile-directory"
        }
    ]
Fluent-Bit container
  1. As we select “awsfirelens” as the logging configuration while adding the Nginx container definition using AWS console, another container definition is automatically added to the task definition with container name “log_router”.
  2. In the “log_router” container, add mount point with the same “Source volume” (logfile-directory) and log path (/var/log/nginx) as in Nginx container.
  3. Now, a custom firelens configuration file — entailing the details of routing different log files to different destinations. This can be done in two ways:
    a) Store the configuration file in an S3 bucket and refer that in the firelens configuration.
    b) Store the file in the fluent-bit container image — which, in essence, is customizing the fluent-bit image.

NOTE: As mentioned in [1], Fargate only supports the file configuration file type, so we will use file type: “Tasks hosted on AWS Fargate only support the file configuration file type.”

Steps to add the configuration file to the container

a) Create a configuration file with name fluentbit.conf with following content:

[INPUT]
    Name    tail
    Path    /var/log/nginx/access.log
    Tag     access
[INPUT]
    Name    tail
    Path    /var/log/nginx/error.log
    Tag     error
[OUTPUT]
    Name                cloudwatch
    Match               access
    region              us-east-1
    log_group_name      nginx-access-logs
    log_stream_prefix   access-logs
    auto_create_group   true
[OUTPUT]
    Name                cloudwatch
    Match               error
    region              us-east-1
    log_group_name      nginx-error-logs
    log_stream_prefix   error-logs
    auto_create_group   true

Quick Notes about the above configuration file
The above configuration uses input [2] and output [3] fluent-bit plug-ins. The important keys here are the “Tag” in input and “Match” in output. Fluent-bit tags the Tag (“access” & “error” in this case) to the respective input plug-in logs. So, logs coming from access.log will be tagged with “access” and those from error.log will be tagged with “error” as used in the two input plug-ins.To route the logs to different destinations, “Match” key is used. A further reading on this can be done at official fluent-bit documentation [4].

b) Build the custom fluent-bit docker image with following Dockerfile on local/EC2 instance with Docker:

FROM amazon/aws-for-fluent-bit:latest
COPY fluentbit.conf .

c) Create an ECR repository and push the Dockerfile built above to the ECR repository.

d) Copy the image URI from ECR repository console.

Log configuration in container definition

  1. Replace the log_router container’s image with the ECR repository URI in container definition and Update to save the image change.

  2. Add “awslogs” in the log_container log configuration. Although, it’s optional, but good to have log_container logging if something goes wrong in the fluent-bit container.

  3. Flip to “Configure via JSON” and find “firelensConfiguration” key under “log_router” container adding following:

"firelensConfiguration": {
    "type": "fluentbit",
    "options": {
        "config-file-type": "file",
        "config-file-value": "fluentbit.conf"
    }
}
Task Roles:

  1. The task execution role should have AmazonECSTaskExecutionRolePolicy managed policy.

  2. The log_container will be creating the CLoudWatch log group, log stream and would be performing logging to CloudWatch, so the task role should have following actions allowed in IAM policy:

"Action": [
                "logs:CreateLogStream",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ]
  1. Save and create the task definition. Run a Fargate task with 1.3.0/1.4.0 platform and hit the task IP address. This should create the log group, stream and there should be log entries.

Extra Shots
By default, the fluent-bit CLoudWatch output plug-in includes task metadata like: ecs_cluster, ecs_task_arn, ecs_task_definition etc [5]. But you want to have only the application logs, then it can be avoided by disabling the enable-ecs-log-metadata flag. The firelens configuration will look as follows:

"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "config-file-type": "file",
    "enable-ecs-log-metadata": "false",
    "config-file-value": "fluentbit.conf"
  }
}

References
[1] Creating a task definition that uses a FireLens configuration – Specifying a custom configuration file – https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html#firelens-taskdef-customconfig
[2] https://docs.fluentbit.io/manual/pipeline/inputs/tail
[3] https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch
[4] https://docs.fluentbit.io/manual/v/0.12/getting_started/routing
[5] Creating a task definition that uses a FireLens configuration – Using Amazon ECS metadata – https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html#firelens-taskdef-metadata