Serilog and ElasticSearch

Mads Tjørnelund Toustrup

Senior R&D Developer

[email protected]

madstt.dk

@madstt

ONUG 08.09.2015

basic setup

  • Ubuntu 14.4 VM - Azure hosted
  • ElasticSearch 1.4.4 with Kibana 3
  • nginx 1.7.11
  • application logs and iis log forwarding

application logs

serilog helpers

NuGet package...

install-package d60.SerilogHelpers -ProjectName MyApp

initialize...

Logging.Initialize("myApp", "local");

Get It!

get a static readonly ILogger in each class that needs to log stuff

    public class MyRequestHandler : IRequestHandler<SomeRequest>
    {
        static readonly ILogger Log = Logging.GetCurrentClassLogger();
        
        public Response Handle(SomeRequest request) {
            Log.Info("Turning on something for {SomeId} now", 
                    request.SomeId);

            ActuallyDoIt(request.SomeId);

            return Ok();
        }

        void ActuallyDoIt(int someId) { ... }
    }

default enrichers

  • UsernameEnricher
  • AzureWebInstanceEnricher
  • MachineNameEnricher
  • RequestOriginEnricher

configuration options

extension methods

  • EnableRollingFiles
  • EnableTraceLogging
  • EnableConsoleLogging
  • EnableErrorMails

configuration options

modes

  • ShipLogsToAggregator
  • ShipLogsToLocalAggregator
  • DoNotShipLogs

configuration example

Logging.Initialize("myApp", AppSettings.Environment.ToString(),
    password: "myPassword",
    mode: AppSettings.Environment == Env.Local
                ? Logging.Mode.DoNotShipLogs
                : Logging.Mode.ShipLogsToAggregator,
    configurationCustomizer: config =>
    {
        config.Enrich.With(new HttpRequestIdEnricher());
        config.Enrich.With(new HttpRequestRawUrlEnricher());
        config.Enrich.With(new CorrelationIdEnricher(() =>
        {
            ...
            
            return correlationId.ToString();
        }));

        if (AppSettings.Environment == Env.Local)
        {
            config.EnableTraceLogging();
            config.EnableRollingFiles();
        }
        else
        {
            config.EnableErrorMails(...);
        }
    });

                        

iis log forwarding

currently two options:

  • Azure iis log forwarding
  • on-premise iis log forwarding

Azure iis log forwarding

  1. azure web apps to azure storage
  2. azure storage to elasticsearch

on-premise iis log forwarding

  • using LogFlow's built-in iis reader
  • attaching environment and application name
  • running as a Windows service

security

first option

Pros:

  • easy to install
  • easy to configure
  • fits into the elasticsearch configuration
  • LDAP and Active Directory integration
  • ip filtering
  • role-based access control

Cons:

  • Pricey: €5920 per node per year
  • Bundle: €29600 for 10 node bundle (!)

security

actual choice

Pros:

  • free and open-sourced
  • proxy server, load balancer, http cache, etc.
  • Apache-like configuration
  • basic authentication
  • SSL
  • persistent http connections

Cons:

  • N/A

other nginx stuff

“Apache is like Microsoft Word, it has a million options but you only need six. nginx does those six things, and it does five of them 50 times faster than Apache.” -- Chris Lea

Stats:

  • 15.15% (132,443,391) share of all sites
  • 22.61% (226,105) share of the top million busiest sites
Source: http://news.netcraft.com/archives/2015/08/13/august-2015-web-server-survey.html