WSO2 IS + ELK Analytics — log masking with Filebeat

Privacy and data protection is a serious topic for discussion. Log files are no exception when it comes to protection. WSO2 Identity Server supports log masking to protect any sensitive data from exposing. For those who are interested in ELK integration with WSO2 IS, there is another approach to mask only the event related data from the Filebeat before the data is published to Logstash.

Prerequisites,

• Up and running ELK Stack
• WSO2 IS connected to Filebeat

The configuration is quite straightforward and easy to do. The masking is done through a Filebeat processor where we can execute JavaScript function inside. Add the following processor configuration to filebeat.yml after the input section.

processors:
  - script:
      lang: javascript
      source: >
        function process(event) {
            var msg = event.Get("message");
            msg = msg.replace(/\"username\":\"(.*?)\"/, "\"username\":\"MASKED\"");
            event.Put("message",msg);
        }
      tag_on_exception: true

Published event contains multiline log record with event data included. Therefore, we need to replace based on the json format to protect other information from possible masking.

JSON format to mask
"fieldName":"fieldValue"

From the published event, message field is extracted into a variable and then the regular expression based masking takes place. Username field is replaced with the word MASKED in the given example.

msg.replace("/\"username\":\"(.*?)\"/", "\"username\":\"MASKED\"");

This masking can take place in the Logstash level but there is a reason to do it in the Filebeat level.

Filebeat level masking never send sensitive data outside the boundary where the log file itself resides in. On the other hand, Logstash level masking may contain a risk of stolen sensitive data in the process of transport.