Docker Logging Pitfalls – Solving multiline stacktraces and docker 16 Kb message split

Alexandru Ersenie
Categories: Tech

Today i’m going to write some words about the pitfalls that i have been experiencing when building up the Logging Stack in our company, somewhere in the middle of changing technology from BareMetal to Docker and Kubernetes.

I have never thought this may get so complex, yet so beautiful to look into and solve. A few words regarding our Logging Stack before diving into the details.

We run self hosted Kubernetes. That is awesome already, isn’t it? Not many have done it and not many do it as i am writing this. It is amazing technology, and i am so happy i can be a part of this shift to containers, stateless applications and microservices.

Besides Kubernetes we have some other components outside the cluster, that need to push their logs as well. To keep it simple, we use TDAgent as log forwarder (stand alone service on traditional servers outside the cluster / daemon set in Kubernetes) and ElasticSearch. The provocations come from getting docker logs out of docker containers…in their entireness

Process and Export Java Multiline StackTraces with TDAgent – Docker splits logs at 16000 bytes

There is nothing that ElasticSearch loves more than being fed with Json Data. After configuring Payara (our application server of choice) to log in JSON, we faced the problem of running it in a docker container.

Every now and then we would face an exception larger than 16000 bytes, that would be cut by docker, and be displayed in Elasticsearch unformatted. Docker does not allow changing this parameter to a higher value, because of objective reasons that i will not detail here.

The only feasible workaround was to use the concatenate filter in tdagent. In order to use this, you need to know how your log message will start, and how will it end. Considering you have a continuous stream of log events from a single source, the expectation is that the following message (the split one) will always be a continuation of the message before.

Let’s take an example. This is how a typical multiline stack trace looks like:


{"_Timestamp":"_current_timestamp","_Level":"INFO","_Version":"Payara 4.1",
"_LoggerName":"javax.enterprise.system.core.security","_ThreadID":"19",
"_ThreadName":"RunLevelControllerThread-1527068660997","_TimeMillis":"1527068663907",
"_LevelValue":"800","_MessageID":"4:15:NCLS-SECURITY-01115", "_LogMessage":
{"_Exception":"4:15: - Format:json - Line:multi - StackTrace:yes - Source: Payara -
Message: Exception thrown from bean",
......rest of the stack which brings the log message to over 16000 bytes
"_StackTrace":"javax.ejb.EJBTransactionRolledbackException\\n\\tat com.sun.ejb... stack test 0"}}

As we can see, a typical log message always begins with:

"{"_Timestamp"

A log message containing a stack trace will on the other hand always end with the following block:

}}

Let’s add some Tdagent configuration snippet to cover this:

# First define the source of your logging stream.
# For testing purposes we will use a synthetic logger pushing data
# to the tdagent daemon
<source>
    @type forward
    port 24224
    tag loggf.*
</source>

# Apply concatenation to all logs tagged with loggf.*
# Define the first line and the last line using regular expressions
# Set the flush_interval to a higher value: if the missing part of
# the log has not been flushed by docker in this time interval,
# flush the data available.
# If the flush timeout has been reached, reroute the data available
# when flushing, in order to not getting it lost
<filter loggf.**>
   @type concat
   key log
   stream_identity_key container_id
   multiline_start_regexp /^\{"_Timestamp/
   multiline_end_regexp /\}\}/
   flush_interval 1
   timeout_label "@NORMAL"
</filter>

# Reroute all tagged events to the @NORMAL label
<match loggf.**>
   @type relabel
   @label @NORMAL
</match>

<label @NORMAL>
# We need to apply transformation to the concatenated field, for docker
# will add the \r sequence that will mess up our final log
# Let's just remove it

<filter loggf.**>
       type record_transformer
       enable_ruby true
       <record>
      logger ${record["log"].gsub(/\r/, '')}
       </record>
       remove_keys log,message
</filter>
# Use the json parser in order to break the mapped fields into json fields
<filter loggf.**>
   @type parser
       format json # apache2, nginx, etc...
       key_name logger
       reserve_data true
       emit_invalid_record_to_error false
       replace_invalid_sequence true
</filter>
<match loggf.**>
  @type stdout
</filter>
</label>

Dealing with reserved keys inside your log stream

At the time of writing this, we were using a version of Payara Application Server that did not allow configuring Log Prefixes. Therefore, all log fields were prefixed by default by an _ character.

Not so much of a problem it it wasn’t for elasticsearch, who treats fields prefixed with _ as reserved keys, and therefore, when analysing data with Kibana, will not show as queriable fields.

An original log message would look like this:


{"_Timestamp":"2018-07-18T15:44:23.907+0000","_Level":"INFO","_Version":"Payara 4.1",
"_LoggerName":"javax.enterprise.system.core.security","_ThreadID":"19","_ThreadName":"RunLevelControllerThread-1527068660997",
"_TimeMillis":"1527068663907","_LevelValue":"800","_MessageID":"2:15:NCLS-SECURITY-01115","_LogMessage":"2:15: - Line:single -
Format:json - StackTrace:no - Source: Payara - Message: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm]
successfully destroyed."}

The good news is TDAgent has a plugin for almost everything. Meet record_transformer
he filter_record_transformer filter plugin mutates/transforms incoming event streams in a versatile manner. If there is a need to add/delete/modify events, this plugin is the first filter to try.
Let’s put this to work for the log message above:

<filter loggf.**>
      type record_transformer
      enable_ruby true
      # Rewriting fiels - self explanatory
      <record>
         payara.level ${record["_Level"]}
         payara.logger ${record["_LoggerName"]}
         payara.level_value ${record["_LevelValue"]}
         payara.messageid ${record["_MessageID"]}
         payara.thread_id ${record["_ThreadID"]}
         payara.thread_name ${record["_ThreadName"]}
         payara.time_millis ${record["_TimeMillis"]}
         @timestamp ${if record.has_key?('_Timestamp'); record["_Timestamp"]; else time.strftime("%Y-%m-%dT%H:%M:%S.%NZ");end}
         # It does get interesting here: the stack trace in case of an exception is written together with the log message inside an extra
         # json nested element
         payara.exception ${if record['_LogMessage'].to_s.include?('_StackTrace');record["_LogMessage"]["_Exception"];  else 'null'; end}
         payara.stacktrace ${if record['_LogMessage'].to_s.include?('_Exception');record["_LogMessage"]["_StackTrace"];  else 'null'; end}

    </record>
    <record>
     # We will remove the nested element after extracting the stack trace and the log message

     removeStackTrace ${if record['_LogMessage'].to_s.include?('_StackTrace');record['_LogMessage'].delete('_StackTrace'); end}
     removeException  ${if record['_LogMessage'].to_s.include?('_Exception'); record['_LogMessage'].delete('_Exception'); end}
      # After removing it, an empty nested element will remain; we need to remove that as well
     payara.message ${if record.has_key?('_Timestamp'); record['_LogMessage'].to_s.gsub(/{}/,'null');  else record["logger"]; end}
     # Once remapping has been performed, we can remove the original fields from the stream, and only forward the processed data
    </record>
        remove_keys _Version,_Level,_LoggerName,_LevelValue,_LogMessage,_ThreadID,_ThreadName,_TimeMillis,_MessageID,_Timestamp,log,dummy,removeStackTrace,removeException,logger
    </filter>

Testing using docker containers

You can build your own docker images, or just use the ones i have used and uploaded in my git repository. You will need two containers:

  1. TDAgent with elasticsearch, concatenate filter and record transformer: https://github.com/githubskipper/tdagent
  2. Synthetic logger: https://github.com/githubskipper/synthetic_logger

When running the scenario above, and checking the tdagent logs, you can now see the formatted logs:

2018-08-07 20:30:51.998535210 +0000 loggf.1a49ca70104e: {"container_id":"1a49ca70104ec0563d51a388f7258af0cee13258a7a15e4962ae09c559aa2e07","container_name":"/logger1","source":"stdout","payara.level":"INFO","payara.logger":"javax.enterprise.system.core.security","payara.level_value":"800","payara.messageid":"1:15:NCLS-SECURITY-01115","payara.thread_id":"19","payara.thread_name":"RunLevelControllerThread-1527068660997","payara.time_millis":"1527068663907","@timestamp":"2018-07-18T15:44:23.907+0000","payara.exception":"null","payara.stacktrace":"null","payara.message":"1:15: - Line:single - Format:json - StackTrace:no - Source: Payara - Message: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created."}
2018-08-07 20:30:51.999495830 +0000 loggf.1a49ca70104e: {"container_id":"1a49ca70104ec0563d51a388f7258af0cee13258a7a15e4962ae09c559aa2e07","container_name":"/logger1","source":"stdout","payara.level":"INFO","payara.logger":"javax.enterprise.system.core.security","payara.level_value":"800","payara.messageid":"2:15:NCLS-SECURITY-01115","payara.thread_id":"19","payara.thread_name":"RunLevelControllerThread-1527068660997","payara.time_millis":"1527068663907","@timestamp":"2018-07-18T15:44:23.907+0000","payara.exception":"null","payara.stacktrace":"null","payara.message":"2:15: - Line:single - Format:json - StackTrace:no - Source: Payara - Message: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully destroyed."}
2018-08-07 20:30:51.999518779 +0000 loggf.1a49ca70104e: {"container_id":"1a49ca70104ec0563d51a388f7258af0cee13258a7a15e4962ae09c559aa2e07","container_name":"/logger1","source":"stdout","payara.level":"INFO","payara.logger":"javax.enterprise.system.core.security","payara.level_value":"800","payara.messageid":"3:15:NCLS-SECURITY-01115","payara.thread_id":"19","payara.thread_name":"RunLevelControllerThread-1527068660997","payara.time_millis":"1527068663907","@timestamp":"2018-07-18T15:44:23.907+0000","payara.exception":"3:15: - Format:json - Line:single - StackTrace:yes - Source: Payara - Message: Exception thrown from bean","payara.stacktrace":"javax.ejb.EJBTransactionRolledbackException","payara.message":"null"}
2018-08-07 20:30:52.000752420 +0000 loggf.1a49ca70104e: {"container_id":"1a49ca70104ec0563d51a388f7258af0cee13258a7a15e4962ae09c559aa2e07","container_name":"/logger1","source":"stdout","payara.level":"INFO","payara.logger":"javax.enterprise.system.core.security","payara.level_value":"800","payara.messageid":"4:15:NCLS-SECURITY-01115","payara.thread_id":"19","payara.thread_name":"RunLevelControllerThread-1527068660997","payara.time_millis":"1527068663907","@timestamp":"2018-07-18T15:44:23.907+0000","payara.exception":"4:15: - Format:json - Line:multi - StackTrace:yes - Source: Payara - Message: Exception thrown from bean","payara.stacktrace":"javax.ejb.EJBTransactionRolledbackException\\n\\tat com.sun.ejb... stack test 0","payara.message":"null"}
2018-08-07 20:30:52.009982837 +0000 loggf.1a49ca70104e: {"container_id":"1a49ca70104ec0563d51a388f7258af0cee13258a7a15e4962ae09c559aa2e07","container_name":"/logger1","source":"stdout","payara.level":"INFO","payara.logger":"javax.enterprise.system.core.security","payara.level_value":"800","payara.messageid":"5:15:NCLS-SECURITY-01115","payara.thread_id":"19","payara.thread_name":"RunLevelControllerThread-1527068660997","payara.time_millis":"1527068663907","@timestamp":"2018-07-18T15:44:23.907+0000","payara.exception":"5:15: - Format:json - Line:multi - StackTrace:yes - Source: Payara - Message:: Exception thrown from bean","payara.stacktrace":"javax.ejb.EJBTransactionRolledbackException\\n\\tat com.sun.ejb.containers.BaseContainer.mapLocal3xException(BaseContainer.java:2359)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.sun.proxy.$Proxy916.clearCache(Unknown Source)\\n\\tat com.uk.apsys.conf.configuration.__EJB31_Generated__DomainConfigurationRepository__Intf____Bean__.clearCache(Unknown Source)\\n\\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2141)\\n\\tat com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2060)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)\\n\\tat EJBLocalObjectInvocationHandlerDelega\nte :90EJBLocalObjectInvocationHandlerDelegate :80EJBLocalObjectInvocationHtest1erDs12","payara.message":"null"}

You can now see how that large message has been concatenated into one single field, and how all those special fields prefixed with _ have been rewritten to payara.something.

Docker caching pitfall

One more thing to watch for. When testing with the out.log synthetic data, if you’re going to modify this, you should append it into another file and use that other file. Docker has
an interesting caching mechanism, which will cache the contents of the initial file. So no “change file and retry”, but “change file, copy into another, retry with the other file”

This is it.Cool, ain’t it?

Thanks for taking a look, let me know if i can help further
Best,
Alex
[/av_textblock]

Print Friendly, PDF & Email