Streaming Runner Logs to something else

[dicko2]

Our use case is as follows:

We have a large number on on-premise build agents and several internal "devops" services we offer to our teams, including things such as Device Testing Farms etc. We want to monitor behavior and search for common issues occur across the company to better serve our internal customers

One way we've been doing this is analyze build logs for common errors, we've been doing this post build so far and have had some success in detecting issues.

A common example would be something like A single node in a Selenium Grid test farm becomes intermittently flakey we see slightly increase number of test failures from it reported in logs with selenium specific errors or time outs.

Currently we batch the logs out and run analysis on them up to an hour after the error occurs, we want to look at moving to a async stream based system where we stream the logs out of the runners

We've been looking at the most unintrusive approach to the runner code so we don't have to maintain a fork and looking at the internals of the runner it appears to use tracing to shift the logs, so we think we can add a diagnostic listener using the app settings file registration and suck the logs out without any change to the runner code itself.

What do you folks think of this approach?

[ssboisen]

@dicko2 what we do is ship the paging logs from the _diag/pages folder using a regular log shipper (we use vector). This allow us to get streaming logs, however one problem is that we see duplicate logs because for some reason the github action runner writes the same logs to multiple page log-files. I havn't debugged the action runner in order to find out why this happens though.