Укрощение неожиданных производственных проблем

#!/usr/bin/perl -w
use strict;
use warnings;
if(!defined $ARGV[0]){
    print "No FilePath Specified!\n";
}

my $LOGFILE = $ARGV[0];

print "Time\tThread\tState\tLock Name\tLock Name Class\tCall Stack\n";

open(my $fh, "<$LOGFILE") or die("Could not open log file.");
foreach my $line (<$fh>) {

	if ($line =~ /^thread_dump_(\d\d\d\d)(\d\d)(\d\d)_(\d\d)(\d\d)\d\d.html:(.*)$/) {
		my $year = $1;
		my $month = $2;
		my $date = $1;
		my $hour = $4;
		my $min = $5;
		my $td = $6;

		my @matches = ($td =~ m/(\>b\<Thread: [^:]+ : priority:\d+, demon:[a-z]+, threadId:\d+, threadState:[A-Z_]+, lockName:[a-zA-Z0-9\$\.\@]+\>br\<\>blockquote\<[ a-zA-Z_\$\.\(\):0-9\>\<]+\>\/blockquote\<)/g);
		
		
		foreach my $x (@matches) {
			if ($x =~ /^\>b\<Thread: ([^:]+)\>\/b\< : priority:\d+, demon:[a-z]+, threadId:\d+, threadState:([A-Z_]+), lockName:([a-zA-Z0-9\$\.]+)(\@?[0-9a-f]*)\>br\<\>blockquote\<([ a-zA-Z_\$\.\(\):0-9\>\<]+)\>\/blockquote\<$/) {
				print "$month/$date/$year $hour:$min\t$1\t$2\t$3$4\t$3\t$5\n";
			} else {
				print STDERR "No match : $x\n";
			}
		}
	} else {
		print STDERR "No match in line : $line\n";
	}   
}
close($fh);

Это будет производить контент, как показано ниже

Агрегация будет выглядеть так

The pivot table like the one above, allows to see where the threads are waiting and/or blocked. The stack traces are readily available to drill down. This allowed to identify memcached client default configuration as a bottle neck in my practice along as pointed to architectural deficiency created by the use of the memcached values through application life cycle. It manifested itself though as BLOCKED threads on com.mchange.v2.resourcepool.BasicResourcePool — JDBC connections not released in time for others to re-use.

Additional clues that can be collected are exceptions from log files (artifact 4) or errors from artifact 5. First one can be collected by the following command with the manual aggregation:

grep -r Exception server.* > exceptions.txt

Second one will have all the aggregation already done. Of particular interest are timeout related messages in the logs:

grep -ir timeout server.* > timeout.txt

Everything else is application specific and will reflect logging messages of a particular system. It can be processed in the same way to extract metrics and correlate them over time.

With all the information at hand, start from the timestamp of the incident and go back in time through the artifacts to find the tipping point where it all started collapsing. You may find multiple issues nested one on top of another. Create an action plan for each issue and work to implement the fixes one by one. It can take multiple iterations in production to iron out all of those, so be patient and keep hunting.