A couple of days ago while working on an automation project I found myself logging progress to the Linux logging sub-system. This worked well enough as it gave me a central place to dump messages without having to install yet _another_ system dependency. This also provides a place that for everyone to look when debugging an issue. 'What do the logs say' is one of the first things I say to myself when something does not work as expected. But what about when the logs stop saying...anything?

So there I was Trucking along in the zone, everything was great. Knocking out task after task. Then my trust log reading command stopped working.

'OK' I think. No problem, so I started the troubleshooting process. Maybe the application is not installing? Nope, it is. Maybe credentials expired? Nope, they are ok. CLI authentication works. Ok, so I roll back a couple commits and boom the messages are back. Time to break this down into the atomic parts. So for the next hour I incremented commits one at a time to see where the issue first appears. (This would be a nice time to use Git Bisect, but I did not :(. )

One commit after another, checking the process. Each time using the following to check the system logs.

From two hours back all the way back to the current commit; everything worked as expected. At this point I was reaching the 2 hour 'I am stuck' limit so I reached out to a colleague and asked for a rubber duck session. They agreed and came to sit with me.

I explain What the end goal is, how I was formulating the solution, the start of the issue, what I have tried, and showed the current state. Vim viewing log works, but tail ... | grep...'ing the logs...AARRGRGG! As soon as I said the words I knew what the problem was.

Tail only prints the last 10 lines to standard out. So grep only had 10 lines to work with. As my process length increased the logging messages got pushed passed the 10 line mark. To confirm this I tried:

Sure enough, the messages were there. :| Just one of those thing I guess.

How about you? Every get so deep into something you missed the obvious?