Master Journalctl: View, Filter, and Manage Systemd Logs

Introduction

Systemd, journald, and journalctl are essential tools for managing and troubleshooting system logs on Linux systems. These components offer a centralized logging solution, collecting kernel and user process logs, and allowing for advanced filtering and real-time monitoring. Whether you’re trying to identify a system issue or configure persistent logging, mastering journalctl is key to optimizing system performance. In this article, we’ll dive into how to view, filter, and manage your system logs effectively, ensuring smooth operations and efficient troubleshooting.

How the systemd Journal Works and Why It Matters

Imagine you’re the captain of a big spaceship (okay, it’s actually a server, but hang with me). You’ve got a lot of different systems working together, each with its own specific tasks. Each of these systems, like engines or control panels, creates logs—little reports on how everything’s going. The issue? These logs are all over the place, coming from all sorts of different places: the kernel, system services, user applications. It’s like trying to read a bunch of handwritten notes scattered around the spaceship—you’ll miss things, and it’s tough to understand it all.

This is where systemd comes in. Think of systemd like the central nervous system of the spaceship, handling everything from the boot process to managing all the systems. It makes sense for it to also gather all the logs into one place. That’s how the systemd journal came to be. It’s a logging system that takes all those separate logs and gives you one clear, organized view.

At the core of this system is the journald daemon. It’s like the ultimate log collector, pulling in information from every part of your system. Whether it’s the kernel (the brain of your spaceship/server), system services (the engines), or even the applications (the astronauts), journald collects all the logs. These logs are saved in a binary format, which is really efficient because it lets you easily work with and analyze the data however you want. So, instead of sifting through endless text logs, you get a simpler way to view your data.

And here’s the cool part: because the logs are saved in binary, you don’t have to stress about manually converting them when you need them in a different format. Let’s say you usually view logs in the standard syslog format for day-to-day monitoring, but then you need to track trends or service interruptions. You can have the system automatically change the logs into a JSON format, perfect for feeding into graphing tools or other data analysis platforms. No extra parsing needed! It’s like your spaceship automatically adjusting its systems to show you exactly what you need right when you need it.

What makes systemd’s journald even better is its flexibility. You can use it alongside your current syslog setup if you want, or you can replace your syslog with it entirely, depending on what you need. It’s like having a versatile tool in your toolbox. But the best part? You can also use it with other logging systems. Let’s say you’ve got a centralized syslog server collecting logs from multiple systems—no problem, you can still use the systemd journal to gather and organize logs from different services on a single server.

When you combine the power of journald with other technologies, you get a logging system that not only makes monitoring your system easier but also helps with troubleshooting and performance analysis. It’s like having a supercharged command center where everything is right at your fingertips, making your job as an administrator a lot easier.

Systemd Journal Management

How to Set the Correct System Time with timedatectl

Imagine this: you’re managing a network of servers all around the world, each creating logs every second. You’re responsible for making sure everything works well, but there’s one problem—different time zones! The logs are scattered across time zones, and trying to make sense of them feels like trying to connect dots on a constantly changing map. This is where setting the correct system time comes in, and systemd’s tool, timedatectl , is your go-to helper.

Here’s the deal: systemd lets you view logs in any time zone you want. By default, it shows the log timestamps in your local time, but you can switch to Coordinated Universal Time (UTC) if needed. Why? Well, if you’re managing systems in different time zones, this simple change can save you time and make it easier to check logs from anywhere in the world.

Before you dive into the journal features, there’s something you should do first—make sure your system time and time zone are set correctly. If you skip this, you could face some odd issues with how your logs show up. You don’t want your log data getting messed up, right? timedatectl is the tool you’ll need here. It’s part of systemd and does more than just show the time—it helps you manage it.

To begin, you can check what time zones are available on your system. Just run this command:

$ timedatectl list-timezones

This will show you a list of time zones your system knows about. Once you find the one that matches your server’s location, you can set it using the set-timezone option. For example, if your server is in New York, you’d set the time zone like this:

$ sudo timedatectl set-timezone America/New_York

And just like that, your server is set to New York time. But how do you check if it worked? You check, of course! Run this command:

$ timedatectl status

You’ll see the current system time along with the time zone, Universal Time (UTC), and whether your system is properly synced. Here’s what it might look like:

Local time: Fri 2021-07-09 14:44:30 EDT
Universal time: Fri 2021-07-09 18:44:30 UTC
RTC time: Fri 2021-07-09 18:44:31
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no

The first line should show the correct local time, confirming that everything is lined up with your time zone. By making sure the time is set right, you’re preventing problems with your logs—because, let’s be honest, nobody likes dealing with confusing timestamps. So, that’s it—once your time zone is set and confirmed, you’re all set! Whether you’re troubleshooting with journalctl or just keeping an eye on the logs, this small but important step will make everything much easier.

Setting Time Zones in Linux

How to View Logs with journalctl

Picture this: you’re managing a server, and you’re trying to solve a puzzle. Every log entry is like a clue that can help you figure out what went wrong—or right. And just like any detective story, you need the right tools to make sense of the clues scattered throughout your system. That’s where journalctl comes in, a tool that lets you see the logs collected by the journald daemon, giving you access to all the system events that have been recorded.

When you run the journalctl command, it’s like opening a time machine to your system’s past. It pulls up every single log entry available, whether it’s from the last boot or from weeks ago. The results are neatly displayed in a pager (think of it as an endless scroll, where you can go through the entries at your own pace). And here’s the interesting part—by default, the oldest log entries show up first, so you’re essentially rewinding time to see what happened.

Here’s how you’d go about using it:

journalctl

The output can be a bit overwhelming at first, with everything from system startup to kernel messages filling the screen. But don’t worry, you’ll get the hang of it. Here’s an example of how the logs might look:

Logs begin at Tue 2015-02-03 21:48:52 UTC, end at Tue 2015-02-03 22:29:38 UTC.
— Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (max allowed 49.
Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (max allowed 49.
Feb 03 21:48:52 localhost.localdomain systemd-journald[139]: Received SIGTERM from PID 1 (systemd).
Feb 03 21:48:52 localhost.localdomain kernel: audit: type=1404 audit(1423000132.274:2): enforcing=1 old_en
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.
Feb 03 21:48:52 localhost.localdomain kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 8 users, 102 roles, 4976 types, 294 bools, 1 sens,
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 83 classes, 104131 rules

As you can see, the log entries are full of useful info, but they can pile up quickly. If your system’s been running for a while, these logs could go on for thousands of lines, showing you just how much data is stored in the journal. If you’ve worked with traditional syslog systems before, the format of these logs will probably feel familiar. But here’s the twist: systemd doesn’t just pull from the usual places like syslog—it collects logs from all kinds of sources. From the kernel to early boot processes, from initrd to stdout and stderr from applications, everything gets collected and put into the journal. Pretty cool, right?

Now, you might notice that the timestamps for each log entry are in local time by default. This is fine if you’re managing a system in one time zone, but what if you’re working across different regions or just want a more standardized approach? No worries! You can switch those timestamps to UTC (Coordinated Universal Time) by simply adding the --utc flag when running journalctl :

journalctl –utc

This will give you all the log entries, but with their timestamps in UTC, which is perfect for tracking events across different time zones or when you need a precise time reference. Whether you’re looking for past events or just need a more accurate time, UTC is the way to go.

So, there you have it! With journalctl , viewing logs is like having a front-row seat to your system’s history, with the ability to rewind and fast-forward through time as you please. Whether you’re managing a server, troubleshooting an issue, or just curious about what’s going on under the hood, systemd and journald give you all the tools you need to stay in the loop.

Systemd Journal Logs Management

How to Filter systemd Logs by Time with journalctl

Imagine you’re in charge of managing a complicated system. Logs are constantly being created—some are important right now, while others can wait or aren’t needed at all. With so much information coming at you, it can feel overwhelming. It’s like trying to find a specific book in a massive library! This is where the journalctl command steps in—helping you filter through all the information easily and accurately.

Let’s start with something simple: the -b flag. It’s one of the most common and easiest ways to filter logs. If you want to focus on the current system session—maybe there’s an issue happening right now, or you just want to see what’s going on live—this option is perfect. Just run:

journalctl -b

This command will show you all the logs from since your last reboot. It’s like getting a snapshot of everything that’s happened since your system last restarted. If your system has rebooted multiple times, journalctl will even mark each reboot with a line like “– Reboot –,” making it easy to separate logs by boot sessions. This way, you can trace issues back to specific reboots, making troubleshooting much smoother.

But what if you need to see logs from earlier boots? Maybe you missed something important the last time the system was running, and now you want to check the history. journalctl lets you easily access logs from past sessions. The system keeps logs from multiple boots, and you can view them by using the --list-boots option. Just type this command:

journalctl –list-boots

The output will give you a list of past boots, each with an identifier that you can use to access their logs. It will look something like this:

2 caf0524a1d394ce0bdbcff75b94444fe Tue 2015-02-03 21:48:52 UTC—Tue 2015-02-03 22:17:00 UTC
-1 13883d180dc0420db0abcb5fa26d6198 Tue 2015-02-03 22:17:03 UTC—Tue 2015-02-03 22:19:08 UTC
0 bed718b17a73415fade0e4e7f4bea609 Tue 2015-02-03 22:19:12 UTC—Tue 2015-02-03 23:01:01 UTC

Now, with this list, you can focus on specific past sessions. For example, to see logs from the previous boot, you just need to type:

journalctl -b -1

Or, if you prefer, use the boot ID to dig deeper into a particular session:

journalctl -b caf0524a1d394ce0bdbcff75b94444fe

So far, so good, right? But sometimes you’re dealing with logs that don’t neatly fit into these boot session windows. Maybe you need to check logs from a specific time period. You know, like the time between when that troublesome service went down and when it came back up. Luckily, journalctl has two powerful tools for this: the --since and --until options. These let you set custom time windows to filter through logs.

For example, let’s say you’re investigating something that happened on January 10th at 5:15 PM. You’d run this:

journalctl –since “2015-01-10 17:15:00”

The great thing is that you don’t always have to be exact with the time. If you just want to see logs for January 10th without worrying about the exact hour, you can do this:

journalctl –since “2015-01-10”

And if you want to narrow it down to a specific time range, you can pair --since and --until . For example, let’s say you need to see everything from January 10th until 3:00 AM on January 11th:

journalctl –since “2015-01-10” –until “2015-01-11 03:00”

What if you’re feeling a bit more laid-back and don’t want to worry about exact times? journalctl has you covered with more relaxed time options. You can use terms like “yesterday,” “today,” “tomorrow,” or even “now.” Here’s an example:

journalctl –since yesterday

Or, if you’re tracking a service issue that started at 9:00 AM and lasted until an hour ago, you can run:

journalctl –since 09:00 –until “1 hour ago”

By using these time-based filters, you can zoom in on the exact events you need, without being overwhelmed by irrelevant data. This feature is super useful when troubleshooting, especially on systems that have been running for a long time. You can pick and choose your time frames, which really helps when you’re trying to figure out what happened.

So, there you have it! With journalctl , you’re not stuck wading through endless logs. You can filter out the noise, focus on what matters, and make troubleshooting a lot easier. Whether it’s filtering by boot session, narrowing it down to a specific time range, or jumping between past reboots, systemd’s journald makes it all more manageable.

Systemd Journal Logs Management

How to Access Logs from Previous Boots Using journalctl

So, you’ve been using systemd to manage your logs with journald, and you’re handling your current system logs just fine. But what if something happened during a previous boot—maybe you missed it or it’s causing problems now? How do you find the logs from those past boots? That’s exactly what journalctl is for.

Here’s the deal: You know that journalctl shows the logs from your current boot, but sometimes you need to dig a little deeper. Maybe you’re tracking down a bug from the last reboot, or you’re troubleshooting a system crash from a few days ago. Whatever the reason, journalctl makes it easy to grab logs from earlier boots without any hassle.

By default, many Linux distributions save logs from past boots automatically. But if yours isn’t doing this, don’t worry—you can set it up. You have a couple of options: you can either create a folder to store the logs, or you can change the system’s settings in the journald configuration file to keep logs after a reboot. Let’s walk through this step by step.

If you want to start by manually creating a directory for logs to stick around, here’s the command:

sudo mkdir -p /var/log/journal

This will create the folder where your logs will be saved. But if you want a more permanent solution that’s simpler, you can edit the journald configuration file. It’s easy: just open the file in your text editor—like nano—and find the Storage setting.

sudo nano /etc/systemd/journald.conf

Inside the file, look for the line that says Storage=volatile and change it to Storage=persistent . This will make sure the logs are saved between reboots.

[Journal] Storage=persistent

Once that’s done, restart the systemd journal service to make the change stick:

sudo systemctl restart systemd-journald

Now, your system will keep logs, and next time you need to check logs from an earlier boot, journalctl has your back.

Want to see all the logs from past boots? Just run:

journalctl –list-boots

This will show a list of boot sessions with their start and end times, like this:

2 caf0524a1d394ce0bdbcff75b94444fe Tue 2015-02-03 21:48:52 UTC—Tue 2015-02-03 22:17:00 UTC
-1 13883d180dc0420db0abcb5fa26d6198 Tue 2015-02-03 22:17:03 UTC—Tue 2015-02-03 22:19:08 UTC
0 bed718b17a73415fade0e4e7f4bea609 Tue 2015-02-03 22:19:12 UTC—Tue 2015-02-03 23:01:01 UTC

Each line represents a different boot session, so you can use this info to find exactly which session’s logs you need. The first column is a reference you can use, the second shows the unique boot ID, and the third and fourth columns tell you when that session started and ended.

So, if you want to check the logs from the last boot, just use:

journalctl -b -1

Alternatively, if you prefer using the boot ID, you can access the logs with:

journalctl -b caf0524a1d394ce0bdbcff75b94444fe

Both methods let you focus on a specific boot session, making it easier to troubleshoot issues at a particular time.

With journalctl , accessing logs from previous boots is a breeze, whether you’re fixing an issue, looking at past events, or just curious about what happened. Now, when something goes wrong, you’ll be able to trace it back through the logs easily.

Systemd Journal Logging

Filter journalctl Logs by Custom Date and Time Ranges

Imagine this: your server’s running smoothly, the logs are stacking up, and everything’s going great—until, out of nowhere, something goes wrong. Maybe the issue happened before the current boot session, or you need to check logs from a different time. That’s when journalctl steps in, offering some really helpful ways to filter logs by custom date and time ranges.

While using boot sessions to view logs is handy, it’s not always enough. Servers with long uptimes often cover multiple boot sessions, so filtering logs by a specific time window becomes really useful. With the --since and --until options in journalctl , you can narrow things down with pretty good accuracy. Let’s break it down.

The --since option is where you start, allowing you to choose when you want to begin viewing logs. The --until option helps you set an endpoint, giving you a time range that fits your needs. And these options aren’t limited to specific dates and times—they’re flexible enough to work however you need them to.

Let’s say you want to start viewing logs from a specific time on a certain date. It’s simple. Just use a command like this:

journalctl –since “2015-01-10 17:15:00”

With that, you’ll get all the logs from January 10th, 2015, at 5:15 PM. But let’s say you’re in a hurry and don’t need to worry about the exact seconds. No problem— journalctl will assume midnight (00:00:00) if you just provide a date, like this:

journalctl –since “2015-01-10” –until “2015-01-11 03:00”

This command will show logs from 12:00 AM on January 10th to 3:00 AM on January 11th, 2015—a great way to get a quick look at that time period. But that’s not all. One of the best things about journalctl is how it works with relative time values. Instead of worrying about exact times, you can just use simple terms like “yesterday,” “today,” or even “an hour ago.”

So, if you want to see what happened “yesterday,” you can quickly run:

journalctl –since yesterday

And if you’re tracking an issue that started at 9:00 AM and lasted until an hour ago? It’s easy:

journalctl –since 09:00 –until “1 hour ago”

This flexibility lets you search logs in a way that feels natural. You can use terms like “today,” “tomorrow,” or “now,” and even apply offsets like “-1 hour ago” to fine-tune your search.

With these powerful filtering options, journalctl becomes more than just a log viewer. It’s a flexible tool for monitoring and troubleshooting, making it easier to zoom in on exactly what you need, when you need it.

So next time you’re tracking down an issue that didn’t happen during the current session, just remember that systemd’s journald and journalctl are here to help, letting you filter logs by the time periods that matter most.

Systemd Journal Logging

Filter journalctl Logs by Service, PID, or User

Imagine you’re troubleshooting an issue on your system. You’ve been staring at logs, trying to figure out what went wrong, but there’s just so much info—where do you even start? Well, here’s the good news: journalctl has some handy tools to help you sort through all that data and focus on what really matters. Whether you need logs from specific services, certain processes, or particular users, you’ve got a full set of tools to make it easier.

Let’s begin with the basics. You’ve probably heard of systemd’s service units. These are what systemd uses to manage things like web servers, databases, or background jobs. If you want to check the logs for one of these services, you can filter them by service unit. It’s like saying, “I only want to see what’s going on with Nginx, nothing else.”

How do you do that? It’s easy. Just use the -u flag, followed by the service name. For example, to see everything related to the Nginx service, you’d run this:

journalctl -u nginx.service

This will show you all the logs for Nginx. But if you’re like me, you probably want to narrow it down to today’s logs. Here’s how:

journalctl -u nginx.service –since today

Now, you’re only seeing today’s logs, which makes things much clearer. But here’s the fun part: What if Nginx is working with PHP-FPM (FastCGI Process Manager) to serve web pages? Well, journalctl lets you view multiple service logs at once. This is super useful when you’re troubleshooting something like a miscommunication between Nginx and PHP-FPM. You just combine the two like this:

journalctl -u nginx.service -u php-fpm.service –since today

Now, you can see both services side by side in order, making it easy to track how they’re working together. It’s like watching a movie where the two main characters are interacting—if one messes up, you’ll know right away.

But what if you need something even more specific? Maybe you’re not looking at the whole service, just a certain process or user. No problem. You can filter logs by the process ID (PID), user ID (UID), or group ID (GID). This is where systemd and journald really shine.

Let’s say you know the PID of the process you’re looking for—maybe it’s a PHP script running at a specific time. You can filter the logs to show only entries for that PID like this:

journalctl _PID=8088

But what if you want to see everything tied to a specific user, like the www-data user running your web server? You can get the user ID by running this command:

id -u www-data

If the UID that pops up is 33, you can filter the logs by that UID using this command:

journalctl _UID=33 –since today

And if that’s still not enough, you can also filter logs by group ID (GID). To see all the group IDs in the journal, just run:

journalctl -F _GID

This will show you all the groups in your journal, and you can pick one to narrow things down further.

There’s more! If you’re dealing with a specific executable, like bash, and don’t want to bother with the service unit, you can filter by the executable path:

journalctl /usr/bin/bash

This will show you all logs related to that executable. Just keep in mind, this might not include logs from any child processes, which is why filtering by the service unit is usually a better option.

In the end, whether you’re filtering by service unit, PID, UID, GID, or even by executable path, these options help you focus on the data that matters. Instead of getting lost in a sea of logs, you can zoom in on exactly what you need to troubleshoot problems or monitor system activity. With these filtering tools, systemd and journald—through the journalctl command—become your go-to helpers for keeping things running smoothly.

Systemd Journal Logging

How to View Kernel Logs Using journalctl -k

Alright, imagine this: you’re troubleshooting your system, it’s acting a bit off, and you need to get the kernel messages to figure out what’s going wrong. But here’s the good news—you don’t have to sift through tons of log files by hand. Thanks to systemd and the journalctl command, you can grab those messages directly from the system journal. Sounds pretty simple, right?

The journalctl command is already super handy, but when you’re just looking for kernel messages—the detailed stuff about how your system’s hardware is working with the software—you’ve got the -k flag. That’s the one that lets you focus only on kernel messages, cutting out all the other clutter. It’s like telling journalctl , “Just show me the kernel info, nothing more.”

Here’s how it works:

journalctl -k

That’s it! This command will immediately show you all the kernel messages from your current boot session. What do you see when you run this? It’s a lot of info, like how your devices were set up, any interactions with hardware, and any kernel errors or warnings that could point to deeper problems. Think of it as the system’s “behind-the-scenes” update on what’s going on at the hardware level.

But what if you need to check past kernel logs? Maybe you want to find out what caused that system crash last week or figure out why there was a hardware failure a few reboots ago. Don’t worry, journalctl has you covered. By using the -b flag, you can easily access kernel logs from past boot sessions. It’s like flipping through old chapters in your system’s story.

For example, if you want to look at kernel logs from five boots ago (maybe you’re tracking down that crash that happened during an update), you can run:

journalctl -k -b -5

This command will show you the kernel logs from the session that happened five reboots ago. But you don’t even need to remember the exact boot number—if you just want to check the logs from the previous boot, use the -1 flag. It’s as easy as saying, “Hey, show me the logs from just before this one.”

journalctl -k -b -1

What’s great about this setup is that it doesn’t just focus on the current boot. You can scroll back in time and check out previous kernel logs to troubleshoot system crashes, hardware issues, or those weird errors that seem to pop up out of nowhere. With these commands, you get a full view of how your system’s been running over several reboots. Whether you’re figuring out a crash from earlier today or looking into past configuration issues, journalctl with the -k flag makes checking kernel logs simple and clear. You’re not just fixing today’s problems; you’re also understanding the past, giving you a complete picture of how your system is working and why.

Systemd Journal Logging

Filter Logs by Severity Level with journalctl -p

Imagine you’re troubleshooting a system issue, the clock is ticking, and the logs are piling up. The challenge? You have to sift through thousands of log entries to find the important ones that need your attention. It’s like trying to find a needle in a haystack, right? You know there’s useful information in those logs, but there’s so much noise—low-priority messages that don’t really help with your current problem. Here’s where the journalctl command, specifically the -p flag, comes in to save the day.

The great thing about journalctl is that it allows you to filter logs based on their severity level, so you can focus on the important stuff and ignore the rest. For example, if you’re only interested in errors or system-critical events, you don’t have to wade through endless informational messages. Instead, you can quickly find the log entries that matter the most. This is especially useful when you’re on a tight deadline and need to spot issues that could affect the system’s functionality.

Here’s how you can do it. If you’re looking for logs that show errors or critical issues, simply run:

journalctl -p err -b

What does this do? It filters the logs to show only those with a severity level of error or higher, including critical, alert, or even emergency messages. These are the logs you really need to focus on, as they indicate problems that could potentially bring down the system or cause major disruptions. It’s like having a spotlight on the troublemakers in your logs—everything else fades into the background.

Now, let’s break down the severity levels you’ll see when using journalctl . These levels follow the standard syslog priority model, making it easy for system administrators to categorize messages. Think of it like a ranking system for log messages, from the most urgent ( emergency ) to the least important ( debugging info ). Here’s a quick rundown:

• 0 (emerg): Emergency – System is unusable (the big red flag).
• 1 (alert): Alert – Immediate action needed (serious stuff).
• 2 (crit): Critical – Critical conditions (this needs fixing fast).
• 3 (err): Error – Error conditions (these should be addressed).
• 4 (warning): Warning – Warning conditions (not urgent, but worth noting).
• 5 (notice): Notice – Normal but significant conditions (important to know, but not critical).
• 6 (info): Informational – Informational messages (just for the record).
• 7 (debug): Debug – Debug-level messages (the lowest priority, usually for troubleshooting).

Now, here’s the cool part. You don’t always have to type out the full word. You can use the numeric equivalent too. For example, if you want to filter logs and show only warnings or higher, you can use either:

journalctl -p warning

Or, if you prefer numbers:

journalctl -p 4

This is great because it lets you control exactly which messages you see, helping you focus on the most important ones. By choosing a severity level, you automatically filter out everything below that level. So, if you just need to focus on those urgent issues, you won’t be distracted by the less important ones.

This feature is incredibly useful when you’re diagnosing system issues. By filtering out lower-priority logs, you’re not bogged down with irrelevant data. Instead, you get a clear view of the most severe problems, letting you resolve them quickly and efficiently.

So next time you’re deep in logs and need to cut through the noise, remember journalctl -p —your trusty tool for filtering logs by severity level and making sure you’re always looking at the logs that matter most.

Systemd Journal Logging

Customize journalctl Log Output Display

You know how it feels when you’re buried under a mountain of logs? Every scroll and click just adds more data, but most of it is just noise. It’s like trying to find one important sentence in a 500-page book. But here’s the good news— journalctl is a lifesaver when you need to customize your log output. It helps make things more manageable, readable, and tailored to what you need.

How to Control Output Length and Formatting in journalctl

By default, when you run journalctl , it shows log entries in a pager (usually less), which is great for scrolling through, but when the log entries get long, it’s easy to get lost trying to read everything. You might even find yourself scrolling sideways just to catch the last part of the information. Let’s make it easier, shall we?

If you want to shorten those long entries and make them more compact, you can use the --no-full option. This will add an ellipsis (…) at the end of each truncated log entry, giving you just the basics. Here’s how it looks:

journalctl –no-full

So, instead of this long log entry:

Feb 04 20:54:13 journalme sshd[937]: Failed password for root from 83.234.207.60…
Feb 04 20:54:13 journalme sshd[937]: Connection closed by 83.234.207.60 [preauth]…

You’ll get something much cleaner, which makes it easier to stay focused. But sometimes, you don’t want to miss any details. In that case, you can use the -a flag to show everything, including any unprintable characters:

journalctl -a

This will show you the full log, no matter how messy it gets.

How to Disable the Pager in journalctl Output

Now, let’s talk about the pager. By default, journalctl uses it to let you scroll through long log files. It’s useful for manual checks, but if you’re using the logs for something more automated—like piping them into other tools or running them through a script—you might want to skip the pager.

In that case, just run:

journalctl –no-pager

This removes the pager and sends the logs straight to your terminal or a file. This is super helpful if you’re redirecting logs or saving them for later analysis. No distractions, just the raw data.

Different Output Formats in journalctl

Here’s where journalctl really stands out. If you’re working with logs in a program or need to integrate them with other tools, the format of your logs can be key. The default format is easy to read, but sometimes it’s just not practical for automated systems or tools. Thankfully, journalctl has options for that, offering different output formats.

If you’re dealing with JSON data or need logs in a machine-friendly format, you can use the -o option to choose the format. Let’s say you’re working with logs from the nginx service and want them in JSON format to make it easier to process:

journalctl -b -u nginx -o json

Here’s what that might look like in your terminal:

{ “__CURSOR”: “s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635”, “__REALTIME_TIMESTAMP”: “1422990364739502”, “__MONOTONIC_TIMESTAMP”: “27200938”, “_BOOT_ID”: “81b58db8fd9046ab9f847ddb82a2fa2d”, “PRIORITY”: “6”, “_UID”: “0”, “_GID”: “0”, “_CAP_EFFECTIVE”: “3fffffffff”, “_MACHINE_ID”: “752737531a9d1a9c1e3cb52a4ab967ee”, “_HOSTNAME”: “desktop”, “SYSLOG_FACILITY”: “3”, “CODE_FILE”: “src/core/unit.c”, “CODE_LINE”: “1402”, “CODE_FUNCTION”: “unit_status_log_starting_stopping_reloading”, “SYSLOG_IDENTIFIER”: “systemd”, “MESSAGE_ID”: “7d4958e842da4a758f6c1cdc7b36dcc5”, “_TRANSPORT”: “journal”, “_PID”: “1”, “_COMM”: “systemd”, “_EXE”: “/usr/lib/systemd/systemd”, “_CMDLINE”: “/usr/lib/systemd/systemd”, “_SYSTEMD_CGROUP”: “/”, “UNIT”: “nginx.service”, “MESSAGE”: “Starting a high-performance web server and a reverse proxy server…”, “_SOURCE_REALTIME_TIMESTAMP”: “1422990364737973” }

If that looks too robotic, you can use the json-pretty option for a more user-friendly version:

journalctl -b -u nginx -o json-pretty

The output will still be in JSON, but it’ll be formatted in a way that’s easier to read and understand.

Other Available Output Formats

But wait, there’s more! journalctl offers lots of output formats, so you can choose the one that fits your needs. Here are some other options:

• cat: Displays only the message field of each log entry.
• export: A binary format perfect for transferring or backing up logs.
• json: Standard JSON format, one entry per line.
• json-pretty: A more human-friendly JSON format.
• json-sse: JSON formatted for server-sent event compatibility.
• short: Default syslog-style output, with just the essentials.
• short-iso: Default format with ISO 8601 wallclock timestamps.
• short-monotonic: Default format with monotonic timestamps.
• short-precise: Default format with microsecond precision.
• verbose: Displays every journal field, including those hidden internally.

Each of these formats has a specific purpose, so you can pick the one that works best for what you’re trying to do. Whether you’re dealing with logs for humans or feeding them into an automated system, journalctl has got you covered.

With all these options, you can fully control how your logs are shown, making it easier to extract, analyze, and process critical log data quickly. You can customize everything—from the length of your log entries to the exact format—so you can focus on what matters most without the clutter.

Systemd Journal Logging

Monitor Live systemd Logs with journalctl

Imagine you’re sitting at your desk, staring at the screen, waiting for an important system event to pop up. You need to catch that log entry the moment it happens. Well, here’s the thing: with journalctl in systemd, you’ve got everything you need to keep an eye on your system logs in real-time, just like you would with the tail command—except you won’t have to switch tools. It’s already built in and ready to go.

Show Recent Log Entries with journalctl -n

As an admin, you probably find yourself checking recent log entries more often than you’d like to admit. To make this task easier, journalctl has a handy option: the -n flag. This command gives you a quick look at the most recent log entries, so you don’t have to scroll through endless lines of log history. By default, if you don’t add any numbers, it’ll show you the last 10 logs:

journalctl -n

This command is perfect for those quick checks when you don’t need to dig into the past but just want to know what’s been happening recently. But let’s say you need more than just the last 10 entries? You can totally customize this. Want to see the last 20? Easy. Just type:

journalctl -n 20

Now, you’ve got a wider view of what’s happened recently, without having to go through the entire log history. You can adjust the number to fit your needs—whether that’s 50 logs or just 5. It’s all about making things simpler.

Follow Real-Time Logs with journalctl -f

Here’s where journalctl gets even better. You might need to watch your logs in real-time, especially if something’s acting up and you need to see new entries as they’re created. With the -f flag, it’s like having a live stream of your logs right in front of you:

journalctl -f

This command will keep showing new log entries as they happen. It’s perfect for tracking down live issues or watching processes as they go. You’ll see everything in real-time, so if there’s a problem, you’ll catch it as it happens. And when you’re done? Just hit CTRL+C to stop watching and go back to your usual terminal prompt.

These features in journalctl make it super easy to monitor both past and live log entries. You won’t have to run multiple commands or rely on other tools. Just use these simple flags, and you’ll have your logs under control, whether you need a quick peek at recent events or want to track every move your system makes. journalctl ’s got you covered for all your log monitoring needs.

Systemd Journal Logging

How to Manage and Clean Up systemd Journal Logs

Picture this: you’re cruising along, handling your system tasks, when suddenly you realize your logs are piling up like paperwork on your desk. Over time, those logs can start causing problems, taking up too much disk space and slowing down your system. But don’t worry, managing systemd logs doesn’t have to be a hassle. journalctl is here to help, giving you control over your log storage to keep your system running smoothly.

Check Disk Usage of systemd Logs with journalctl –disk-usage

First things first—before you clean up your logs, it’s good to check how much space they’re using. This way, you can make an informed choice about what to keep and what to get rid of. Luckily, journalctl makes this easy. You can quickly check how much space your logs are taking up by using the --disk-usage flag:

journalctl –disk-usage

This command will give you an output like: “Archived and active journals take up 8.0M in the file system.” That’s your journal log storage in a nutshell. Now you can decide if it’s time for a cleanup or if things are still manageable.

Delete Old Logs with journalctl –vacuum-size and journalctl –vacuum-time

As time goes by, your logs will grow. Eventually, you’ll want to shrink them to free up disk space. No problem— journalctl gives you two easy options for this: --vacuum-size and --vacuum-time . You can clean up logs either based on how big they are or how old they are.

Shrink by Size with –vacuum-size

If you want to control how much space your logs are taking up, you can set a limit with --vacuum-size . This option will delete old logs to keep the total size of the journal under the limit you set. For example, if you want to shrink the journal to 1GB, you can use:

sudo journalctl –vacuum-size=1G

This command will make sure your journal logs stay no larger than 1GB, removing older entries to stay within that size.

Shrink by Time with –vacuum-time

Maybe size isn’t your main concern, and you just want to clear out logs older than a certain time period. No problem. journalctl ’s --vacuum-time option lets you do that. For example, if you want to keep logs from the past year but delete anything older, run:

sudo journalctl –vacuum-time=1years

This will remove logs older than one year while keeping the more recent ones for you to check out if needed.

Configure Disk Space Limits for Journald Logs

Cleaning up logs manually is great, but what if you could have your system handle this automatically? That’s where the journald.conf file comes in. You can set limits on how much space systemd logs can use, so your system can manage log files without you having to check in constantly.

To set these limits, open the journald.conf configuration file:

sudo nano /etc/systemd/journald.conf

Once inside, you can adjust settings like:

• SystemMaxUse (how much disk space the journal can use in persistent storage)
• SystemKeepFree (how much space should remain free)
• SystemMaxFileSize (maximum size for individual journal files before they’re rotated)
• RuntimeMaxUse (limits for volatile storage, which gets wiped after reboot)
• RuntimeKeepFree (reserves space for other processes)
• RuntimeMaxFileSize (limits for journal files in volatile storage)

Here’s an example of what it might look like after you tweak it:

[Journal]
SystemMaxUse=2G
SystemKeepFree=500M
SystemMaxFileSize=50M
RuntimeMaxUse=1G
RuntimeKeepFree=100M
RuntimeMaxFileSize=25M

These settings help ensure your system runs smoothly, managing log space without using up too much storage.

Important Considerations

While adjusting these settings, remember that SystemMaxFileSize and RuntimeMaxFileSize apply to archived logs. After running a cleanup command (like --vacuum-size or --vacuum-time ), these limits will control the size and number of journal files but won’t mess with your overall retention policy.

By setting these values and regularly cleaning up your logs, you can make sure your system stays efficient and optimized—without worrying about logs taking over your storage.

Now you’re ready to manage systemd logs with confidence, ensuring your system stays lean and efficient without getting bogged down by too much log data.

Systemd Journal Logging

Troubleshooting Common journalctl and systemd Journal Issues

You know how it feels—you run a journalctl command expecting to see some logs, but instead, you get a blank screen. No logs, nothing. It’s a common issue that can even leave experienced admins scratching their heads. Whether you’re troubleshooting a problem or just trying to monitor recent system activity, it’s pretty frustrating when the logs don’t show up as expected. But don’t worry, there are a few common reasons for this, and the good news is, you can usually figure it out and fix these problems without too much hassle.

Why Is journalctl Not Showing Logs?

Let’s say you run the journalctl command, but instead of getting the log entries you need, all you see is a blank screen. It’s like the system has gone silent—no answers, no logs. This can definitely be a pain when you’re troubleshooting. But before you start panicking, let’s walk through a few likely causes.

The Journal Database Is Empty or Missing

First off, if there are no logs to be found, it could simply be because the journal is empty. Imagine you’ve just set up a fresh system or are working in a minimal container environment—these setups might not have created any logs yet. If your system is set to store logs in volatile memory (temporary storage), those logs will get wiped every time the system reboots. Want to check if your system is saving logs after reboots? You can do this by looking for the journal directory:

ls /var/log/journal

If it’s missing or empty, you’re probably dealing with a system that doesn’t save logs persistently. The fix? Easy—just create the directory and restart the journald service:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles –create –prefix /var/log/journal
sudo systemctl restart systemd-journald

Once persistent logging is on, your logs will stay around even after a reboot.

The Logging Service May Not Be Running

Another reason you might not see logs is if the logging service isn’t running. If systemd-journald is down, it won’t collect any logs. To check if the service is running, simply use this command:

systemctl status systemd-journald

If the service is inactive or failed, a quick restart should fix the issue:

sudo systemctl restart systemd-journald

This will get log collection back on track and should have you up and running in no time.

Filters May Be Too Narrow

Sometimes, the issue isn’t with the logs themselves—it’s how you’re trying to access them. Maybe you’ve set filters that are just too specific, like looking for logs from a service that doesn’t exist or a time range that doesn’t cover the entries you need. To test this, try running journalctl without any filters at all. If logs show up, then you know your filters are the problem. From there, you can start narrowing things down with less restrictive search terms.

Logs May Have Been Rotated or Deleted

Here’s the thing—systemd logs have size and time limits. If those limits are exceeded, the logs are purged to free up space. You can check how much disk space the journal is using with this command:

journalctl –disk-usage

If you’ve been strict with your disk space settings, or you’ve manually run a vacuum command, older logs might be gone. In this case, you might want to adjust your retention policies to keep logs around longer.

Permission Denied When Using journalctl

Now, let’s talk about permissions. If you try to run journalctl as a regular user, you might get a “permission denied” message. This is totally normal. By default, access to the system logs is restricted to the root user and members of the systemd-journal group. It’s a security measure to protect sensitive system information.

Running journalctl with Elevated Privileges

The easiest solution? Just run the command with sudo . This gives you the elevated privileges needed to view the logs:

sudo journalctl

If you find that you’re always needing access to the logs and don’t want to type sudo every time, you can give your user permanent access.

Granting Your User Access via Group Membership

You can add your user to the systemd-journal group, which will give non-root users access to the logs. Here’s how you do it:

sudo usermod -aG systemd-journal yourusername

Replace yourusername with your actual login name. After that, log out and back in, and you should be able to use journalctl without needing sudo every time.

Verifying Journal Permissions

If adding your user to the group didn’t fix the issue, it might be a permissions problem with the journal directory. The directory /var/log/journal should be owned by root and grouped under systemd-journal . If the permissions are off, you can fix them with these commands:

sudo chown root:systemd-journal /var/log/journal
sudo chmod 2755 /var/log/journal

These commands ensure that users in the systemd-journal group can securely access the logs.

Logs Not Persisting After Reboot

Here’s another problem that might drive you crazy—logs that disappear every time the system reboots. This usually happens when your system is set up to store logs in volatile memory (which gets wiped on reboot). This is common on minimal Linux distributions or containerized systems that don’t focus on keeping logs.

Enabling Persistent Logging

If you want logs to survive a reboot, you’ll need to set up persistent storage for your logs. Here’s how you do it:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles –create –prefix /var/log/journal
sudo systemctl restart systemd-journald

Once that’s done, all future logs will be saved to disk and persist across reboots.

Verifying journald Configuration

If logs still aren’t sticking around, it’s time to check your journald configuration file. Open it up with your favorite text editor:

sudo nano /etc/systemd/journald.conf

Look for the Storage= directive under the [Journal] section. Make sure it’s set to persistent:

[Journal]
Storage=persistent

Save the file, restart the service, and your logs should now be stored persistently.

Debugging a Failed systemd Service

Imagine this: you have a systemd-managed service that fails to start or crashes unexpectedly. journalctl is your best tool here. Unlike traditional log systems that scatter log entries across multiple files, systemd collects everything in one place. Plus, it attaches useful metadata like timestamps and priority levels, making it easier to figure out what went wrong.

Reviewing the Service Status

When a service fails, the first thing you’ll want to do is check its status. This command gives you an overview of the service’s current state and the most recent log entries:

systemctl status nginx.service

This will show you issues like misconfigured paths, permission errors, or unexpected exit codes.

Viewing the Full Log History

To dive deeper into the logs for a failed service, use the -u flag with journalctl :

journalctl -u nginx.service

This will give you a chronological list of logs related to the service. You can narrow it down to just the current boot with the -b flag:

journalctl -u nginx.service -b

Investigating Failure Context

Sometimes, the issue isn’t obvious from the service logs alone. To get a clearer picture, use time-based filters. For example, to view logs from the last 10 minutes:

journalctl -u nginx.service –since “10 minutes ago”

You can also enable verbose logging or look for extended error messages with:

journalctl -xe

Monitoring SSH Login Attempts

SSH is one of the most common ways to access a Linux server, but it’s also a prime target for brute-force attacks and unauthorized access attempts. Luckily, journalctl makes it easy to monitor SSH activity.

Viewing SSH Logs

Systemd tracks SSH activity under the sshd or ssh service (depending on your distribution). To view all SSH-related entries, run:

journalctl -u ssh.service

Or, on some systems:

journalctl -u sshd.service

These logs include details like login attempts, authentication failures, session closures, and key negotiations—perfect for keeping an eye on who’s accessing your system.

Following SSH Activity in Real-Time

If you need to monitor SSH activity as it happens, use the -f flag with journalctl . It’s like the tail -f command but built right into systemd:

journalctl -f -u ssh.service

You’ll see new login events as they happen, which is super useful if you’re worried about an intrusion or just want to monitor active login attempts in real-time.

Filtering by Login Events

If you’re specifically looking for failed login attempts or successful logins, you can filter the logs using keyword searches. For example, to find failed logins:

journalctl -u ssh.service | grep “Failed password”

Or, for successful logins:

journalctl -u ssh.service | grep “Accepted password”

You can narrow down your search with time filters, too. For example, to see logins from the last hour:

journalctl -u ssh.service –since “1 hour ago”

This is incredibly useful for security audits or investigating unauthorized access attempts.

By following these tips and using the right commands, you can troubleshoot and monitor your system with ease. Whether it’s digging into failed services, tracking down a rogue SSH login, or managing your journal logs, journalctl and systemd are here to help.

Systemd Journal Logging

Frequently Asked Questions (FAQs)

Why does journalctl require root access? Imagine you’re looking through logs with journalctl , but instead of seeing what you expect, you get a message saying you don’t have permission to view them. Annoying, right? Here’s why— journalctl requires root access by default because it deals with some sensitive info. These logs contain details from system services, kernel messages, user sessions, and background processes. You might see things like usernames, environment variables, error messages, and security attempts that could be a problem if unauthorized users see them. But don’t worry! You don’t always need root access to view the logs. If you’re not the root user but part of the systemd-journal group, you can still access them. Want to get access? Just add your user to the group with this command:

sudo usermod -aG systemd-journal yourusername

Once you log out and back in, you should be all set, as long as the journal directories have the right file permissions.

How do I make logs persistent across reboots?

Here’s a common problem: your logs disappear every time you reboot. By default, logs are stored in a temporary directory ( /run/log/journal ), which gets cleared when the system shuts down. So, to make sure your logs stay around, you need to enable persistent storage in systemd-journald . Here’s how:

• Create the directory for persistent logs:
  Copy

  sudo mkdir -p /var/log/journal

• Set the correct permissions:
  Copy

  sudo systemd-tmpfiles –create –prefix /var/log/journal

• Restart the journal service:
  Copy

  sudo systemctl restart systemd-journald

Want to make sure your logs always stay after reboot? Open the journald configuration file ( /etc/systemd/journald.conf ) and check that this line is set:

Storage=persistent

With that, all logs will be written to disk and won’t disappear after a reboot. Easy, right?

Can I clear journalctl logs?

Logs taking up too much space? No problem! journalctl lets you clean up old log entries with a few commands. You can clear logs based on their size or how old they are, depending on what you need. Here’s how:

• Remove logs older than a certain time:
  Copy

  sudo journalctl –vacuum-time=2weeks

• Limit the total disk space used by logs:
  Copy

  sudo journalctl –vacuum-size=500M

• Keep only a certain number of log files:
  Copy

  sudo journalctl –vacuum-files=10

These commands will only delete logs that go over the limits you set. If you want to delete everything, you can manually remove the journal files from /var/log/journal , but it’s not usually a good idea on production systems.

How do I access systemd logs?

Accessing system logs is super important, and journalctl makes it simple. With this command, you can see logs for every system service that systemd manages—whether it’s boot messages, system errors, or user session logs.

• To view all logs:
  Copy

  journalctl

• To filter logs for a specific service, like nginx :
  Copy

  journalctl -u nginx.service

• You can narrow things down even more by boot session, priority, time range, or by combining multiple services for the full picture. Unlike traditional log files, journalctl gives you a unified, indexed view of everything.

Which command is used to view systemd logs?

To view logs, use this simple command:

journalctl

It pulls logs from the systemd-journald service, which collects logs from the kernel, system services, and even user sessions. You can narrow down your search with options like:

• View logs for a specific service (e.g., ssh ):
  Copy

  journalctl -u ssh.service

• Filter logs by priority (e.g., errors):
  Copy

  journalctl -p err

• Watch logs in real-time:
  Copy

  journalctl -f

journalctl is really versatile when it comes to accessing system logs!

How to see kernel logs through journalctl?

You know how dmesg is your go-to command for kernel logs? Well, you can get the same logs with journalctl . This is really useful when you need to troubleshoot kernel issues or monitor hardware problems.

• To see only kernel messages, use the -k flag:
  Copy

  journalctl -k

• You can also use the -b flag to see logs from the current or a previous boot:
  Copy

  journalctl -k -b -1

This is great if you’re trying to spot issues that might have started during a past system session.

What is the difference between dmesg and journalctl?

Here’s the deal—both dmesg and journalctl can show kernel logs, but they serve different purposes. Let’s compare:

In short: dmesg focuses on real-time kernel logs, while journalctl offers a more complete and structured log system that spans across the whole system. It’s more persistent, has richer metadata, and is easier to filter.

Systemd Journal Logging

Conclusion

In conclusion, mastering systemd, journald, and journalctl is essential for efficient log management and troubleshooting in Linux systems. By using journalctl’s powerful filtering options, real-time monitoring, and persistent logging capabilities, administrators can optimize system performance and swiftly address issues. Whether you’re managing logs by time, service, or severity, these tools help simplify complex log management tasks. Looking ahead, as systemd continues to evolve, staying updated on the latest features and best practices will ensure smooth, streamlined log management. For system administrators, mastering these tools will be key to improving efficiency and minimizing downtime.

Docker system prune: how to clean up unused resources (2025)