linux-image-4.16.0-1-amd64 boots just up to before sddm, presents a console login for tty1, and waits forever if I do not login.
This is new for me. 4.15.0-1 and 4.15.0-3 sail right past this, taking me to the desktop.
How do I debug this? It took me a couple tries simply to realize I need to turn off boot flags “quiet splash”. I found some services failing without Internet, but disabling these has no effect. It looks like the session @tty1 gets destroyed before I can investigate, tenths of a second after I login.
Any thoughts? Is there a record of activity for tty1?
Actually, I think the login is just exposed because some other “something” is hanging. I use ‘–autologin’ in the getty@tty1.service and it still hangs, although now at the prompt where I can poke around. Fiddling with systemd services doesn’t seem relevant.
Whatever it is hangs before “kernel: random: crng init done”. However, sometimes it seems other activities have hung before crng init, as there are sometimes log entries AFTER the hang but BEFORE “kernel: random: crng init done”.
To debugging: dmesg and journalctl have been most helpful – BUT these still provide little insight beyond when the hang and activities logging before and after. Maybe there is a kernel image built with more debugging?
At this point I should probably revert to “known good” just to clear away various fiddles I’ve made.
I suspect crng init is causing this. RE https://bugzilla.redhat.com/show_bug.cgi?id=1572944
It’s surprising “randomly” typing on the keyboard helps fill the entropy pool.
OK I have a new suspicion: Is there a coin miner hijacking me??
There is at intervals a multithread proc with varying fake names running from a /tmp subdirectory.
$ ps ax
23079 ? Sl 0:00 /tmp/49AC-6715-6117-DF13/dbus-daemon
$ netstat -natp
tcp 0 0 x.x.x.x:56230 192.99.69.170:14444 ESTABLISHED 23079/dbus-daemon
Perhaps the coin miner is blocking startup until I supply creds??
In the meantime I’ve blocked port 14444
# /sbin/iptables -A OUTPUT -p tcp --dport 14444 -j DROP
# iptables-save
$ systemctl restart netfilter-persistent.service
and killed the existing pid
OK so I realized I had a PyCryptoMiner malware infection.
Port 14444 was the giveaway, because this is a standard well-documented PyCryptoMiner port. I blocked this port. Then I tried to track down the infection. I’m still not positive I found all malware. However, deleting ~/.ssh stopped the malware running at boot (after login).
HOPEFULLY this resolves the infection for me. I’m not sure the SSH exploit has been patched, but I think I came across a list of defensive actions I can take to harden SSH.
PS: Killing the proc running under /tmp doesn’t stop the malware because there is a “parent” proc running out of ~/.config that just launches it again at an interval (hoping you have gone AFK). And even this ~/.config is slippery. At login, something using ~/.ssh was creating the ~/.config malware using a random bin filename that would not be noticed from a cursory glance at top or ps. Deleting the executable in /tmp -> recreated. Deleting the executable in ~/.config -> recreated at next login. Deleting ~/.ssh -> resolved for now…
have you checked your .profile, .bashrc and .bash_profile? also see https://github.com/Saren-Arterius/dbus-daemon-trojan-sample
** You link to EXACTLY what had infected me! How did you come across this??
Thanks for suggestions. Yes I had tried those. Ultimately (and somewhat embarrassingly) I found the little bugger in an autostart program pretending to be dbus-daemon. This app would hold boot/startup until it had reached out to the mothership for the latest instructions. It would then confirm/download another program to do the mining, also naming this new app with an existing running process – thus making it difficult to spot, except that this app is running from /tmp instead of somewhere normal. So, a cascade of 3 apps using names of valid running processes, requiring to locate the “head” of this snake – otherwise it simply reinstalls itself.
At length, I am amused to “finally” discover and remove a virus on linux. I have no clue where it came from. I saw something about a repo being compromised for some time; maybe I updated/installed from there, I cannot say. As with virtually every Windoze virus I have repaired for family and friends, I chalk this infection up to user error, to me being reckless with what I install. Word!