Auditing F5 BIG-IP for SSH Persistence and Lateral Movement into Linux

AIMeter and incident scope

The public report says attackers used an F5 BIG-IP appliance to gain SSH access and then moved into enterprise Linux networks. The report does not give enough detail to pin that on one exploit chain, so I would not overstate the initial entry path without better evidence. The important part is the pivot: a trusted network appliance became an SSH foothold inside the environment.

That changes the investigation. You are no longer asking only, “Was the box compromised?” You also have to ask, “What did this box trust, and where did that trust reach?” On BIG-IP, that usually means admin credentials, SSH keys, config access, API tokens, and network adjacency that ordinary Linux hosts do not have.

I like to split this kind of incident into two questions:

1. Did the attacker gain durable control of the appliance?
2. Did they reuse that control to move into Linux systems?

If you ask those in the wrong order, it is easy to miss persistence, rebuild the wrong system first, or leave reused credentials live long enough for the attacker to return.

Why a BIG-IP appliance becomes a pivot point

Management plane vs data plane on F5

BIG-IP sits in an awkward spot in many networks. The data plane moves application traffic. The management plane is where admins log in, change config, sync state, and troubleshoot. Those are different trust domains.

That matters because a compromise on the management side is more than “an appliance bug.” It is access to the place where the box stores its own secrets and network knowledge. Depending on deployment, the management plane may also have direct routes to internal subnets, jump-host-like connectivity, and access patterns that security tools trust more than a random workstation.

When I audit a compromised BIG-IP, I start with one question: what can this box see that a normal server cannot?

Usually the answer includes some mix of:

• administrative accounts
• SSH keys used for automation
• application or API credentials stored in config
• internal DNS or routing context
• privileged access to Linux hosts for monitoring or management

That is why a compromise here often turns into a lateral movement event, not just an appliance incident.

Why SSH access changes the trust boundary

SSH is a stronger signal than an HTTP session or a short-lived exploit shell. SSH usually means the attacker has moved from “code execution somewhere” to “durable interactive access with identity.”

Once SSH is available, a few things get easier:

• they can return without re-exploiting the original bug
• they can use normal admin workflows to blend in
• they can stage tools, copy keys, and test reachability to internal hosts
• they can reuse host trust if the appliance is allowed to SSH into Linux systems

That last point is the one that tends to matter most. If the appliance is used for automation, monitoring, backups, or admin workflows, Linux hosts may already accept traffic from it as trusted. Attackers like that path because it looks normal until you compare timing, source addresses, and command history.

Fast triage after suspected intrusion

Preserve volatile evidence before making changes

The first mistake I see is logging in and “just checking a few things” before collecting volatile state. On a management appliance, that can wipe out the evidence you need.

Before you change anything, preserve:

• current network connections
• active SSH sessions
• running processes
• logged-in users
• recent auth and command history
• running config and any exportable support bundle

If your process allows it, capture a memory image or at least a support archive first. If not, take the least disruptive collection path you have and avoid restarting services until you know why you would restart them.

A good rule: if you suspect active compromise, start with observation, not cleanup.

Build a minimal timeline from logs and operator history

You do not need a perfect timeline on the first pass. You need one that is useful.

I usually build the earliest timeline from:

• SSH auth logs
• admin login history
• config change history
• shell command history, if present
• task scheduler activity
• any outbound connection logs to Linux hosts

On appliances, logs may be split across system auth logs, product-specific logs, and exported audit trails. The goal is not completeness. The goal is to answer:

• When did SSH access first appear?
• Which account or key was used?
• What changed after that?
• Which internal hosts were contacted next?

If you can line those up, you can usually tell whether the attacker was still exploring or had already started moving sideways.

Hunting for SSH persistence on the appliance

Review local users, shell access, and sudo-style privilege paths

Start with the account model. On many appliances, attackers do not bother creating an obviously named backdoor user. They often modify an existing account, enable shell access where it was previously restricted, or abuse a privileged service account.

Look for:

• new local users
• unexpected UID 0 accounts
• changes to login shells
• accounts that suddenly gained admin-level access
• privilege paths that allow passwordless command execution

On BIG-IP, check both the underlying OS view and the appliance’s own administrative model. The box may have one set of Linux accounts and another set of appliance-specific accounts or roles. If either layer changed, that is a persistence clue.

Useful questions:

• Was the default admin or root path altered?
• Did a previously disabled shell become available?
• Did a service account gain a usable shell?
• Are there accounts that do not match your identity management source?

If you find a user that should never have shell access, treat it as a strong persistence candidate until you prove otherwise.

Inspect SSH keys, authorized_keys files, and alternate login entries

If the attacker wanted durable access, SSH keys are the first place I would look.

Check:

• ~/.ssh/authorized_keys for admin and service accounts
• root’s SSH key material
• any files with recent edits in .ssh/
• keys with unknown comment fields or odd usernames
• keys that belong to automation but are no longer documented

Also check for alternate login paths. On Linux-like systems those can include:

• modified SSH daemon config
• forced commands
• wrapper scripts called by login
• PAM or auth configuration changes
• environment files that alter shell startup behavior

The useful mindset is: where else could a login be redirected? Attackers do not always need a new key file if they can alter the authentication path itself.

A quick inventory pattern:

## Find recent changes to SSH material for interactive users
find /home /root -path '*/.ssh/*' -type f -printf '%TY-%Tm-%Td %TT %p\n' 2>/dev/null | sort

## Review authorized keys
for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
  [ -f "$f" ] && echo "== $f ==" && sed 's/^/  /' "$f"
done

If you find keys you do not recognize, compare them to your automation inventory before you remove anything. Unknown does not always mean malicious, but unknown plus recent plus interactive is enough to escalate the case.

Check startup hooks, cron jobs, and service wrappers for re-entry

Persistence on appliances often hides in boring places.

Look for:

• cron jobs that fetch payloads or reopen sessions
• boot-time scripts that restore configs
• wrapper scripts around SSH-related services
• scheduled tasks that sync files to remote systems
• jobs that run as root but were not in your baseline

A compromise that adds one crontab line is often enough to restore SSH access after a reboot or a password reset. I also check for suspicious use of common tools like curl, wget, scp, sftp, nc, and ssh in scheduled scripts. Those tools are normal in admin workflows, which is exactly why they get abused.

For a quick sweep:

## Scheduled tasks and startup-related files
crontab -l 2>/dev/null
for u in root admin; do
  crontab -u "$u" -l 2>/dev/null
done

ls -la /etc/cron* 2>/dev/null
grep -RIn --color=never -E 'ssh|scp|curl|wget|nc|bash -c|sh -c' /etc/cron* /var/spool/cron 2>/dev/null

If the appliance has platform-specific startup locations, check those too. The exact path matters less than the behavior: any automatic re-entry mechanism should stay suspect until you can explain it.

Look for tampering in system binaries, profiles, and config backups

Once SSH persistence is on the table, I start looking for hiding attempts.

That includes:

• modified shell profiles
• replacement binaries with the same names as standard tools
• altered config backups
• timestamps that do not match expected change windows
• unexpected file ownership or permissions

The question is not just “Are files changed?” but “Do those changes help the attacker survive cleanup?”

Examples:

• shell profile changes that run a command at login
• .bashrc or .profile edits that hide commands or open tunnels
• modified backup files that reintroduce a backdoor during restore
• binaries that intercept admin actions or log credentials

On RPM-based systems, package verification is useful:

rpm -Va 2>/dev/null | head -200

That output is noisy, but it quickly shows whether core packages have drifted. Pair that with file hashing on config and startup artifacts so you can compare against a known-good baseline.

Tracing lateral movement into enterprise Linux

Identify outbound SSH sessions and unusual source addresses

If the appliance pivoted into Linux hosts, the network usually leaves a trail.

Look for:

• outbound SSH from the appliance to internal Linux servers
• sessions to hosts that are not normally administered from that source
• traffic at odd hours
• repeated attempts across several hosts
• source addresses that map to the appliance management IP, not a normal admin workstation

If you have network flow records, start there. If not, pull from host logs on the Linux side and compare them with the appliance’s own activity.

I care less about one SSH connection than about the pattern:

• same source
• many destinations
• short burst of logins
• coincident with the time the appliance first looked suspicious

That pattern often separates normal automation from attacker movement.

Correlate appliance activity with Linux host authentication logs

On Linux hosts, authentication logs are the most useful anchor. You want to know whether the appliance was the source of logins, and whether those logins succeeded.

Useful places to correlate include:

• auth.log or secure
• journalctl SSH entries
• sudo usage after login
• new authorized keys or account creation
• login shells and last output

A useful triage command set on Linux hosts:

## Authentication and SSH activity
grep -Ei 'sshd|sudo|authentication|Accepted|Failed' /var/log/auth.log 2>/dev/null | tail -200

## If using systemd journals
journalctl -u ssh -u sshd --since "2026-05-24 00:00" 2>/dev/null | tail -200

## Interactive login history
last -ai | head -50

What you are trying to prove is not just “someone logged in.” You want to show whether the appliance was the source of trust. If the appliance used a privileged automation key to log into a Linux host, that is very different from a random brute-force attempt.

Separate admin automation from attacker-driven reuse of trust

This is where good investigations get sharper. Many environments already use the appliance for legitimate automation: backups, monitoring, config sync, deployment jobs, or scripted diagnostics.

Do not confuse “normal machine-to-machine SSH” with “safe machine-to-machine SSH.”

To separate them, compare:

• source IPs
• time of day
• command sets
• target hosts
• account names
• presence of interactive behavior

Normal automation tends to be:

• consistent
• narrow in scope
• documented
• tied to one or a few service accounts
• regular in timing

Attacker-driven reuse tends to be:

• exploratory
• broader
• bursty
• using admin accounts out of schedule
• followed by more auth attempts or file transfers

If the appliance used an automation key that also exists on other systems, rotate that key immediately and treat every downstream host as potentially exposed.

Practical commands and checks for defenders

File integrity and account inventory checks

Here is the sort of fast checklist I use when I need a defensible first pass.

## Account inventory
getent passwd | sort
getent group | sort

## Recent account changes
grep -E '^(admin|root|service|backup)' /etc/passwd /etc/shadow 2>/dev/null
lastlog | head -50
last -ai | head -100

## SSH configuration and keys
find /root /home -path '*/.ssh/*' -type f -ls 2>/dev/null
find /etc -maxdepth 2 -type f \( -name 'sshd_config' -o -name '*ssh*' \) -ls 2>/dev/null

## Integrity checks
rpm -Va 2>/dev/null | sed -n '1,200p'
sha256sum /etc/passwd /etc/shadow /etc/ssh/sshd_config 2>/dev/null

On the appliance, pair these with the vendor’s supported inventory and diagnostics commands. The goal is to spot drift in accounts, keys, and config files before you decide whether the box is trustworthy enough to keep running.

Log pivots that expose suspicious SSH reuse

A good log pivot is one that turns a vague suspicion into a concrete chain.

Look for:

• first successful SSH login on the appliance
• new key installation or password change around the same time
• subsequent outbound SSH from the appliance
• new logins on Linux hosts from the appliance IP
• any sudo or privilege escalation on those Linux hosts

If you can, build a simple timeline table:

This kind of table helps in incident calls because it keeps people focused on evidence, not assumptions.

Network indicators that support containment decisions

Network data helps decide how aggressive containment should be.

Useful indicators:

• outbound SSH to internal hosts that the appliance never manages
• SSH sessions to multiple Linux servers in a short window
• management-plane traffic from unusual admin sources
• unexpected DNS lookups from the appliance
• remote connections to IPs not in your admin allowlist

If the appliance is still connected to important Linux segments, assume trust may already be reused. At that point, containment is less about watching and more about shrinking the blast radius.

What a good containment plan looks like

Rotate credentials, keys, and API tokens in the right order

Do not rotate randomly. Rotate in a sequence that cuts off attacker access before you restore convenience.

A sensible order is:

1. revoke suspicious SSH keys and session tokens
2. rotate appliance admin credentials
3. rotate service-account keys used for Linux access
4. rotate secrets stored on or reachable from the appliance
5. reissue certificates or API tokens if the appliance handled them

The main idea is to cut off both the original foothold and the lateral path. If you rotate only the appliance password but leave a reusable SSH key in place, the attacker may still have a valid route into Linux hosts.

Reimage or rebuild when integrity is uncertain

If you cannot explain all changes on the appliance, rebuild it. I mean that literally.

A network appliance with unknown persistence should not be “cleaned up” with hope and a few file deletions. If you cannot prove integrity, you cannot prove the box is safe to keep in the trust path.

That usually means:

• preserve evidence first
• export configs and logs as needed
• compare to a known-good baseline
• restore from trusted media or reimage
• reapply config from a verified source
• rotate secrets after restoration, not before

If business pressure pushes back, frame it as a risk decision: an untrusted appliance can become a repeat entry point into every Linux host that trusts it.

Segment management access and remove unnecessary SSH paths

The best containment outcome is not just recovery. It is reducing the number of places where one appliance can act like a jump host.

Practical controls include:

• management plane only reachable from a restricted admin network
• no direct SSH from appliances to production Linux hosts unless required
• service accounts constrained by source IP and host allowlists
• jump hosts and bastions instead of broad trust relationships
• logging on both ends of every admin SSH path

If the BIG-IP does not need to initiate SSH to a Linux segment, make that path impossible.

Hardening BIG-IP to reduce repeat compromise

Restrict management-plane exposure and admin sources

This is the easiest hardening win and the one people postpone the longest.

Reduce exposure by:

• limiting management interfaces to a dedicated admin subnet
• blocking admin access from user networks
• requiring VPN or bastion access for remote administration
• disabling any management path that is not actually needed

If the management plane is internet-reachable or broadly reachable inside the enterprise, you are betting a lot on perfect patching and perfect credentials. That is not a great bet.

Enforce key hygiene, MFA, and privileged access controls

SSH keys are not magic. They are just credentials people forget to rotate.

Improve key hygiene by:

• assigning keys to named humans or bounded automation only
• removing shared keys
• storing ownership and expiration in an inventory
• rotating keys on schedule
• tying admin access to MFA or privileged access workflows where supported

For appliance admin access, I prefer controls that answer two questions:

• Who can log in?
• From where can they log in?

If either answer is “basically anywhere,” the appliance is a pivot waiting to happen.

Monitor for config drift, new accounts, and shell enablement

If you only look for active exploitation, you will miss the quieter persistence layer.

Monitor for:

• new local accounts
• changes to admin role assignments
• shell enablement events
• edits to SSH keys and config files
• unexpected startup or cron changes
• drift in package or binary integrity

A simple alert on “new SSH key added to a privileged account” is often more useful than a hundred generic IDS rules.

What to report internally and what to keep for forensics

Evidence that shows persistence versus one-time access

When you write the internal report, separate transient access from durable control.

Good persistence evidence includes:

• new or modified authorized keys
• new local accounts or shell changes
• startup or cron re-entry
• altered login wrappers or auth config
• commands that survive reboot

One-time access evidence includes:

• a single auth failure burst
• one short shell session with no follow-on changes
• no account, key, or config drift
• no outbound reuse of trust

That distinction matters for leadership because it changes the response. One-time access may still be serious, but durable persistence usually means a much broader reset.

Evidence that shows pivoting into Linux environments

For the pivot claim, preserve:

• appliance logs showing outbound SSH or admin actions
• Linux auth logs showing logins from the appliance IP
• sudo logs and command history on the Linux hosts
• file transfer logs, if present
• network flow records connecting the two sides

If you can show “appliance compromise” and “Linux host login” in the same timeline, you have a strong case that the trust boundary was crossed.

Closing checklist for post-incident audits

Use this as the final pass before you declare the incident contained:

• confirm whether SSH persistence existed on the appliance
• confirm whether any local accounts, keys, or startup hooks changed
• confirm whether the appliance initiated SSH to Linux hosts
• confirm which Linux hosts accepted that trust
• rotate the right credentials in the right order
• reimage or rebuild if integrity is uncertain
• restrict management-plane exposure
• remove unnecessary appliance-to-Linux SSH paths
• add drift monitoring for users, keys, and startup tasks
• preserve evidence for lessons learned and future detection

The real lesson here is not that BIG-IP is special by itself. It is that any appliance sitting on a privileged trust boundary can become a much more valuable foothold than a random server. If an attacker gets SSH on it, treat the rest of the environment as potentially reachable until you can prove otherwise.