Debian 13: APT upgrade gone wrong — roll back one package without a domino collapse

December 30, 2025 • February 3, 2026 • Read: 23 min • Views: 9

Was this helpful?

You upgraded a Debian 13 box, one package went sideways, and now the system is behaving like it’s auditioning for a tragedy: services flapping, strange symbol errors, maybe even an unbootable unit if you got unlucky. You don’t want a full rollback. You want one package back where it was. You also don’t want APT to “help” by downgrading half your stack, because you’ve seen that movie and the ending is always overtime.

This is the production-grade way to do it: identify exactly what changed, pick the correct version, downgrade it with controlled dependency impact, then pin/hold so it stays put until you’re ready. Along the way, we’ll talk about snapshots, dependency edges, and why “just apt install the old .deb” is how you become a cautionary tale.

Ground rules: what “rollback one package” really means on Debian

Debian doesn’t have a built-in “package rollback” button because APT isn’t a time machine; it’s a dependency solver. Rolling back a single package is possible, but you’re negotiating with relationships:

Dependencies: the package you want might require a newer library than the old version did, or vice versa.
ABI compatibility: libraries and kernel-facing bits have compatibility promises… until they don’t, or your app uses an edge case.
Repository state: the version you want must be available from a configured source or your local cache.
Policy: APT will choose “the best” candidate by pins, origin, priority, and suite. If you don’t set policy, you get vibes.

The goal is not “force install an old .deb.” The goal is: downgrade one package to a known-good version, while preventing an uncontrolled cascade of downgrades or removals.

Second goal: make the system explain itself. If you can’t predict what APT will do, you’re not rolling back—you’re gambling.

Joke #1: APT is like a very confident intern: fast, helpful, and absolutely capable of rearranging your entire house to find the stapler.

Fast diagnosis playbook

When you’re under the gun, don’t start with “try random downgrades.” Start with the minimum checks that reveal the dependency and packaging shape of the incident.

First: identify what changed and what is currently broken

What package version is installed now?
What version was installed before?
What service or binary fails, and why (symbols, config, permission, ABI)?
Is dpkg in a half-configured state?

Second: determine if rollback is feasible without a chain reaction

Does the target package depend on libraries that are now newer than the old version expects?
Will downgrading it force downgrades of libraries other packages need?
Is the old version still available from your configured repos or cache?

Third: implement a controlled rollback

Simulate with APT and inspect proposed changes.
Downgrade with explicit version selection.
Pin/hold to prevent re-upgrade.
Run a minimal verification: service status, logs, and a sanity query.

Fourth: prevent recurrence

Snapshot before upgrades (filesystem or VM snapshot).
Stage upgrades (canary host first).
Track version deltas in change management, not in someone’s memory.

Facts and context that make APT behavior less mysterious

These aren’t trivia night answers. They’re the mental model you need when APT starts suggesting “remove 37 packages” and calls it a feature.

APT is a resolver, dpkg is the installer. APT decides what should happen; dpkg executes, tracks state, and can be left half-done if something interrupts it.
“Depends” is strict, “Recommends” is (mostly) optional. Debian historically treated Recommends as optional, then APT defaulted to installing them, which surprises people who learned Debian in earlier eras.
Versioned dependencies are common in core stacks. Libraries often require “>= x.y” and sometimes “<< x.z” (rare, but spicy). Those constraints dictate how far you can roll back.
Pin priorities control candidate selection. It’s not magic: pins create a partial ordering; APT picks the highest priority candidate version that satisfies dependencies.
Debian’s “stable” is conservative; “testing” moves. If your Debian 13 system pulls from mixed suites or third-party repos, you’re effectively running a custom distribution.
Downgrades are supported, but not frictionless. APT will do them, but many maintainers primarily test forward upgrades. You’re swimming upstream.
Package maintainer scripts can change state outside files. postinst scripts may enable services, create users, migrate data, or rewrite configs. Rolling back a package doesn’t always roll back those side effects.
“apt-get” vs “apt” isn’t just vibes. The newer apt command is friendlier for humans; apt-get is stable for scripts. In an incident, prefer predictability.
Debian has had multiarch for years. A single package name can have multiple architectures installed (like :amd64 and :i386), and that can complicate “one package” rollbacks.

The 12+ practical tasks: commands, outputs, and decisions

Each task includes: a runnable command, example output, what it means, and the decision you make from it. You won’t need all of them, but you should know which one answers which question.

Task 1: Confirm dpkg isn’t half-configured

cr0x@server:~$ sudo dpkg --audit
The following packages are only half configured, probably due to problems configuring them the first time.
 libssl3:amd64

What it means: Your system isn’t in a clean package state. Rolling back while dpkg is mid-flight is how you get “works on my terminal” followed by a reboot into regret.

Decision: Fix dpkg state first: run sudo dpkg --configure -a and resolve errors before attempting downgrades.

Task 2: Get the installed version and the candidate version

cr0x@server:~$ apt-cache policy libssl3
libssl3:
  Installed: 3.3.1-1
  Candidate: 3.3.2-1
  Version table:
     3.3.2-1 500
        500 http://deb.debian.org/debian trixie/main amd64 Packages
 *** 3.3.1-1 100
        100 /var/lib/dpkg/status

What it means: Installed differs from candidate. You can downgrade/upgrade based on what’s available. The Version table shows where versions come from and their pin priorities.

Decision: If the version you want isn’t listed, you’ll need to add a repository that provides it, or find it in the local cache.

Task 3: Find what upgraded recently (APT history)

cr0x@server:~$ sudo zgrep -h "Start-Date\|Upgrade:" /var/log/apt/history.log*
Start-Date: 2025-12-29  02:13:41
Upgrade: nginx:amd64 (1.26.1-2, 1.26.2-1), nginx-common:amd64 (1.26.1-2, 1.26.2-1)
End-Date: 2025-12-29  02:14:12

What it means: This tells you what actually changed, not what you think changed.

Decision: If the problematic package is in that list, rollback is a valid hypothesis. If it isn’t, you may be chasing the wrong thing (like a library upgrade).

Task 4: Check dpkg logs for maintainer script failures

cr0x@server:~$ sudo grep -E "install |upgrade |configure " /var/log/dpkg.log | tail -n 8
2025-12-29 02:13:45 upgrade nginx:amd64 1.26.1-2 1.26.2-1
2025-12-29 02:13:46 status half-configured nginx:amd64 1.26.2-1
2025-12-29 02:13:47 status installed nginx:amd64 1.26.2-1

What it means: If you see “half-configured” that never resolves, you likely have a postinst issue (config syntax, permissions, systemd reload failure).

Decision: If a maintainer script broke, rollback might help, but fixing config might be faster and safer than downgrading.

Task 5: Inspect why a package is installed and who depends on it

cr0x@server:~$ apt-cache rdepends --installed nginx | head -n 12
nginx
Reverse Depends:
  nginx-extras
  nginx-full
  myapp-web

What it means: Rolling back nginx will affect anything that depends on it.

Decision: If a critical internal package depends on a minimum nginx version, rolling back nginx may force rolling back that package too. Decide if that’s acceptable.

Task 6: Show dependency constraints (the ones that bite)

cr0x@server:~$ apt-cache show nginx | sed -n '1,80p'
Package: nginx
Version: 1.26.2-1
Depends: nginx-common (= 1.26.2-1), libc6 (>= 2.38), libpcre2-8-0 (>= 10.34)

What it means: Version-locked dependencies ((= 1.26.2-1)) mean you can’t roll back nginx alone; you must roll back the paired package too.

Decision: Plan rollback at the “source package set” level when you see tight equality dependencies.

Task 7: Check what versions are available for downgrade

cr0x@server:~$ apt list -a nginx 2>/dev/null | sed -n '1,6p'
nginx/trixie 1.26.2-1 amd64
nginx/trixie 1.26.1-2 amd64

What it means: The repo offers both versions. Good: you can ask APT for an exact version without hunting .debs.

Decision: If the desired version is missing, stop and fix the availability problem first (repo suite, snapshot, or cache).

Task 8: Dry-run the downgrade and inspect the blast radius

cr0x@server:~$ sudo apt-get -s install nginx=1.26.1-2 nginx-common=1.26.1-2
Reading package lists... Done
Building dependency tree... Done
The following packages will be DOWNGRADED:
  nginx nginx-common
0 upgraded, 0 newly installed, 2 downgraded, 0 to remove and 0 not upgraded.
Inst nginx [1.26.2-1] (1.26.1-2 Debian:13.0/trixie [amd64])
Inst nginx-common [1.26.2-1] (1.26.1-2 Debian:13.0/trixie [amd64])

What it means: Perfect outcome: only the target packages downgrade. No removals, no surprise library downgrades.

Decision: Proceed if the simulation is narrow. If APT wants to remove packages or downgrade core libs, do not proceed without a plan (pinning, alternative version, or snapshot rollback).

Task 9: Execute the downgrade with explicit versions

cr0x@server:~$ sudo apt-get install nginx=1.26.1-2 nginx-common=1.26.1-2
Reading package lists... Done
Building dependency tree... Done
The following packages will be DOWNGRADED:
  nginx nginx-common
Need to get 0 B/1,233 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
dpkg: warning: downgrading nginx:amd64 from 1.26.2-1 to 1.26.1-2

What it means: Dpkg is downgrading; this is supported behavior.

Decision: Watch for maintainer script errors. If they occur, grab logs immediately and decide whether to fix config, rerun configure, or revert with snapshots.

Task 10: Verify the service and surface errors quickly

cr0x@server:~$ systemctl status nginx --no-pager -l
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-12-29 02:21:06 UTC; 12s ago
       Docs: man:nginx(8)

What it means: Service is up. Don’t stop here—status is not correctness, it’s just a pulse.

Decision: If it’s not active, inspect journal logs next.

Task 11: Read the journal for the package’s failure mode

cr0x@server:~$ sudo journalctl -u nginx -b --no-pager -n 30
Dec 29 02:20:58 server nginx[1422]: nginx: [emerg] unknown directive "http2_push_preload" in /etc/nginx/conf.d/site.conf:12
Dec 29 02:20:58 server systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE

What it means: This is not a binary/ABI issue. It’s a config directive mismatch. Rolling back might “fix” it, but the real fix is to reconcile config with supported syntax.

Decision: Decide whether to revert config or adjust config for the new version. Downgrading a web server to accommodate stale config is usually the wrong hill to die on.

Task 12: Verify shared library / symbol errors (classic post-upgrade pain)

cr0x@server:~$ /usr/bin/myapp-web
/usr/bin/myapp-web: symbol lookup error: /lib/x86_64-linux-gnu/libssl.so.3: undefined symbol: EVP_MD_get_type

What it means: ABI mismatch. Your binary expects a symbol not present in the currently installed library version (or vice versa).

Decision: Identify which package provides the library and pick a compatible version pair (or rebuild the app). Rolling back just the application package may not help if the library changed.

Task 13: Map a file to its owning package (when you only know the broken .so)

cr0x@server:~$ dpkg -S /lib/x86_64-linux-gnu/libssl.so.3
libssl3:amd64: /lib/x86_64-linux-gnu/libssl.so.3

What it means: The broken symbol is in libssl3. That’s the rollback target, or the app needs rebuilding against the newer ABI.

Decision: Check available versions of libssl3, and check who depends on it before downgrading a core crypto library.

Task 14: Inspect who depends on a core library before touching it

cr0x@server:~$ apt-cache rdepends --installed libssl3 | head -n 20
libssl3
Reverse Depends:
  openssh-client
  curl
  nginx
  myapp-web

What it means: Downgrading libssl3 might affect multiple critical packages. This is where “rollback one package” becomes “change the foundation.”

Decision: If the reverse-dep list includes security-sensitive core tooling (ssh, curl), prefer rebuilding the app or using container isolation rather than downgrading the system library.

Task 15: Find cached .debs already on the box (when repos moved on)

cr0x@server:~$ ls -1 /var/cache/apt/archives | grep -E '^nginx_'
nginx_1.26.1-2_amd64.deb
nginx_1.26.2-1_amd64.deb

What it means: You may have the exact .deb you need without changing repos. Useful for fast rollback during an incident.

Decision: If you install from cache, still prefer APT with explicit version to keep dependency management intact.

Task 16: See what APT would remove if you attempt a downgrade (red flag)

cr0x@server:~$ sudo apt-get -s install libssl3=3.3.0-2
Reading package lists... Done
Building dependency tree... Done
The following packages have unmet dependencies:
 openssh-client : Depends: libssl3 (>= 3.3.1-1) but 3.3.0-2 is to be installed
E: Unable to correct problems, you have held broken packages.

What it means: The downgrade is incompatible with other installed packages. APT refuses, which is a gift.

Decision: Don’t force it with dpkg. Either roll back the dependent packages too (if acceptable), or choose another strategy (rebuild app, container, or snapshot rollback).

Choose the rollback target: version, origin, and scope

Rolling back “the package that broke” is often wrong because the visible break is just the first domino. The real cause is typically one of these:

Library ABI shift (openssl, libc, zlib, libstdc++): you get symbol lookup errors or crashes.
Config schema change (nginx, systemd units, database configs): service won’t start, logs complain about unknown directives or invalid values.
Tooling behavior change (python packaging, node tooling, openssh defaults): your scripts silently fail or behave differently.
Kernel/userspace interface mismatch (drivers, eBPF tooling): particularly with out-of-tree modules or vendor agents.

Decide: do you roll back the app, the library, or both?

My bias: roll back the app before you roll back the OS plumbing. Downgrading application packages tends to be localized. Downgrading crypto libraries and libc is how you expand the incident into “why can’t we ssh anymore?”

But sometimes the app is fine and the library is the regression. Then you have three practical options:

Upgrade forward (preferred when a fixed version exists). If the bug is in 1.2.3 and 1.2.4 fixes it, rollback is a distraction.
Rebuild the app against the new library ABI if you control the build. This avoids system downgrades.
Downgrade the library only if (a) the repo offers a compatible set and (b) the reverse dependencies can tolerate it.

Decide: which repository “owns” the version you want

In Debian land, your available versions depend on your configured sources and their priorities. Mixing suites (stable/testing/unstable) is possible, but it’s like mixing fuel grades with a shrug.

Use apt-cache policy and apt list -a to see what’s actually available. If the version isn’t available, you can’t downgrade cleanly with APT without changing your sources or using cached packages.

Downgrade safely: minimizing blast radius

Here’s the workflow that keeps you out of dependency hell:

Simulate the downgrade with apt-get -s.
Inspect the action list: downgraded packages, removed packages, newly installed packages.
Downgrade with explicit versions for the package set (often includes -common or -data packages).
Immediately pin/hold the packages to stop APT from “fixing” them on the next upgrade.
Verify runtime behavior: service, logs, and a real request/query.

Simulate like you mean it

Simulation is not optional. In production, you don’t “try” package operations. You predict them.

If APT wants to remove packages, read the list carefully. Removal can be rational (conflicts), but it’s also how you end up removing your monitoring agent and then spending two hours “debugging a performance issue” with no graphs.

Know when to stop and use a snapshot

If the rollback requires downgrading core components like libc6, systemd, openssl, or the kernel, you’re no longer doing a “one package rollback.” You’re doing a distribution partial rewind. That’s where filesystem/VM snapshots shine: they restore a coherent system state.

Joke #2: The fastest rollback is the one you took a snapshot for yesterday. The second fastest is the one you can explain to future-you without crying.

Pinning and holding: keeping APT from “fixing” your fix

Downgrading is step one. Keeping it downgraded is step two. Step two is where people fail quietly, then get surprised during the next unattended upgrade window.

Use a hold for immediate containment

A hold is blunt, fast, and reversible. It prevents upgrades of that package.

cr0x@server:~$ sudo apt-mark hold nginx nginx-common
nginx set on hold.
nginx-common set on hold.

Meaning: These packages won’t be upgraded until you unhold them.

Decision: Use holds for incident containment. Then implement pinning if you need a longer-lived policy.

Use APT pinning when you need policy, not a bandaid

Pinning is better when you run mixed repositories or when you want to allow upgrades generally but force a specific package to stay at a chosen version.

Create a preferences file, for example:

cr0x@server:~$ sudo tee /etc/apt/preferences.d/nginx-rollback.pref >/dev/null <<'EOF'
Package: nginx nginx-common
Pin: version 1.26.1-2
Pin-Priority: 1001
EOF

Meaning: Priority > 1000 tells APT it may select this version even if it’s a downgrade.

Decision: Use high-priority pins sparingly. You’re overriding the resolver; do it with a comment in change management and a reminder to remove it later.

Confirm the pin is active

cr0x@server:~$ apt-cache policy nginx | sed -n '1,12p'
nginx:
  Installed: 1.26.1-2
  Candidate: 1.26.1-2
  Version table:
     1.26.2-1 500
        500 http://deb.debian.org/debian trixie/main amd64 Packages
 *** 1.26.1-2 1001
        500 http://deb.debian.org/debian trixie/main amd64 Packages
        100 /var/lib/dpkg/status

Meaning: Candidate equals your pinned version. That’s what you want.

Decision: If Candidate is still the newer version, your preferences file is wrong (package name mismatch, wrong version string, or a different pin priority outranks it).

Three corporate mini-stories from the trenches

1) Incident caused by a wrong assumption: “It’s just the app package”

The team ran a fleet of Debian web nodes with a small Go service fronting a TLS-terminating proxy. After a routine upgrade, health checks started failing on a handful of hosts, then more. The error everyone latched onto was in the app logs: TLS handshake failures. So the assumption formed quickly: “the app upgrade broke TLS.”

Someone rolled back the app package on two hosts. No change. Someone else rolled back the proxy. Still broken. The incident commander started pushing broader rollbacks, one by one, because the systems were still reachable and the business dashboard was not. The upgrade window turned into a roulette wheel.

The actual root cause was a library mismatch introduced by a partial upgrade: a host rebooted mid-upgrade (power event in a rack row), leaving dpkg in an inconsistent state. Some nodes had the new crypto library and an older dependent package; others had the reverse. The proxy was fine, the app was fine. The ABI alignment was not.

When they finally looked at dpkg --audit and the dpkg log, it was obvious. The fix wasn’t “rollback the app.” It was “finish the upgrade cleanly” on the half-upgraded nodes, then restart services. The lesson landed hard: the error message pointed at TLS, but the failure mode was packaging state.

Afterward they added a gate: no automated rollout proceeds unless dpkg is clean and the last APT transaction completed. It was boring. It also prevented a repeat.

2) Optimization that backfired: “Let’s speed up upgrades by mixing suites”

A platform group wanted newer features in one component—call it a reverse proxy or a database driver—without waiting for the stable cadence. They did what plenty of smart people do under deadline pressure: added a testing suite line to sources.list and pinned it “low.” The intent was to cherry-pick a package occasionally. The reality was that dependency graphs don’t respect intent.

It worked for months. Then a routine upgrade pulled in a newer library from testing because it had a higher pin priority than expected for that package’s origin (a subtle preference rule mismatch). The proxy upgraded, its modules pulled in a tighter library dependency, and suddenly several other packages were eligible for upgrade. APT did its job: found a consistent solution. The solution just wasn’t consistent with operational safety.

Two weeks later, they tried to roll back “the one package we upgraded from testing.” APT proposed downgrading the library, which meant downgrading other packages that had learned to expect the newer one. The rollback turned into a tug-of-war between suites. Every “fix” made the next upgrade harder.

The eventual repair was not heroic. They removed the mixed suite, reverted to a single suite plus a controlled backport approach, and used explicit pins with an expiry date (yes, literally a ticket reminding them to remove the pin). The improvement was real: upgrades became predictable again. The “optimization” had traded short-term speed for long-term instability.

3) Boring but correct practice that saved the day: snapshots and a canary host

A finance-adjacent system had strict uptime demands and strict change controls. The team wasn’t flashy. They were careful. Before any upgrade, their process was: snapshot, upgrade a canary, run smoke tests, then proceed. Everyone rolled their eyes at the paperwork until the day it mattered.

A Debian 13 upgrade introduced a regression in a package used for PDF generation. The service started, but output was subtly wrong for certain fonts. If you’re thinking “who cares,” the auditors cared. The canary caught it because the smoke test included generating a representative document and hashing it. Not a unit test; a real-world test.

Instead of scrambling, they pinned that single package version, rolled back only that package on the canary, confirmed output correctness, then rolled the same rollback to production nodes. No cascade. No random removals. No drama.

Meanwhile, they opened an internal issue to evaluate the newer package version and determine whether to patch config or wait for a fixed build. The snapshot never needed to be restored, but knowing it was there changed the whole posture of the incident. Calm beats clever.

Common mistakes: symptoms → root cause → fix

This section is where incidents go to stop repeating. Treat it like a checklist of traps.

1) “APT wants to remove half the system”

Symptoms: On a downgrade attempt, APT proposes removing large chunks (desktop, python stack, database).

Root cause: You’re trying to install a version that conflicts with the currently installed dependency set; often caused by mixed suites, third-party repos, or version-locked dependencies.

Fix: Stop. Simulate and inspect. Pin the specific version you want and ensure the corresponding dependency set exists in the same repository suite. If core libs are involved, prefer snapshot rollback or forward upgrade.

2) “Unmet dependencies, held broken packages”

Symptoms: APT errors with unmet dependencies; dpkg can’t correct problems.

Root cause: You have a partial upgrade, broken pins, or conflicting repository priorities. Sometimes a package is on hold and blocking a required upgrade/downgrade path.

Fix: Check holds with apt-mark showhold. Validate pin priorities with apt-cache policy. Repair dpkg state with dpkg --configure -a and apt-get -f install if appropriate.

3) “Service won’t start after upgrade; rollback ‘fixes’ it”

Symptoms: systemd unit fails; logs show unknown directive or invalid config option.

Root cause: Config drift. Your config relies on behavior removed or changed in the new version. Rolling back restores the older parser, not correctness.

Fix: Update config to match current version, or explicitly manage versioned config. Use journalctl to identify the line and directive. Rollback only as a temporary containment measure.

4) “Symbol lookup error”

Symptoms: Executables fail with undefined symbol errors in shared libraries.

Root cause: ABI mismatch between binary and library versions; common with locally built binaries or vendor-provided agents.

Fix: Prefer rebuilding/reinstalling the binary for the new library. If you must roll back a library, verify reverse dependencies and avoid downgrading core security tooling.

5) “Downgrade succeeded, but next upgrade re-breaks it”

Symptoms: Everything works after downgrade, then unattended upgrades bring the bad version back.

Root cause: No hold or pin was applied; APT still considers the newer version the candidate.

Fix: Use apt-mark hold for immediate containment, then implement pinning with /etc/apt/preferences.d/ for longer control.

6) “dpkg locked, upgrades stuck”

Symptoms: APT says lock is held; operations fail.

Root cause: Another apt/dpkg process is running (unattended-upgrades, apt-daily), or a previous process crashed leaving a lock but still-running dpkg state is the real issue.

Fix: Check running processes, don’t just delete lock files. Stop timers if needed, then repair dpkg state.

Checklists / step-by-step plan

Checklist A: Contain the incident (5–15 minutes)

Confirm what is broken (service down? binary error? wrong output?).
Check dpkg state (dpkg --audit). Repair if needed.
Identify what changed (apt history, dpkg log tail).
Decide if rollback is appropriate (config issue vs ABI issue vs genuine regression).
Simulate downgrade to measure blast radius (apt-get -s install pkg=ver).

Checklist B: Roll back one package without collateral damage

List available versions: apt list -a package.
Check dependencies: apt-cache show package.
Check reverse dependencies: apt-cache rdepends --installed package.
Simulate downgrade with explicit versions for all tightly coupled packages (often -common, -data).
Execute the downgrade with explicit versions.
Hold or pin immediately.
Verify service and logs.

Checklist C: Prevent the next domino collapse

Remove mixed suites unless you have an explicit policy for them.
Use canary upgrades and smoke tests that reflect actual workloads.
Snapshot before upgrades (VM, LVM, btrfs, ZFS—whatever you have).
Track pins and holds as temporary measures with owners and expiry dates.

FAQ

1) Can I downgrade a package with dpkg directly?

You can, but you usually shouldn’t. Dpkg doesn’t resolve dependencies. If you install an older .deb that conflicts with installed dependencies, you’ll create a broken system state that APT then has to unwind. Prefer apt-get install pkg=version so dependency resolution stays coherent.

2) Why does APT want to downgrade a bunch of other packages?

Because your requested version changes dependency constraints. If the older version requires older libraries or conflicts with newer ones, APT tries to find a consistent set. The solver is doing its job; you’re asking it to solve a harder problem than you intended. That’s your cue to simulate, inspect, and potentially change strategy.

3) What’s the difference between a hold and a pin?

A hold blocks changes to a package (upgrade/downgrade) unless you explicitly unhold it. Pinning influences candidate selection and can be as surgical or as dangerous as you make it. Holds are great for quick containment; pins are for policy.

4) How do I see if a package is held?

cr0x@server:~$ apt-mark showhold
nginx
nginx-common

If a package you need to move is held, that hold may be why APT says “held broken packages.” Unhold deliberately, not impulsively.

5) What if the old version isn’t in my repositories anymore?

First check /var/cache/apt/archives. If it’s not there, you need a repository that provides the version (for example, a controlled internal mirror or snapshot service). Avoid random .debs from the internet; the integrity chain and dependency closure matter.

6) Is it safe to roll back libraries like libssl or libc?

Sometimes, but it’s rarely the best first move. Those libraries are widely depended upon and are security-sensitive. If a single app breaks due to a library update, prefer rebuilding that app or isolating it (container, chroot) rather than downgrading system-wide crypto or libc.

7) How do I know which package provides a failing shared object?

Use dpkg -S /path/to/file. That maps a file to the owning package. Then you can inspect versions and decide whether the library package is your true rollback target.

8) What if the failure is due to a config change and not the binary?

Then rollback is a temporary crutch. Fix the config to be compatible with the new package version. Use journalctl -u service for the exact directive/line. Also check for new default configs in /usr/share/doc or /etc conffile prompts you may have skipped during upgrade.

9) Should I use `apt` or `apt-get` during incidents?

I use apt-get for incident work because its interface is stable and predictable, especially for simulations (-s) and scripted repeatability. apt is fine, but friendliness is not the same as control.

10) How do I audit what APT changed last night?

Use /var/log/apt/history.log and /var/log/dpkg.log. They tell you what APT planned and what dpkg actually did. If those two disagree, that’s a clue (interruption, maintainer script failures, or manual intervention).

Conclusion: next steps you can do today

Rolling back one Debian 13 package without a dependency collapse is not black magic. It’s policy, simulation, and restraint. Your playbook is:

Make dpkg clean before touching anything else.
Prove what changed via logs, not memory.
Simulate the downgrade and refuse big blast radii.
Downgrade with explicit versions for tightly coupled packages.
Hold/pin immediately, then verify with real workload checks.
After the fire is out, remove ad-hoc mixed repos and adopt snapshots + canaries.

One quote worth keeping taped to your monitor, because it’s as true in packaging as it is everywhere else: Hope is not a strategy. — Edsger W. Dijkstra

If you do nothing else: start snapshotting before upgrades and always dry-run downgrades. That alone will save you from the next “APT suggested removing my monitoring stack” moment, which is a uniquely modern form of self-inflicted blindness.