pfSense TroubleShouting

How to clone pfsense (backup the configuration and restore it to a fresh installation)

PfsenseBackupAndRestoreConfig

How to change gateway & DNS from the CLI

netstat -r
route del default x.y.z.q # the IP next to default above
echo "nameserver 192.168.1.1" > /etc/resolv.conf
route add default 192.168.1.1

Error when connecting to pfsense's WebUI

If you're getting any error like bad gateway or timeout when connecting to the WebUI

First try the CLI command: /etc/rc.php-fpm_restart OR Login via CLI (keyboard/screen or SSH) and select option "16) Restart PHP-FPM"

If this is not enough try the CLI command run sudo /etc/rc.initial

if you get a 502 Bad Gateway error nginx, run in console service nginx restart

you can also try option (11) Restart webConfigurator.

---

Unable to Check for Updates

First, make sure you have pfSense-base or pfSense-base-nanobsd installed:

pkg info -x pfSense|grep base

If pfSense-base / pfSense-base-nanobsd is missing run:

pkg install pfSense-base

If you can't upgrade it

First try this:

certctl rehash

Then this:

/usr/sbin/pkg-static update -f
/usr/sbin/pkg-static upgrade -f

In rare edge cases it is possible for the pkg database in /var/db/pkg/ to become corrupted. In the unlikely event this happens to a firewall, it can usually be corrected by running a few commands to re-create the database. See https://docs.netgate.com/pfsense/en/latest/packages/fixing-a-broken-pkg-database.html

If pkg-static update -f fails

You may either

a) switch from stable to developer and back again (see another section below)

b) or try a more hackish aproach:

Look at the contents of this dir:

head /usr/local/share/pfSense/keys/pkg/trusted/*
==> /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 <==
function: sha256
fingerprint: 30be9cc2e7b2b3ba1ff3b2be1795f3f0719ab9a65322695703dc7b8f004035a8

You can fetch this fingerprint again with this command (be careful to set the red part as apropriate for the version you are about to go to):

fetch -qo /usr/local/share/pfSense/keys/pkg/trusted/ https://raw.githubusercontent.com/pfsense/pfsense/RELENG_2_3_0/src/usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406

After the fetch run pkg update -f again.

If pkg-static upgrade -f fails

If an error is shown that the kernel package is locked, you can unlock it with "pkg unlock <name>" where <name> is what the error complains about, then run the "pkg update -f" command again. Be sure to lock it back after you're done.

Troubleshooting firewall / NAT problems

(see also https://www.netgate.com/docs/pfsense/firewall/firewall-rule-troubleshooting.html)

Diagnostics > States > Reset States

Status > Filter Reload. Click Reload Filter wait for the process to stop, then scroll to the bottom of the page to see if the last line says Done or if it stops. If it stops, for example in a particular package, then there may be a problem with that package.

/usr/local/etc/rc.filter_configure_sync

(check for any errors on the webUI or clog /var/log/system.log | tail )

pfctl -f /tmp/rules.debug

Moving from developer branch back to stable

Set System > Update > Update Settings to stable and from the console use option 13) Update from console

You can go back to devel from Set System > Update > Update Settings

OLD PC, error: "Under 512MB of RAM detected. Not enabling APC"

diagnosing

sysctl hw | egrep 'hw.(phys|user|real)'
hw.physmem: 3681644544
hw.usermem: 3526725632
hw.realmem: 4194304 <---THIS IS THE RAM REPORTED BY BIOS (old PC)

grep memory /var/run/dmesg.boot
real memory = 4299161600 (4100 MB)
avail memory = 3614662656 (3447 MB)
agp0: aperture size is 256M, detected 32.764k stolen memory

fix?

Re: pfSense thinks I have less than 65mb RAM??
« Reply #6 on: October 20, 2015, 10:19:23 pm »

hw.realmem: 1048576

There's the issue, definitely a BIOS bug. The BIOS is reporting 1 MB RAM there.

Re: Under 512MB of RAM detected. Not enabling APC
« Reply #11 on: July 21, 2016, 02:48:53 pm »
Had this same issue. Turns out my solution was to install the 64-bit version.

WebUI not responding or 502 Bad Gateway & 100% CPU usage by check_reload_status

Description

After a reboot I can connect to the WebUI but after a while it goes down
If you retry to connect browser is waiting for ever or giving 502 bad gateway error
If you try 11) Restart webConfigurator nothing happens
If you try 16) Restart PHP-FPM the probelm persists but now I also get 100% CPU usage from check_reload_status

Forum posts

Re: High CPU load (check_reload_status) + IPv6 Gateways
« Reply #17 on: September 06, 2012, 02:57:32 pm »
check_reload_status itself doesn't tend to be the problem in these cases. Something (such as rc.newwanip) calls pfSctl which routes a request through check_reload_status which in turn tries to take another action - the problem seems to happen more often when there are either:
(a) a lot of programs trying to make calls through check_reload_status
(b) whatever check_reload_status is calling is not behaving as expected

Usually running something like this is enough to bring it back:
killall -9 php; killall -9 lighttpd; /etc/rc.restart_webgui

Once the callers/callees of check_reload status are gone the load tends to return to normal.

Re: New 502 Bad Gateway
« Reply #54 on: October 11, 2017, 03:06:43 pm »
If it is related to memory or a connection or network queue, then in particular the output of these could be helpful:

/usr/bin/netstat -Ln
/usr/bin/netstat -xn
/usr/sbin/swapinfo -h
/usr/bin/top | /usr/bin/head -n10
/bin/ps uxawwd
/usr/bin/sockstat

Another set of notes for the 502 error

report 1

clog -f /var/log/nginx-error.log

---

8<----
2016/07/05 12:40:01 [error] 48883#0: *257237 upstream timed out (60: Operation
timed out) while reading response header from upstream, client: 192.168.254.4,
server: , request: "GET /getstats.php HTTP/1.1", upstream:
"fastcgi://unix:/var/run/php-fpm.socket", host: "vai.revpol.com:4443",
referrer: "https://vai.revpol.com:4443/"
clog: ERROR: could not write output (Bad address)
----8<----

At that same time, system.log shows that the php-fpm.socket socket does not exist:
----8<----
Jul 5 12:47:00 vai vai.revpol.com nginx: 2016/07/05 12:47:00 [crit] 48883#0:
*257442 connect() to unix:/var/run/php-fpm.socket failed (2: No such file or
directory) while connecting to upstream, client: 192.168.254.4, server: ,
request: "GET /getstats.php HTTP/1.1", upstream:
"fastcgi://unix:/var/run/php-fpm.socket:", host: "vai.revpol.com:4443",
referrer: "https://vai.revpol.com:4443/"
----8<----

From the console menu, I can choose option 16 (Restart PHP-FPM), and then the
web gui is accessible again.

report 2

The crash log gives the following:

[28-Jul-2016 09:32:13 Etc/UTC] PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/suhosin.so' - /usr/local/lib/php/20131226/suhosin.so: Undefined symbol "ps_globals" in Unknown on line 0

report 3

nginx-error log contains a bunch of these lines:

2016/03/02 18:31:09 [error] 44734#0: *20649 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.2, server: , request: "GET /ifstats.php?if=re2 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fw.mydomain.se", referrer: "https://fw.mydomain.se/graph.php?ifnum=opt1&ifname=OPT1&timeint=1&initdelay=6"

Bug #6406

Updated by Kill Bill about 1 year ago

seems pretty replicable here when you leave the dashboard page open in a browser for a couple of hours. (Not 2.3.1 specific, was there with 2.3.0 as well.) Note that restarting the webserver alone (console option 11) does not help at all.

Updated by Jim Pingle 5 months ago

Bryan Fehl wrote:

I just ran into this myself. Strangely, this issue causes all clients who try to connect with OpenVPN to just hang indefinitely.

That's normal, OpenVPN uses PHP scripts for authentication and some certificate verification. So if PHP is wedged, then OpenVPN can't authenticate.

Restarting Web Configurator & PHP fixes the issue. It seems to only happen when i leave the PFsense web gui open in my browser for an extended period of time, like if i leave the Dashboard tab open overnight. The only package i have installed is openvpn-client-export. I hope this helps.

Which dashboard widgets do you have visible?

Right now I have the following widgets open:

System Information
Picture
Interfaces
S.M.A.R.T. Status
Gateways
Thermal Sensors
CARP Status
NTP Status
Services Status

Traffic Graphs

IPSec

Bug #6318

Original report

Since 2.3_1 the webconfigurator is continually being non responsive. Attempting to access my https website on port 444 the page hangs and eventually responds with 504 Gateway Time-out - nginx on both IE & Firefox. The nginx-error log file shows the following:

2016/05/05 18:27:54 [error] 30498#0: *254302 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.xxx.10, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "pf.xxxxx.biz:444"
2016/05/05 19:17:32 [alert] 30180#0: close() socket failed (9: Bad file descriptor)
2016/05/05 19:20:44 [error] 87811#0: *12 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.xxx.10, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "pf.xxxxx.biz:444"
2016/05/05 19:27:58 [error] 87811#0: *447 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.xxx.10, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "pf.xxxxx.biz:444"

Updated by Rick Strangman about 1 year ago

I have no issues since removing the IPsec widget. Now on 2.3.1 and have not seen a lockup

Updated by Jim Pingle 7 months ago

it happens every 5-20 minutes (timing varies) while a browser is left on Status > IPsec. The requests are not piling up, they only take about 300ms to complete. Leaving a browser open on Status > IPsec with firebug or similar running, it's easy to spot when it stops responding. When it happens, there are always two PHP child processes:

ps uxawww | grep 'php'
root 64113 0.5 0.9 272496 38300 - S 1:43PM 0:02.26 php-fpm: pool nginx (php-fpm)
root 267 0.0 0.6 268400 25140 - Ss 3:01AM 0:02.49 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root 64043 0.0 0.9 285304 38604 - I 1:43PM 0:00.19 php-fpm: pool nginx (php-fpm)

Updated by Eric Machabert 5 months ago

We are also seeing this on 2.3.3
Running netstat -an shows request filling up the Recv-Q for IPC connection to /var/run/php-fpm.socket.

corelation with openvpn problem

Post by J. Echter
> same issue here. I cannot access OpenVPN after a while and locally i get
> bad gateway when connecting to pfSense webui.

It seems that I have also witnessed that OpenVPN stops working when this occurs.

I recently had a user complain that they could not connect using OpenVPN and
when I went to investigate, I was seeing the "Bad Gateway" in the gui.

At that time, I was in a rush, so I just rebooted the firewall and moved on.