How to verify if all services are running fine on Redhat Satellite Server ?

By Published Linux/Unix

In this post, we will look how to verify all services on a redhat Satellite server are running or not.

Here you will know how to troubleshoot Satellite server issues at first point in case of any issue with it.

 

I. Check to ping all services parts of Satellite Server

[root@ngeSatellite ~]# hammer ping
candlepin:
    Status:          ok
    Server Response: Duration: 9ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 14ms
pulp:
    Status:          FAIL
    Server Response:
pulp_auth:
    Status: FAIL
foreman_tasks:
    Status:          ok
    Server Response: Duration: 3ms

 

II.  In case anything gets failed, we need to further check all services and start the corresponding services.

[root@ngeSatellite ~]# katello-service status
Redirecting to 'foreman-maintain service'
Running Status Services
================================================================================
Get status of applicable services:

Displaying the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, puppetserver, foreman-proxy
/ displaying rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Sun 2021-10-03 04:49:17 BST; 3 weeks 4 days ago
  Process: 90365 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 90370 (code=killed, signal=KILL)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:06 BST; 1 months 10 days ago
  Process: 90170 ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast (code=exited, status=0/SUCCESS)
  Process: 90435 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 90419 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 90443 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─  4045 postgres: candlepin candlepin 127.0.0.1(40782) idl
           ├─  5517 postgres: candlepin candlepin 127.0.0.1(46590) idl
           ├─  5518 postgres: candlepin candlepin 127.0.0.1(46592) idl
           ├─  7248 postgres: foreman foreman [local] idle
           ├─  9495 postgres: candlepin candlepin 127.0.0.1(57788) idl
           ├─  9496 postgres: candlepin candlepin 127.0.0.1(57790) idl
           ├─  9657 postgres: foreman foreman [local] idle
           ├─  9659 postgres: foreman foreman [local] idle
           ├─  9660 postgres: foreman foreman [local] idle
           ├─  9664 postgres: foreman foreman [local] idle
           ├─  9676 postgres: foreman foreman [local] idle
           ├─  9678 postgres: foreman foreman [local] idle
           ├─  9679 postgres: foreman foreman [local] idle
           ├─ 21696 postgres: foreman foreman [local] idle
           ├─ 53716 postgres: foreman foreman [local] idle
           ├─ 90443 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─ 90444 postgres: logger process
           ├─ 90446 postgres: checkpointer process
           ├─ 90447 postgres: writer process
           ├─ 90448 postgres: wal writer process
           ├─ 90449 postgres: autovacuum launcher process
           ├─ 90450 postgres: stats collector process
           ├─ 91609 postgres: candlepin candlepin 127.0.0.1(43928) idl
           ├─ 91610 postgres: candlepin candlepin 127.0.0.1(43930) idl
           ├─ 91611 postgres: candlepin candlepin 127.0.0.1(43932) idl
           ├─ 91626 postgres: candlepin candlepin 127.0.0.1(43936) idl
           ├─ 91627 postgres: candlepin candlepin 127.0.0.1(43938) idl
           ├─ 91685 postgres: foreman foreman [local] idle
           ├─ 91695 postgres: foreman foreman [local] idle
           ├─ 91697 postgres: foreman foreman [local] idle
           ├─ 93445 postgres: foreman foreman [local] idle
           ├─ 93446 postgres: foreman foreman [local] idle
           ├─101355 postgres: foreman foreman [local] idle
           ├─101357 postgres: foreman foreman [local] idle
           ├─101358 postgres: foreman foreman [local] idle
           ├─105158 postgres: foreman foreman [local] idle
           ├─107512 postgres: foreman foreman [local] idle
           ├─129289 postgres: foreman foreman [local] idle
           ├─129291 postgres: foreman foreman [local] idle
           ├─129292 postgres: foreman foreman [local] idle
           └─129296 postgres: foreman foreman [local] idle

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying qdrouterd
● qdrouterd.service - Qpid Dispatch router daemon
   Loaded: loaded (/usr/lib/systemd/system/qdrouterd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:05 BST; 1 months 10 days ago
 Main PID: 90421 (qdrouterd)
   CGroup: /system.slice/qdrouterd.service
           └─90421 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf

Oct 27 00:55:44 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:44.168325 +0100 SERVER (info) [1257]: Accepted connection to :5647 from 10.4.65.9:13158
Oct 27 00:55:44 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:44.252624 +0100 SERVER (info) [1258]: Accepted connection to :5647 from 10.4.65.9:13160
Oct 27 00:55:46 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:46.709694 +0100 SERVER (info) [1259]: Accepted connection to :5647 from 10.4.65.8:24780
Oct 27 00:55:46 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:46.787504 +0100 SERVER (info) [1260]: Accepted connection to :5647 from 10.4.65.8:24782
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.021685 +0100 SERVER (info) [1261]: Accepted connection to :5647 from 10.4.65.62:41252
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.025105 +0100 SERVER (info) [1262]: Accepted connection to :5647 from 10.4.65.63:48662
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.105058 +0100 SERVER (info) [1263]: Accepted connection to :5647 from 10.4.65.62:41254
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.118759 +0100 SERVER (info) [1264]: Accepted connection to :5647 from 10.4.65.63:48666
Oct 27 00:57:41 ngeSatellite qdrouterd[90421]: 2021-10-27 00:57:41.290754 +0100 SERVER (info) [1265]: Accepted connection to :5647 from 10.4.65.61:37330
Oct 27 00:57:41 ngeSatellite qdrouterd[90421]: 2021-10-27 00:57:41.364724 +0100 SERVER (info) [1266]: Accepted connection to :5647 from 10.4.65.61:37332
- displaying qpidd
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qpidd.service.d
           └─wait-for-port.conf
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
  Process: 90417 ExecStartPost=/bin/bash -c while ! nc -z 127.0.0.1 5671; do sleep 1; done (code=exited, status=0/SUCCESS)
 Main PID: 90415 (qpidd)
   CGroup: /system.slice/qpidd.service
           └─90415 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying squid
● squid.service - Squid caching proxy
   Loaded: loaded (/usr/lib/systemd/system/squid.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:05 BST; 1 months 10 days ago
  Process: 90168 ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 90427 ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 90414 ExecStartPre=/usr/libexec/squid/cache_swap.sh (code=exited, status=0/SUCCESS)
 Main PID: 90438 (squid)
   CGroup: /system.slice/squid.service
           ├─90438 /usr/sbin/squid -f /etc/squid/squid.conf
           └─90440 (squid-1) -f /etc/squid/squid.conf

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying pulp_celerybeat
● pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2021-10-03 04:50:37 BST; 3 weeks 4 days ago
  Process: 90492 ExecStart=/usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler (code=exited, status=1/FAILURE)
 Main PID: 90492 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying pulp_resource_manager
● pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
 Main PID: 90501 (celery)
   CGroup: /system.slice/pulp_resource_manager.service
           ├─90501 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid
           └─90933 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid

Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     self.__read_concern))
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     **kwargs)
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     server = topology.select_server(selector)
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     address))
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     self._error_message(selector))
Oct 28 16:15:37 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
- displaying pulp_streamer
● pulp_streamer.service - The Pulp lazy content loading streamer
   Loaded: loaded (/usr/lib/systemd/system/pulp_streamer.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
 Main PID: 90499 (pulp_streamer)
   CGroup: /system.slice/pulp_streamer.service
           └─90499 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac

Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     return self.gen.next()
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 712, in _get_socket
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     server = self._get_topology().select_server(selector)
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     address))
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     self._error_message(selector))
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: [-] 127.0.0.1 - - [28/Oct/2021:15:06:54 +0000] "GET /var/lib/pulp/content/units/rpm/88/78c41d465008218e812aedfe41795077ed1b39489bb3cd0192f3074e925eca/rsyslog-5.8.10-12.el6.x86_64.rpm HTTP/1.1" 500 - "-" "urlgrabber/3.9.1 yum/3.2.29"
- displaying pulp_workers
● pulp_workers.service - Pulp Celery Workers
   Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2021-09-17 23:09:08 BST; 1 months 10 days ago
  Process: 89535 ExecStop=/usr/bin/python -m pulp.server.async.manage_workers stop (code=exited, status=0/SUCCESS)
  Process: 90504 ExecStart=/usr/bin/python -m pulp.server.async.manage_workers start (code=exited, status=0/SUCCESS)
 Main PID: 90504 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/pulp_workers.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying smart_proxy_dynflow_core
● smart_proxy_dynflow_core.service - Foreman smart proxy dynflow core service
   Loaded: loaded (/usr/lib/systemd/system/smart_proxy_dynflow_core.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/smart_proxy_dynflow_core.service.d
           └─90-limits.conf
   Active: active (running) since Fri 2021-09-17 23:09:09 BST; 1 months 10 days ago
     Docs: https://github.com/theforeman/smart_proxy_dynflow
  Process: 90503 ExecStart=/usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid (code=exited, status=0/SUCCESS)
 Main PID: 90713 (ruby)
   CGroup: /system.slice/smart_proxy_dynflow_core.service
           └─90713 ruby /usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying tomcat
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
 Main PID: 90502 (java)
   CGroup: /system.slice/tomcat.service
           └─90502 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Oct 27 18:23:55 ngeSatellite server[90502]: Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
Oct 27 18:23:55 ngeSatellite server[90502]: java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:194)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
Oct 27 18:23:55 ngeSatellite server[90502]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Oct 27 18:23:55 ngeSatellite server[90502]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Oct 27 18:23:55 ngeSatellite server[90502]: at java.lang.Thread.run(Thread.java:748)
- displaying dynflowd
● dynflowd.service - Foreman jobs daemon
   Loaded: loaded (/usr/lib/systemd/system/dynflowd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:15 BST; 1 months 10 days ago
     Docs: https://theforeman.org
  Process: 88832 ExecStop=/usr/sbin/dynflowd stop (code=exited, status=0/SUCCESS)
  Process: 90731 ExecStart=/usr/sbin/dynflowd start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/dynflowd.service
           ├─91230 dynflow_executor
           └─91234 dynflow_executor_monitor

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying goferd
● goferd.service - Gofer Agent
   Loaded: loaded (/usr/lib/systemd/system/goferd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:09 BST; 1 months 10 days ago
 Main PID: 90730 (python)
   CGroup: /system.slice/goferd.service
           └─90730 /usr/bin/python /usr/bin/goferd --foreground

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:10 BST; 1 months 10 days ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 88831 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 13764 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 90725 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─  9637 Passenger RackApp: /usr/share/foreman
           ├─  9666 Passenger RackApp: /usr/share/foreman
           ├─ 13792 (wsgi:pulp)     -DFOREGROUND
           ├─ 13793 (wsgi:pulp)     -DFOREGROUND
           ├─ 13794 (wsgi:pulp)     -DFOREGROUND
           ├─ 13795 (wsgi:pulp-cont -DFOREGROUND
           ├─ 13796 (wsgi:pulp-cont -DFOREGROUND
           ├─ 13797 (wsgi:pulp-cont -DFOREGROUND
           ├─ 13798 (wsgi:pulp_forg -DFOREGROUND
           ├─ 13799 PassengerWatchdog
           ├─ 13802 PassengerHelperAgent
           ├─ 13808 PassengerLoggingAgent
           ├─ 14032 /bin/bash /usr/bin/tfm-ruby /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb
           ├─ 14040 scl enable tfm bash /tmp/tmp.4QRuUoSAF4
           ├─ 14041 /bin/bash /var/tmp/scl1hHFqq
           ├─ 14064 bash /tmp/tmp.4QRuUoSAF4
           ├─ 14065 Passenger AppPreloader: /usr/share/foreman
           ├─ 23719 /usr/sbin/httpd -DFOREGROUND
           ├─ 26775 /usr/sbin/httpd -DFOREGROUND
           ├─ 26873 /usr/sbin/httpd -DFOREGROUND
           ├─ 78681 /usr/sbin/httpd -DFOREGROUND
           ├─ 90725 /usr/sbin/httpd -DFOREGROUND
           ├─ 94378 /usr/sbin/httpd -DFOREGROUND
           ├─101340 Passenger RackApp: /usr/share/foreman
           ├─110121 /usr/sbin/httpd -DFOREGROUND
           ├─110163 /usr/sbin/httpd -DFOREGROUND
           ├─115635 /usr/sbin/httpd -DFOREGROUND
           ├─115659 /usr/sbin/httpd -DFOREGROUND
           ├─127419 /usr/sbin/httpd -DFOREGROUND
           ├─128968 /usr/sbin/httpd -DFOREGROUND
           ├─129256 /usr/sbin/httpd -DFOREGROUND
           ├─129277 /usr/sbin/httpd -DFOREGROUND
           ├─129278 /usr/sbin/httpd -DFOREGROUND
           └─129279 Passenger RackApp: /usr/share/foreman

Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)     return self.gen.next()
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 712, in _get_socket
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)     server = self._get_topology().select_server(selector)
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)     address))
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768)     self._error_message(selector))
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:ERROR: (13793-16768) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
Oct 28 16:15:35 ngeSatellite pulp[13793]: pulp.server.webservices.application:CRITICAL: *************************************************************
- displaying puppetserver
● puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:10:02 BST; 1 months 10 days ago
  Process: 88830 ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop (code=exited, status=0/SUCCESS)
  Process: 90732 ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start (code=exited, status=0/SUCCESS)
 Main PID: 90760 (java)
   CGroup: /system.slice/puppetserver.service
           └─90760 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying foreman-proxy
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:10:04 BST; 1 months 10 days ago
 Main PID: 91975 (ruby)
   CGroup: /system.slice/foreman-proxy.service
           └─91975 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Oct 27 19:04:24 ngeSatellite smart-proxy[91975]: qualys-ngel-int.ind.ngelinux.net - - [27/Oct/2021:19:04:24 BST] "GET /console/framework/skins/wlsconsole/images/%252E%252E%252Fconsole.portlet?_nfpb=false&_pageLabel=HomePage1&handle=com.tangosol.coherence.mvel2.sh.ShellSession(%22java.lang.Runtime.getRuntime().exec("mstsc /v:10.4.160.126:52264");"); HTTP/1.0" 400 546
Oct 27 19:04:24 ngeSatellite smart-proxy[91975]: - ->
Oct 27 19:04:34 ngeSatellite smart-proxy[91975]: qualys-ngel-int.ind.ngelinux.net - - [27/Oct/2021:19:04:34 BST] "GET /console/framework/skins/wlsconsole/images/%252E%252E%252Fconsole.portlet?_nfpb=true&_pageLabel=HomePage1&handle=com.tangosol.coherence.mvel2.sh.ShellSession(%22java.lang.Runtime.getRuntime().exec("mstsc /v:10.4.160.126:52264");"); HTTP/1.0" 400 545
Oct 27 19:04:34 ngeSatellite smart-proxy[91975]: - ->
Oct 27 19:04:44 ngeSatellite smart-proxy[91975]: qualys-ngel-int.ind.ngelinux.net - - [27/Oct/2021:19:04:44 BST] "GET /console/framework/skeletons/wlsconsole/js/%252E%252E%252Fconsole.portlet?_nfpb=false&_pageLabel=HomePage1&handle=com.tangosol.coherence.mvel2.sh.ShellSession(%22java.lang.Runtime.getRuntime().exec("curl http://10.4.160.126:52264");"); HTTP/1.0" 400 549
Oct 27 19:04:44 ngeSatellite smart-proxy[91975]: - ->
Oct 27 19:50:50 ngeSatellite smart-proxy[91975]: ngeSatellite.ind.ngelinux.net - - [27/Oct/2021:19:50:50 BST] "GET /pulp/status/disk_usage HTTP/1.1" 200 627
Oct 27 19:50:50 ngeSatellite smart-proxy[91975]: - -> /pulp/status/disk_usage
Oct 28 07:51:00 ngeSatellite smart-proxy[91975]: ngeSatellite.ind.ngelinux.net - - [28/Oct/2021:07:51:00 BST] "GET /pulp/status/disk_usage HTTP/1.1" 200 627
Oct 28 07:51:00 ngeSatellite smart-proxy[91975]: - -> /pulp/status/disk_usage
- All services displayed                                              [FAIL]
Some services are not running (rh-mongodb34-mongod, pulp_celerybeat)
--------------------------------------------------------------------------------
Scenario [Status Services] failed.

The following steps ended up in failing state:

  [service-status]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="service-status"
[root@ngeSatellite ~]#

 

III. Re-start the stopped or pending services.

### We have 2 services in stopped state, we will start them up.
[root@ngeSatellite ~]# systemctl stop rh-mongodb34-mongod.service
[root@ngeSatellite ~]# systemctl start rh-mongodb34-mongod.service
[root@ngeSatellite ~]# systemctl status rh-mongodb34-mongod.service
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-10-28 16:17:13 BST; 5s ago
  Process: 14943 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 14948 (mongod)
   CGroup: /system.slice/rh-mongodb34-mongod.service
           └─14948 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] **        We suggest setting it to 'never'
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten]
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] **        We suggest setting it to 'never'
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten]
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] Initializing full-time diagnostic data capture with directory '/v...c.data'
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [thread1] waiting for connections on port 27017
Oct 28 16:17:13 ngeSatellite mongodb-scl-helper[14943]: child process started successfully, parent exiting
Oct 28 16:17:13 ngeSatellite systemd[1]: Started High-performance, schema-free document-oriented database.
Hint: Some lines were ellipsized, use -l to show in full.




[root@ngeSatellite ~]# systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2021-10-03 04:50:37 BST; 3 weeks 4 days ago
  Process: 90492 ExecStart=/usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler (code=exited, status=1/FAILURE)
 Main PID: 90492 (code=exited, status=1/FAILURE)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[root@ngeSatellite ~]# systemctl restart pulp_celerybeat.service

[root@ngeSatellite ~]# systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-10-28 16:19:06 BST; 3s ago
 Main PID: 17385 (celery)
   CGroup: /system.slice/pulp_celerybeat.service
           └─17385 /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.s...

Oct 28 16:19:06 ngeSatellite systemd[1]: Started Pulp's Celerybeat.
Oct 28 16:19:07 ngeSatellite pulp[17385]: celery.beat:INFO: beat: Starting...
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
[root@ngeSatellite ~]#

 

III. Now check the status again.

[root@ngeSatellite ~]# katello-service status
Redirecting to 'foreman-maintain service'
Running Status Services
================================================================================
Get status of applicable services:

Displaying the following service(s):
rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, goferd, httpd, puppetserver, foreman-proxy
- displaying rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-10-28 16:17:13 BST; 2min 8s ago
  Process: 14943 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 14948 (mongod)
   CGroup: /system.slice/rh-mongodb34-mongod.service
           └─14948 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run

Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten]
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] **        We suggest setting it to 'never'
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten]
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
Oct 28 16:17:13 ngeSatellite mongod.27017[14948]: [thread1] waiting for connections on port 27017
Oct 28 16:17:13 ngeSatellite mongodb-scl-helper[14943]: child process started successfully, parent exiting
Oct 28 16:17:13 ngeSatellite systemd[1]: Started High-performance, schema-free document-oriented database.
Oct 28 16:18:41 ngeSatellite mongod.27017[14948]: [conn29] received client metadata from 127.0.0.1:46732 conn29: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.9" }, os: { type: "Linux", name: "Red Hat Enterprise Linux Server release 7.9 (Maipo)", architecture: "x86_64", version: "Kernel 3.10.0-1160.25.1.el7.x86_64" } }
Oct 28 16:19:21 ngeSatellite mongod.27017[14948]: [conn34] received client metadata from 127.0.0.1:46824 conn34: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.9" }, os: { type: "Linux", name: "Red Hat Enterprise Linux Server release 7.9 (Maipo)", architecture: "x86_64", version: "Kernel 3.10.0-1160.25.1.el7.x86_64" } }
- displaying postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:06 BST; 1 months 10 days ago
  Process: 90170 ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast (code=exited, status=0/SUCCESS)
  Process: 90435 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 90419 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 90443 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─  5517 postgres: candlepin candlepin 127.0.0.1(46590) idl
           ├─  7248 postgres: foreman foreman [local] idle
           ├─  9495 postgres: candlepin candlepin 127.0.0.1(57788) idl
           ├─  9496 postgres: candlepin candlepin 127.0.0.1(57790) idl
           ├─ 13772 postgres: candlepin candlepin 127.0.0.1(38630) idl
           ├─ 15776 postgres: candlepin candlepin 127.0.0.1(40798) idl
           ├─ 21696 postgres: foreman foreman [local] idle
           ├─ 53716 postgres: foreman foreman [local] idle
           ├─ 90443 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─ 90444 postgres: logger process
           ├─ 90446 postgres: checkpointer process
           ├─ 90447 postgres: writer process
           ├─ 90448 postgres: wal writer process
           ├─ 90449 postgres: autovacuum launcher process
           ├─ 90450 postgres: stats collector process
           ├─ 91609 postgres: candlepin candlepin 127.0.0.1(43928) idl
           ├─ 91610 postgres: candlepin candlepin 127.0.0.1(43930) idl
           ├─ 91611 postgres: candlepin candlepin 127.0.0.1(43932) idl
           ├─ 91626 postgres: candlepin candlepin 127.0.0.1(43936) idl
           ├─ 91627 postgres: candlepin candlepin 127.0.0.1(43938) idl
           ├─ 91685 postgres: foreman foreman [local] idle
           ├─ 91695 postgres: foreman foreman [local] idle
           ├─ 91697 postgres: foreman foreman [local] idle
           ├─ 93445 postgres: foreman foreman [local] idle
           ├─ 93446 postgres: foreman foreman [local] idle
           ├─101355 postgres: foreman foreman [local] idle
           ├─101357 postgres: foreman foreman [local] idle
           ├─101358 postgres: foreman foreman [local] idle
           ├─105158 postgres: foreman foreman [local] idle
           ├─107512 postgres: foreman foreman [local] idle
           ├─129289 postgres: foreman foreman [local] idle
           ├─129291 postgres: foreman foreman [local] idle
           ├─129292 postgres: foreman foreman [local] idle
           └─129296 postgres: foreman foreman [local] idle

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying qdrouterd
● qdrouterd.service - Qpid Dispatch router daemon
   Loaded: loaded (/usr/lib/systemd/system/qdrouterd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:05 BST; 1 months 10 days ago
 Main PID: 90421 (qdrouterd)
   CGroup: /system.slice/qdrouterd.service
           └─90421 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf

Oct 27 00:55:44 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:44.168325 +0100 SERVER (info) [1257]: Accepted connection to :5647 from 10.4.65.9:13158
Oct 27 00:55:44 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:44.252624 +0100 SERVER (info) [1258]: Accepted connection to :5647 from 10.4.65.9:13160
Oct 27 00:55:46 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:46.709694 +0100 SERVER (info) [1259]: Accepted connection to :5647 from 10.4.65.8:24780
Oct 27 00:55:46 ngeSatellite qdrouterd[90421]: 2021-10-27 00:55:46.787504 +0100 SERVER (info) [1260]: Accepted connection to :5647 from 10.4.65.8:24782
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.021685 +0100 SERVER (info) [1261]: Accepted connection to :5647 from 10.4.65.62:41252
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.025105 +0100 SERVER (info) [1262]: Accepted connection to :5647 from 10.4.65.63:48662
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.105058 +0100 SERVER (info) [1263]: Accepted connection to :5647 from 10.4.65.62:41254
Oct 27 00:56:24 ngeSatellite qdrouterd[90421]: 2021-10-27 00:56:24.118759 +0100 SERVER (info) [1264]: Accepted connection to :5647 from 10.4.65.63:48666
Oct 27 00:57:41 ngeSatellite qdrouterd[90421]: 2021-10-27 00:57:41.290754 +0100 SERVER (info) [1265]: Accepted connection to :5647 from 10.4.65.61:37330
Oct 27 00:57:41 ngeSatellite qdrouterd[90421]: 2021-10-27 00:57:41.364724 +0100 SERVER (info) [1266]: Accepted connection to :5647 from 10.4.65.61:37332
- displaying qpidd
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qpidd.service.d
           └─wait-for-port.conf
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
  Process: 90417 ExecStartPost=/bin/bash -c while ! nc -z 127.0.0.1 5671; do sleep 1; done (code=exited, status=0/SUCCESS)
 Main PID: 90415 (qpidd)
   CGroup: /system.slice/qpidd.service
           └─90415 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying squid
● squid.service - Squid caching proxy
   Loaded: loaded (/usr/lib/systemd/system/squid.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:05 BST; 1 months 10 days ago
  Process: 90168 ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 90427 ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 90414 ExecStartPre=/usr/libexec/squid/cache_swap.sh (code=exited, status=0/SUCCESS)
 Main PID: 90438 (squid)
   CGroup: /system.slice/squid.service
           ├─90438 /usr/sbin/squid -f /etc/squid/squid.conf
           └─90440 (squid-1) -f /etc/squid/squid.conf

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying pulp_celerybeat
● pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-10-28 16:19:06 BST; 15s ago
 Main PID: 17385 (celery)
   CGroup: /system.slice/pulp_celerybeat.service
           └─17385 /usr/bin/python /usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler

Oct 28 16:19:06 ngeSatellite systemd[1]: Started Pulp's Celerybeat.
Oct 28 16:19:07 ngeSatellite pulp[17385]: celery.beat:INFO: beat: Starting...
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Oct 28 16:19:07 ngeSatellite pulp[17385]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
- displaying pulp_resource_manager
● pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
 Main PID: 90501 (celery)
   CGroup: /system.slice/pulp_resource_manager.service
           ├─90501 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid
           └─90933 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid

Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     self.__read_concern))
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 850, in __send_message
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     **kwargs)
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 777, in _send_message_with_response
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     server = topology.select_server(selector)
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     address))
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744)     self._error_message(selector))
Oct 28 16:17:01 ngeSatellite pulp[90501]: kombu.async.hub:ERROR: (90501-51744) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
- displaying pulp_streamer
● pulp_streamer.service - The Pulp lazy content loading streamer
   Loaded: loaded (/usr/lib/systemd/system/pulp_streamer.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
 Main PID: 90499 (pulp_streamer)
   CGroup: /system.slice/pulp_streamer.service
           └─90499 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac

Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     return self.gen.next()
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 712, in _get_socket
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     server = self._get_topology().select_server(selector)
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 141, in select_server
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     address))
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)   File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 117, in select_servers
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176)     self._error_message(selector))
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: pulp.streamer.server:ERROR: (90499-94176) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
Oct 28 16:06:55 ngeSatellite pulp_streamer[90499]: [-] 127.0.0.1 - - [28/Oct/2021:15:06:54 +0000] "GET /var/lib/pulp/content/units/rpm/88/78c41d465008218e812aedfe41795077ed1b39489bb3cd0192f3074e925eca/rsyslog-5.8.10-12.el6.x86_64.rpm HTTP/1.1" 500 - "-" "urlgrabber/3.9.1 yum/3.2.29"
- displaying pulp_workers
● pulp_workers.service - Pulp Celery Workers
   Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2021-09-17 23:09:08 BST; 1 months 10 days ago
  Process: 89535 ExecStop=/usr/bin/python -m pulp.server.async.manage_workers stop (code=exited, status=0/SUCCESS)
  Process: 90504 ExecStart=/usr/bin/python -m pulp.server.async.manage_workers start (code=exited, status=0/SUCCESS)
 Main PID: 90504 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/pulp_workers.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying smart_proxy_dynflow_core
● smart_proxy_dynflow_core.service - Foreman smart proxy dynflow core service
   Loaded: loaded (/usr/lib/systemd/system/smart_proxy_dynflow_core.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/smart_proxy_dynflow_core.service.d
           └─90-limits.conf
   Active: active (running) since Fri 2021-09-17 23:09:09 BST; 1 months 10 days ago
     Docs: https://github.com/theforeman/smart_proxy_dynflow
  Process: 90503 ExecStart=/usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid (code=exited, status=0/SUCCESS)
 Main PID: 90713 (ruby)
   CGroup: /system.slice/smart_proxy_dynflow_core.service
           └─90713 ruby /usr/bin/smart_proxy_dynflow_core -d -p /var/run/foreman-proxy/smart_proxy_dynflow_core.pid

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying tomcat
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:07 BST; 1 months 10 days ago
 Main PID: 90502 (java)
   CGroup: /system.slice/tomcat.service
           └─90502 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Oct 27 18:23:55 ngeSatellite server[90502]: Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
Oct 27 18:23:55 ngeSatellite server[90502]: java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:194)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
Oct 27 18:23:55 ngeSatellite server[90502]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Oct 27 18:23:55 ngeSatellite server[90502]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Oct 27 18:23:55 ngeSatellite server[90502]: at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Oct 27 18:23:55 ngeSatellite server[90502]: at java.lang.Thread.run(Thread.java:748)
- displaying dynflowd
● dynflowd.service - Foreman jobs daemon
   Loaded: loaded (/usr/lib/systemd/system/dynflowd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:15 BST; 1 months 10 days ago
     Docs: https://theforeman.org
  Process: 88832 ExecStop=/usr/sbin/dynflowd stop (code=exited, status=0/SUCCESS)
  Process: 90731 ExecStart=/usr/sbin/dynflowd start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/dynflowd.service
           ├─91230 dynflow_executor
           └─91234 dynflow_executor_monitor

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying goferd
● goferd.service - Gofer Agent
   Loaded: loaded (/usr/lib/systemd/system/goferd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:09 BST; 1 months 10 days ago
 Main PID: 90730 (python)
   CGroup: /system.slice/goferd.service
           └─90730 /usr/bin/python /usr/bin/goferd --foreground

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:09:10 BST; 1 months 10 days ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 88831 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 13764 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 90725 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─ 13792 (wsgi:pulp)     -DFOREGROUND
           ├─ 13793 (wsgi:pulp)     -DFOREGROUND
           ├─ 13794 (wsgi:pulp)     -DFOREGROUND
           ├─ 13795 (wsgi:pulp-cont -DFOREGROUND
           ├─ 13796 (wsgi:pulp-cont -DFOREGROUND
           ├─ 13797 (wsgi:pulp-cont -DFOREGROUND
           ├─ 13798 (wsgi:pulp_forg -DFOREGROUND
           ├─ 13799 PassengerWatchdog
           ├─ 13802 PassengerHelperAgent
           ├─ 13808 PassengerLoggingAgent
           ├─ 14032 /bin/bash /usr/bin/tfm-ruby /usr/share/gems/gems/passenger-4.0.18/helper-scripts/rack-preloader.rb
           ├─ 14040 scl enable tfm bash /tmp/tmp.4QRuUoSAF4
           ├─ 14041 /bin/bash /var/tmp/scl1hHFqq
           ├─ 14064 bash /tmp/tmp.4QRuUoSAF4
           ├─ 14065 Passenger AppPreloader: /usr/share/foreman
           ├─ 23719 /usr/sbin/httpd -DFOREGROUND
           ├─ 26775 /usr/sbin/httpd -DFOREGROUND
           ├─ 26873 /usr/sbin/httpd -DFOREGROUND
           ├─ 78681 /usr/sbin/httpd -DFOREGROUND
           ├─ 90725 /usr/sbin/httpd -DFOREGROUND
           ├─ 94378 /usr/sbin/httpd -DFOREGROUND
           ├─101340 Passenger RackApp: /usr/share/foreman
           ├─110121 /usr/sbin/httpd -DFOREGROUND
           ├─110163 /usr/sbin/httpd -DFOREGROUND
           ├─115635 /usr/sbin/httpd -DFOREGROUND
           ├─115659 /usr/sbin/httpd -DFOREGROUND
           ├─127419 /usr/sbin/httpd -DFOREGROUND
           ├─128968 /usr/sbin/httpd -DFOREGROUND
           ├─129256 /usr/sbin/httpd -DFOREGROUND
           ├─129277 /usr/sbin/httpd -DFOREGROUND
           ├─129278 /usr/sbin/httpd -DFOREGROUND
           └─129279 Passenger RackApp: /usr/share/foreman

Oct 28 16:17:31 ngeSatellite pulp[13793]: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized
Oct 28 16:17:31 ngeSatellite pulp[13793]: pulp.server.webservices.application:INFO: *************************************************************
Oct 28 16:17:31 ngeSatellite pulp[13793]: gofer.messaging.adapter.qpid.connection:INFO: open: URL: ssl://localhost|SSL: ca: /etc/pki/katello/certs/katello-default-ca.crt|key: None|certificate: /etc/pki/katello/qpid_client_striped.crt|host-validation: False
Oct 28 16:17:31 ngeSatellite pulp[13793]: gofer.messaging.adapter.qpid.connection:INFO: opened: qpid+ssl://localhost:5671
Oct 28 16:17:31 ngeSatellite pulp[13793]: gofer.messaging.adapter.connect:INFO: connected: qpid+ssl://localhost:5671
Oct 28 16:17:31 ngeSatellite pulp[13793]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Oct 28 16:18:03 ngeSatellite pulp[13798]: crane.data:INFO: loading metadata from /var/lib/pulp/published/docker/v2/app
Oct 28 16:18:03 ngeSatellite pulp[13798]: crane.data:INFO: finished loading metadata
Oct 28 16:19:03 ngeSatellite pulp[13798]: crane.data:INFO: loading metadata from /var/lib/pulp/published/docker/v2/app
Oct 28 16:19:03 ngeSatellite pulp[13798]: crane.data:INFO: finished loading metadata
- displaying puppetserver
● puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:10:02 BST; 1 months 10 days ago
  Process: 88830 ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop (code=exited, status=0/SUCCESS)
  Process: 90732 ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start (code=exited, status=0/SUCCESS)
 Main PID: 90760 (java)
   CGroup: /system.slice/puppetserver.service
           └─90760 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/server/apps/puppetserver/jruby-1_7.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying foreman-proxy
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-09-17 23:10:04 BST; 1 months 10 days ago
 Main PID: 91975 (ruby)
   CGroup: /system.slice/foreman-proxy.service
           └─91975 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Oct 27 19:04:24 ngeSatellite smart-proxy[91975]: qualys-ngel-int.ind.ngelinux.net - - [27/Oct/2021:19:04:24 BST] "GET /console/framework/skins/wlsconsole/images/%252E%252E%252Fconsole.portlet?_nfpb=false&_pageLabel=HomePage1&handle=com.tangosol.coherence.mvel2.sh.ShellSession(%22java.lang.Runtime.getRuntime().exec("mstsc /v:10.4.160.126:52264");"); HTTP/1.0" 400 546
Oct 27 19:04:24 ngeSatellite smart-proxy[91975]: - ->
Oct 27 19:04:34 ngeSatellite smart-proxy[91975]: qualys-ngel-int.ind.ngelinux.net - - [27/Oct/2021:19:04:34 BST] "GET /console/framework/skins/wlsconsole/images/%252E%252E%252Fconsole.portlet?_nfpb=true&_pageLabel=HomePage1&handle=com.tangosol.coherence.mvel2.sh.ShellSession(%22java.lang.Runtime.getRuntime().exec("mstsc /v:10.4.160.126:52264");"); HTTP/1.0" 400 545
Oct 27 19:04:34 ngeSatellite smart-proxy[91975]: - ->
Oct 27 19:04:44 ngeSatellite smart-proxy[91975]: qualys-ngel-int.ind.ngelinux.net - - [27/Oct/2021:19:04:44 BST] "GET /console/framework/skeletons/wlsconsole/js/%252E%252E%252Fconsole.portlet?_nfpb=false&_pageLabel=HomePage1&handle=com.tangosol.coherence.mvel2.sh.ShellSession(%22java.lang.Runtime.getRuntime().exec("curl http://10.4.160.126:52264");"); HTTP/1.0" 400 549
Oct 27 19:04:44 ngeSatellite smart-proxy[91975]: - ->
Oct 27 19:50:50 ngeSatellite smart-proxy[91975]: ngeSatellite.ind.ngelinux.net - - [27/Oct/2021:19:50:50 BST] "GET /pulp/status/disk_usage HTTP/1.1" 200 627
Oct 27 19:50:50 ngeSatellite smart-proxy[91975]: - -> /pulp/status/disk_usage
Oct 28 07:51:00 ngeSatellite smart-proxy[91975]: ngeSatellite.ind.ngelinux.net - - [28/Oct/2021:07:51:00 BST] "GET /pulp/status/disk_usage HTTP/1.1" 200 627
Oct 28 07:51:00 ngeSatellite smart-proxy[91975]: - -> /pulp/status/disk_usage
- All services are running                                            [OK]
--------------------------------------------------------------------------------

[root@ngeSatellite ~]#

Seems useful article, right !!

In case some service does not come up then you can continue your troubleshooting with that.

It usually helps us to address most of the issues with Satellite server.

0 0 votes
Article Rating

Tags:

Subscribe
Notify of
guest

1 Comment
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
Beat M.
Beat M.
1 year ago

Hello all,

thanks for this page to debug RH Satellite issues, I didn’t know “hammer ping” before.
It didn’t resolve my problems, but it provided an excellent starting point to debug my issues with pulp, celery and mongod.

Cheers,
Beat

1
Reply