Join the Community
and take part in the story

AWS S3 CLI access/secret key


#23
root@gateway-2:~# dpkg -l puppet-module-openio-openiosds
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version            Architecture       Description
+++-===========================-==================-==================-============================================================
ii  puppet-module-openio-openio 3.0.1-1            all                Puppet module for OpenIO SDS solution

#24

After pipeline was replaced and oioswift restarted,

root@gateway-2:~# gridinit_cmd status @oioswift
KEY                STATUS      PID GROUP
OPENIO-oioswift-0  BROKEN       -1 OPENIO,oioswift,oioswift-0

#25

Ok.
If you want to retrieve a functional oioswift you can rollback on the pipeline configuration and restart oioswift.

Despite that, you have a very old version of this package module …
Could you give me the output of:
dpkg -l | grep openio on your swift node.

Am i right if i say you are on ubuntu on your swift node and centos on your openio node ?


#26

Hi Sébastien,

root@gateway-2:~# dpkg -l | grep openio
ii  openio-asn1c                       0.9.27.1-1                         amd64        Free, Open Source ASN.1 compiler
ii  openio-gridinit                    1.6-2                              amd64        OpenIO gridinit daemon
ii  openio-sds                         3.2.4-1                            amd64        OpenIO Cloud Storage Solution
ii  openio-sds-logrotate               1.2-1                              all          OpenIO SDS logrotate configuration
ii  openio-sds-rsyslog                 1.2-2                              all          OpenIO SDS rsyslog configuration
ii  openio-sds-swift                   0.8.2-1                            all          Swift Gateway for OpenIO SDS
ii  puppet-module-openio-gridinit      1.1.9-1                            all          Puppet module for gridinit from OpenIO
ii  puppet-module-openio-openiosds     3.0.1-1                            all          Puppet module for OpenIO SDS solution

Yes, I’m on ubuntu on my swift node and centos on my openio nodes just like what I mentioned my cluster in previous post.


#27

Hi,
You have a very old version of our product, when did you install it?
Here is a uptodate node (ubuntu) for example:

ii  openio-asn1c                     0.9.27.1-1                                 amd64        Free, Open Source ASN.1 compiler
ii  openio-gridinit                  1.6-2                                      amd64        OpenIO gridinit daemon
ii  openio-sds                       4.1.11-1                                   amd64        OpenIO Cloud Storage Solution
ii  openio-sds-logrotate             1.3-1                                      all          OpenIO SDS logrotate configuration
ii  openio-sds-rsyslog               1.2-2                                      all          OpenIO SDS rsyslog configuration
ii  openio-sds-swift                 1.1.0-1                                    all          Swift Gateway for OpenIO SDS
ii  puppet-module-openio-gridinit    1.1.9-1                                    all          Puppet module for gridinit from OpenIO
ii  puppet-module-openio-openiosds   4.0.15-1                                   all          Puppet module for OpenIO SDS solution

Is it the same on your centos nodes ?


#28

I installed swift/s3 Gateway for ubuntu this week. i followed this page: http://docs.openio.io/install-swift-s3-ubuntu/installation.html

On CentOS nodes, it looks not old.

[root@io-x03 ~]# rpm -qa | grep openio
openio-sds-release-17.04-1.el.noarch
openio-sds-rsyslog-1.2-1.el7.oio.noarch
openio-gridinit-1.6-2.el7.oio.x86_64
puppet-openio-sds-4.0.15-1.el7.oio.noarch
openio-sds-common-4.1.11-1.el7.oio.x86_64
openio-sds-logrotate-1.7-1.el7.oio.noarch
openio-sds-mod-httpd-4.1.11-1.el7.oio.x86_64
openio-asn1c-0.9.27-1.el7.oio.x86_64
openio-gridinit-utils-1.6-2.el7.oio.x86_64
openio-sds-server-4.1.11-1.el7.oio.x86_64

#29
root@gateway-2:~# cat /etc/apt/sources.list.d/openio-sds.list 
deb http://mirror.openio.io/pub/repo/openio/sds/16.10/Ubuntu/ xenial/

#30

ok i wonder how it’s possible but you have been redirected on a old documentation page.
I’m sorry for this.
Here is documentation:
http://docs.openio.io/17.04/install-swift-s3-ubuntu/installation.html


#31

That’s nice you give a new link. I will try it and give you feedback.


#32

How about OpenStack repository? Does it mean no step here?


#33

For your ubuntu node, you just have to modify the repository and:
apt clean, apt-get update; apt-get install openio-sds openio-sds-logrotate openio-sds-swift puppet-module-openio-openiosds for updating your packages.

Just check your puppet file between 16.10 and 17.04 before applying it.

For the openstack repository, applying the manifest should add it.


#34

Nice to have this line. Now the versions look new.

root@gateway-2:~# dpkg -l | grep openio
ii  openio-asn1c                       0.9.27.1-1                         amd64        Free, Open Source ASN.1 compiler
ii  openio-gridinit                    1.6-2                              amd64        OpenIO gridinit daemon
ii  openio-sds                         4.1.11-1                           amd64        OpenIO Cloud Storage Solution
ii  openio-sds-logrotate               1.3-1                              all          OpenIO SDS logrotate configuration
ii  openio-sds-rsyslog                 1.2-2                              all          OpenIO SDS rsyslog configuration
ii  openio-sds-swift                   1.1.0-1                            all          Swift Gateway for OpenIO SDS
ii  puppet-module-openio-gridinit      1.1.9-1                            all          Puppet module for gridinit from OpenIO
ii  puppet-module-openio-openiosds     4.0.15-1                           all          Puppet module for OpenIO SDS solution`

And continue

root@gateway-2:~# puppet module install openstack-keystone
Notice: Preparing to install into /etc/puppet/modules ...
Notice: Module openstack-keystone 8.0.1 is already installed.
root@gateway-2:~# 
root@gateway-2:~# sed -i "s@'upstart'@undef@" /etc/puppet/modules/keystone/manifests/params.pp
root@gateway-2:~# 

But here

root@gateway-2:~# puppet apply --no-stringify_facts ~/openio-gw.pp 
Error: Invalid parameter admin_password on Class[Keystone] at /root/openio-gw.pp:13 on node gateway-2
Error: Invalid parameter admin_password on Class[Keystone] at /root/openio-gw.pp:13 on node gateway-2

Here is my openio.pp

root@gateway-2:~# cat openio-gw.pp 
$openio_proxy_url = "http://192.168.2.91:6006"
$admin_token = 'KEYSTONE_ADMIN_UUID'
$swift_passwd = 'SWIFT_PASS'
$admin_passwd = 'ADMIN_PASS'
$demo_passwd = 'DEMO_PASS'
$region = 'RegionOne'

# Deploy Openstack Keystone
class { 'keystone':
  admin_token         => $admin_token,
  admin_password      => $admin_passwd,
  database_connection => 'sqlite:////var/lib/keystone/keystone.db',
}

# Use Apache httpd service with mod_wsgi
class { 'keystone::wsgi::apache':
  ssl => false,
}

# Adds the admin credential to keystone.
class { 'keystone::roles::admin':
  email               => 'test@openio.io',
  password            => $admin_passwd,
  admin               => 'admin',
  admin_tenant        => 'admin',
  admin_user_domain   => 'admin',
  admin_project_domain => 'admin',
}

# Installs the service user endpoint.
class { 'keystone::endpoint':
  public_url   => "http://${ipaddress}:5000",
  admin_url    => "http://${ipaddress}:5000",
  internal_url => "http://${ipaddress}:35357",
  region       => $region,
}

# Openstack Swift service credentials
keystone_user { 'swift':
  ensure   => present,
  enabled  => true,
  password => $swift_passwd,
}
keystone_user_role { 'swift@services':
  roles  => ['admin'],
  ensure => present
}
keystone_service { 'openio-swift':
  ensure      => present,
  type        => 'object-store',
  description => 'OpenIO SDS swift proxy',
}
keystone_endpoint { 'localhost-1/openio-swift':
  ensure       => present,
  type         => 'object-store',
  public_url   => "http://${ipaddress}:6007/v1.0/AUTH_%(tenant_id)s",
  admin_url    => "http://${ipaddress}:6007/v1.0/AUTH_%(tenant_id)s",
  internal_url => "http://${ipaddress}:6007/v1.0/AUTH_%(tenant_id)s",
}

# Demo account credentials
keystone_tenant { 'demo':
  ensure  => present,
  enabled => true,
}
keystone_user { 'demo':
  ensure  => present,
  enabled => true,
  password => $demo_passwd,
}
keystone_role { '_member_':
  ensure => present,
}
keystone_user_role { 'demo@demo':
  roles  => ['admin','_member_'],
  ensure => present
}

# Deploy OpenIO Swift/S3 gateway
class {'openiosds':}
openiosds::namespace {'OPENIO':
    ns => 'OPENIO',
}
openiosds::oioswift {'oioswift-0':
  ns                 => 'OPENIO',
  ipaddress          => '0.0.0.0',
  sds_proxy_url      => $openio_proxy_url,
  admin_password     => $swift_passwd,
  memcache_servers   => "${ipaddress}:6019",
  region_name        => $region,
  middleware_swift3 => {'location' => $region},
}
openiosds::memcached {'memcached-0':
  ns => 'OPENIO',
}

#35

Could you remove admin_passwd in the keystone class please.
class { 'keystone': admin_token => $admin_token, database_connection => 'sqlite:////var/lib/keystone/keystone.db', }


#36

Puppet apply works. But

root@gateway-2:~# gridinit_cmd status @oioswift
KEY                STATUS      PID GROUP
OPENIO-oioswift-0  DOWN         -1 OPENIO,oioswift,oioswift-0
root@gateway-2:~# 
root@gateway-2:~# 
root@gateway-2:~# gridinit_cmd restart @oioswift
DONE    	OPENIO-oioswift-0	Success
root@gateway-2:~# 
root@gateway-2:~# gridinit_cmd status @oioswift
KEY                STATUS      PID GROUP
OPENIO-oioswift-0  BROKEN       -1 OPENIO,oioswift,oioswift-0

from /var/log/syslog

Dec 28 19:26:21 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log INF gridinit Restart [OPENIO-oioswift-0]
Dec 28 19:26:21 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 1854
Dec 28 19:26:21 gateway-2 liberasurecode[1854]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 28 19:26:21 gateway-2 liberasurecode[1854]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 28 19:26:21 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 1857
Dec 28 19:26:21 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 28 19:26:21 gateway-2 liberasurecode[1857]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 28 19:26:21 gateway-2 liberasurecode[1857]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 28 19:26:24 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 1860
Dec 28 19:26:24 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 28 19:26:24 gateway-2 liberasurecode[1860]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 28 19:26:24 gateway-2 liberasurecode[1860]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 28 19:26:26 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 1863
Dec 28 19:26:26 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 28 19:26:26 gateway-2 liberasurecode[1863]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 28 19:26:26 gateway-2 liberasurecode[1863]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 28 19:26:26 gateway-2 /usr/bin/gridinit[1066]: 1066 E891 log ERR gridinit Process alert: Process broken [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf

#37

I want to point out that puppet apply threw not only admin_password error in keystone class but also a second error in openiosds::oioswift {'oioswift-0':

root@gateway-2:~# puppet apply --no-stringify_facts ~/openio-gw.pp 
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Warning: Scope(Class[Keystone]): Keystone under Eventlet has been deprecated during the Kilo cycle. Support for deploying under eventlet will be dropped as of the M-release of OpenStack.
Error: Invalid parameter admin_password on Openiosds::Oioswift[oioswift-0] at /root/openio-gw.pp:92 on node gateway-2
Error: Invalid parameter admin_password on Openiosds::Oioswift[oioswift-0] at /root/openio-gw.pp:92 on node gateway-2

I took a try and therefore changed admin_password to password and then puppet apply was able to continue until finished.


#38

From /var/log/syslog, the puppet apply above actually met some errors:

root@gateway-2:~# tail -f /var/log/syslog 
\Dec 29 10:17:51 gateway-2 systemd[1]: Started Session 3 of user root.
Dec 29 10:19:44 gateway-2 puppet-agent[1666]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)
Dec 29 10:21:44 gateway-2 puppet-agent[1666]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)
Dec 29 10:23:25 gateway-2 systemd[1]: Starting Cleanup of Temporary Directories...
Dec 29 10:23:25 gateway-2 systemd-tmpfiles[2528]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
Dec 29 10:23:25 gateway-2 systemd[1]: Started Cleanup of Temporary Directories.
Dec 29 10:23:44 gateway-2 puppet-agent[1666]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)
Dec 29 10:25:44 gateway-2 puppet-agent[1666]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)
Dec 29 10:26:53 gateway-2 systemd[1]: Started Session 4 of user root.
Dec 29 10:27:45 gateway-2 puppet-agent[1666]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)

Dec 29 10:31:45 gateway-2 puppet-agent[1666]: message repeated 2 times: [ Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known)]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit reconfigure : ignoring section [Default]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit reconfigure : managing service section [Service.OPENIO-oioswift-0]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Loaded service file [/etc/gridinit.d/OPENIO-oioswift-0]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit reconfigure : managing service section [Service.OPENIO-memcached-0]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Loaded service file [/etc/gridinit.d/OPENIO-memcached-0]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Configuration loaded from [/etc/gridinit.conf]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Already started [OPENIO-memcached-0]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Already started [OPENIO-oioswift-0]
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Starting service [OPENIO-oioswift-0] with pid 2899
Dec 29 10:32:28 gateway-2 systemd[1]: Starting OpenStack Identity service...
Dec 29 10:32:28 gateway-2 systemd[1]: Started OpenStack Identity service.
Dec 29 10:32:28 gateway-2 liberasurecode[2899]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 10:32:28 gateway-2 liberasurecode[2899]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 10:32:28 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Starting service [OPENIO-oioswift-0] with pid 2945
Dec 29 10:32:29 gateway-2 liberasurecode[2945]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 10:32:29 gateway-2 liberasurecode[2945]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.575 2917 WARNING keystone.assignment.core [-] Deprecated: Use of the identity driver config to automatically configure the same assignment driver has been deprecated, in the "O" release, the assignment driver will need to be expicitly configured if different than the default (SQL).
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.709 2917 WARNING oslo_config.cfg [-] Option "admin_workers" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.709 2917 WARNING oslo_config.cfg [-] Option "public_workers" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.710 2917 WARNING oslo_config.cfg [-] Option "admin_bind_host" from group "DEFAULT" is deprecated. Use option "admin_bind_host" from group "eventlet_server".
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.710 2917 WARNING oslo_config.cfg [-] Option "admin_bind_host" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.711 2917 WARNING oslo_config.cfg [-] Option "admin_port" from group "DEFAULT" is deprecated. Use option "admin_port" from group "eventlet_server".
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.711 2917 WARNING oslo_config.cfg [-] Option "admin_port" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.770 2917 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.810 2917 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.830 2917 WARNING oslo_config.cfg [-] Option "enable" from group "ssl" is deprecated. Use option "enable" from group "eventlet_server_ssl".
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.830 2917 WARNING oslo_config.cfg [-] Option "enable" from group "eventlet_server_ssl" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.830 2917 WARNING oslo_config.cfg [-] Option "public_bind_host" from group "DEFAULT" is deprecated. Use option "public_bind_host" from group "eventlet_server".
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.830 2917 WARNING oslo_config.cfg [-] Option "public_bind_host" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.831 2917 WARNING oslo_config.cfg [-] Option "public_port" from group "DEFAULT" is deprecated. Use option "public_port" from group "eventlet_server".
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.831 2917 WARNING oslo_config.cfg [-] Option "public_port" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.858 2917 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.900 2917 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.919 2917 WARNING root [-] Running keystone via eventlet is deprecated as of Kilo in favor of running in a WSGI server (e.g. mod_wsgi). Support for keystone under eventlet will be removed in the "M"-Release.
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.920 2917 ERROR keystone.common.environment.eventlet_server [-] Could not bind to 0.0.0.0:35357
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root [-] Failed to start the admin server
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root Traceback (most recent call last):
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root   File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 88, in serve
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root     server.launch_with(launcher)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root   File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 54, in launch_with
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root     self.server.listen()
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root   File "/usr/lib/python2.7/dist-packages/keystone/common/environment/eventlet_server.py", line 110, in listen
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root     backlog=backlog)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root   File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 44, in listen
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root     sock.listen(backlog)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root   File "/usr/lib/python2.7/socket.py", line 228, in meth
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root     return getattr(self._sock,name)(*args)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root error: [Errno 98] Address already in use
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.921 2917 ERROR root
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 CRITICAL keystone [-] error: [Errno 98] Address already in use
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone Traceback (most recent call last):
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/bin/keystone-all", line 10, in <module>
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     sys.exit(main())
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/dist-packages/keystone/cmd/all.py", line 39, in main
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     eventlet_server.run(possible_topdir)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 156, in run
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     serve(*servers)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 88, in serve
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     server.launch_with(launcher)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/dist-packages/keystone/server/eventlet.py", line 54, in launch_with
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     self.server.listen()
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/dist-packages/keystone/common/environment/eventlet_server.py", line 110, in listen
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     backlog=backlog)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 44, in listen
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     sock.listen(backlog)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone   File "/usr/lib/python2.7/socket.py", line 228, in meth
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone     return getattr(self._sock,name)(*args)
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone error: [Errno 98] Address already in use
Dec 29 10:32:29 gateway-2 keystone[2917]: 2017-12-29 10:32:29.922 2917 ERROR keystone
Dec 29 10:32:29 gateway-2 systemd[1]: keystone.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 10:32:29 gateway-2 systemd[1]: keystone.service: Unit entered failed state.
Dec 29 10:32:29 gateway-2 systemd[1]: keystone.service: Failed with result 'exit-code'.
Dec 29 10:32:30 gateway-2 systemd[1]: keystone.service: Service hold-off time over, scheduling restart.
Dec 29 10:32:30 gateway-2 systemd[1]: Stopped OpenStack Identity service.
Dec 29 10:32:30 gateway-2 systemd[1]: Starting OpenStack Identity service...
Dec 29 10:32:30 gateway-2 systemd[1]: Started OpenStack Identity service.
Dec 29 10:32:31 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log INF gridinit Starting service [OPENIO-oioswift-0] with pid 2983
Dec 29 10:32:31 gateway-2 /usr/bin/gridinit[1117]: 1117 5A3C log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 29 10:32:31 gateway-2 liberasurecode[2983]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 10:32:31 gateway-2 liberasurecode[2983]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.659 2970 WARNING keystone.assignment.core [-] Deprecated: Use of the identity driver config to automatically configure the same assignment driver has been deprecated, in the "O" release, the assignment driver will need to be expicitly configured if different than the default (SQL).
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.741 2970 WARNING oslo_config.cfg [-] Option "admin_workers" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.742 2970 WARNING oslo_config.cfg [-] Option "public_workers" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.742 2970 WARNING oslo_config.cfg [-] Option "admin_bind_host" from group "DEFAULT" is deprecated. Use option "admin_bind_host" from group "eventlet_server".
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.742 2970 WARNING oslo_config.cfg [-] Option "admin_bind_host" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.742 2970 WARNING oslo_config.cfg [-] Option "admin_port" from group "DEFAULT" is deprecated. Use option "admin_port" from group "eventlet_server".
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.743 2970 WARNING oslo_config.cfg [-] Option "admin_port" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.835 2970 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.912 2970 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.934 2970 WARNING oslo_config.cfg [-] Option "enable" from group "ssl" is deprecated. Use option "enable" from group "eventlet_server_ssl".
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.934 2970 WARNING oslo_config.cfg [-] Option "enable" from group "eventlet_server_ssl" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.934 2970 WARNING oslo_config.cfg [-] Option "public_bind_host" from group "DEFAULT" is deprecated. Use option "public_bind_host" from group "eventlet_server".
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.935 2970 WARNING oslo_config.cfg [-] Option "public_bind_host" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.935 2970 WARNING oslo_config.cfg [-] Option "public_port" from group "DEFAULT" is deprecated. Use option "public_port" from group "eventlet_server".
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.935 2970 WARNING oslo_config.cfg [-] Option "public_port" from group "eventlet_server" is deprecated for removal.  Its value may be silently ignored in the future.
Dec 29 10:32:31 gateway-2 keystone[2970]: 2017-12-29 10:32:31.965 2970 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:32 gateway-2 keystone[2970]: 2017-12-29 10:32:32.010 2970 WARNING keystone.middleware.core [-] The admin_token_auth middleware presents a security risk and should be removed from the [pipeline:api_v3], [pipeline:admin_api], and [pipeline:public_api] sections of your paste ini file.
Dec 29 10:32:32 gateway-2 keystone[2970]: 2017-12-29 10:32:32.032 2970 WARNING root [-] Running keystone via eventlet is deprecated as of Kilo in favor of running in a WSGI server (e.g. mod_wsgi). Support for keystone under eventlet will be removed in the "M"-Release.
Dec 29 10:32:32 gateway-2 keystone[2970]: 2017-12-29 10:32:32.033 2970 ERROR keystone.common.environment.eventlet_server [-] Could not bind to 0.0.0.0:35357
Dec 29 10:32:32 gateway-2 keystone[2970]: 2017-12-29 10:32:32.034 2970 ERROR root [-] Failed to start the admin server
Dec 29 10:32:32 gateway-2 keystone[2970]: 2017-12-29 10:32:32.034 2970 ERROR root Traceback (most recent call last):

#39

I re-installed everything on Gateway node (ubuntu 16.04) . This time there’s no such errors as in last post except some warnings about deprecated keystone. I checked keystone service and it looks ok as it worked two days ago.

root@gateway-2:~# ps -ef | grep keystone
keystone 12433     1  1 10:55 ?        00:00:09 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12502 12433  0 10:55 ?        00:00:01 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12503 12433  0 10:55 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12504 12433  0 10:55 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12505 12433  0 10:55 ?        00:00:01 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12506 12433  0 10:55 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12507 12433  0 10:55 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12510 12433  0 10:55 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log
keystone 12511 12433  0 10:55 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file=/etc/keystone/keystone.conf --log-file=/var/log/keystone/keystone.log

But can’t see swift and oioswift still broken.

root@gateway-2:~# gridinit_cmd status @oioswift
KEY                STATUS      PID GROUP
OPENIO-oioswift-0  BROKEN       -1 OPENIO,oioswift,oioswift-0

#40

Some warnings in keystone.log:

2017-12-29 10:56:03.909 12502 WARNING keystone.common.controller [req-5e32578f-179b-45c6-9024-b672292a68f2 - - - - -] RBAC: Bypassing authorization

2017-12-29 10:56:14.334 12504 WARNING keystone.common.controller [req-c3917d04-0428-45f4-a96a-53d5b08a6c15 - - - - -] RBAC: Bypassing authorization
2017-12-29 10:56:14.341 12504 WARNING keystone.common.wsgi [req-c3917d04-0428-45f4-a96a-53d5b08a6c15 - - - - -] Could not find domain: Default
2017-12-29 10:56:14.343 12504 INFO eventlet.wsgi.server [req-c3917d04-0428-45f4-a96a-53d5b08a6c15 - - - - -] 127.0.0.1 - - [29/Dec/2017 10:56:14] "GET /v3/domains/Default HTTP/1.1" 404 340 0.013026
2017-12-29 10:56:14.354 12504 INFO keystone.common.wsgi [req-e4ff6fd3-6bf9-4372-92ea-eff90005a8f2 - - - - -] GET http://127.0.0.1:35357/v3/domains?name=Default
2017-12-29 10:56:14.355 12504 WARNING keystone.common.controller [req-e4ff6fd3-6bf9-4372-92ea-eff90005a8f2 - - - - -] RBAC: Bypassing authorization
2017-12-29 10:56:14.363 12504 INFO eventlet.wsgi.server [req-e4ff6fd3-6bf9-4372-92ea-eff90005a8f2 - - - - -] 127.0.0.1 - - [29/Dec/2017 10:56:14] "GET /v3/domains?name=Default HTTP/1.1" 200 513 0.011808
2017-12-29 10:56:14.367 12504 INFO keystone.common.wsgi [req-6387a41c-8ba3-4ee3-a7a6-3298f504e780 - - - - -] GET http://127.0.0.1:35357/v3/users?domain_id=default&name=swift


2017-12-29 10:56:16.522 12505 WARNING keystone.common.controller [req-6e41d115-b635-40ce-a034-079eaa4ae6ca - - - - -] RBAC: Bypassing authorization
2017-12-29 10:56:16.527 12505 WARNING keystone.common.wsgi [req-6e41d115-b635-40ce-a034-079eaa4ae6ca - - - - -] Could not find role: admin
2017-12-29 10:56:16.529 12505 INFO eventlet.wsgi.server [req-6e41d115-b635-40ce-a034-079eaa4ae6ca - - - - -] 127.0.0.1 - - [29/Dec/2017 10:56:16] "GET /v3/roles/admin HTTP/1.1" 404 336 0.019248
2017-12-29 10:56:16.547 12505 INFO keystone.common.wsgi [req-db09364b-1411-474f-95ea-15b3f977eddd - - - - -] GET http://127.0.0.1:35357/v3/roles?name=admin
2017-12-29 10:56:16.548 12505 WARNING keystone.common.controller [req-db09364b-1411-474f-95ea-15b3f977eddd - - - - -] RBAC: Bypassing authorization
2017-12-29 10:56:16.555 12505 INFO eventlet.wsgi.server [req-db09364b-1411-474f-95ea-15b3f977eddd - - - - -] 127.0.0.1 - - [29/Dec/2017 10:56:16] "GET /v3/roles?name=admin HTTP/1.1" 200 518 0.010436

#41

I ran gridinit_cmd restart @oioswift and got info from syslog:

Dec 29 11:22:41 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log INF gridinit Restart [OPENIO-oioswift-0]
Dec 29 11:22:41 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 13044
Dec 29 11:22:42 gateway-2 liberasurecode[13044]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 11:22:42 gateway-2 liberasurecode[13044]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 11:22:42 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 13047
Dec 29 11:22:42 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 29 11:22:42 gateway-2 liberasurecode[13047]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 11:22:42 gateway-2 liberasurecode[13047]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 11:22:45 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 13050
Dec 29 11:22:45 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 29 11:22:45 gateway-2 liberasurecode[13050]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 11:22:45 gateway-2 liberasurecode[13050]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 11:22:47 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log INF gridinit Starting service [OPENIO-oioswift-0] with pid 13053
Dec 29 11:22:47 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log WRN gridinit Process alert: Process restarted [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf
Dec 29 11:22:47 gateway-2 liberasurecode[13053]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory
Dec 29 11:22:47 gateway-2 liberasurecode[13053]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory
Dec 29 11:22:47 gateway-2 /usr/bin/gridinit[10586]: 10586 A939 log ERR gridinit Process alert: Process broken [OPENIO-oioswift-0] /usr/bin/swift-proxy-server  /etc/oio/sds/OPENIO/oioswift-0/proxy-server.conf

can’t see swift-proxy up:

root@gateway-2:~# ps -ef | grep swift
root     13062  1872  0 11:25 pts/1    00:00:00 grep --color=auto swift