https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14354 Lustre <p>Our customer's log data on IML server is around the 200 GB (database only) . We are waiting about 20-25 minutes for fetching logs from web ui, sometimes the page gives timeout due to heavy load of log entries. We need to delete old logs from customer'S IML. Do you have and document/procedure for deleting old logs?</p> LU-14354

Slow response fetching logs via IML

Bug Minor Resolved Fixed Will Johnson Ozgur Dagli Thu, 21 Jan 2021 15:49:03 +0000 Sat, 17 Apr 2021 12:44:10 +0000 Sat, 17 Apr 2021 12:44:10 +0000 0 5 <p>Will</p> <p>Can you please assist?</p> <p>Thanks</p> <p>Peter</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>I'm not sure which version of IML you are running but IML comes with a logrotate.cfg to configure logging and how much space should be used. In addition to this, IML records log entries from each of your servers in the database<a href="https://github.com/whamcloud/integrated-manager-for-lustre/blob/wasm-components/chroma_core/services/syslog/__init__.py)." class="external-link" target="_blank" rel="nofollow noopener">. </a>When too many log entries have accumulated in the database IML should automatically archive these entries and store them in the /var/log/chroma/db_log directory. The point at which this is done depends on the following two values: <br/> DBLOG_HW<br/> DBLOG_LW</p> <p>These values are defined under /usr/share/chroma-manager/settings.py. If you need to change these settings you can create a local_settings.py under /usr/share/chroma-manager and set the values accordingly. It's probably worth restarting IML after adjusting these values such that IML will initialize with these new values and purge old data as needed.</p> <p>You should also be able to adjust log sizes and rotation in the logrotate files under /etc/logrotate.d/* and it should not affect IML.</p> <p>Regards,</p> <p>Will Johnson</p> <p>Hello I have done config that you send to us.</p> <p> </p> <p> </p> <p>I have changed settings in /usr/share/chroma-manager/settings.py with:</p> <p> </p> <p> </p> <p>DBLOG_HW = 64000</p> <p>DBLOG_LW = 58000</p> <p> </p> <p>Also added these settings to</p> <p> </p> <p>Local_settings.py</p> <p> </p> <p>After that I have invoked</p> <p> </p> <p>“chroma-config restart”</p> <p> </p> <p>Nothing happens. Postgresql database size remain same: 178 GB</p> <p> </p> <p>Also I have tried to go vacuum analyze on database, it took long time ang gives error after 40 hours</p> <p> </p> <p> </p> <p>How can I re-initialize database and log files?</p> <p> </p> <p>NoteÇ After the backup I can purge any log and statistics. I’m sending the files as attachment.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p>Regards.</p> <p>Hello again. </p> <p> </p> <p>Wa have waited long for to do necessary steps. Due to heavy workload of our customer, its was not possible to proceed that issue.</p> <p> </p> <p>Now this is the only problem on customer which waiting for resulotion. Could you  speed up the response time?</p> <p> </p> <p>Best regards</p> <p> </p> <p>IML version is:</p> <p> </p> <p>Intel®️ Manager for Lustre IML* software 4.0.3.0</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>There are three types of tables you can look at clearing out from your database after you've made a backup:</p> <p>1. chroma_core_logmessage*<br/> 2. chroma_core_series*<br/> 3. chroma_core_sample_*</p> <p>These tables will hold log and metric data and possibly some other things as well. I would start by trying to clear out the logmessage tables first and see what that brings it down to.</p> <p>Regards,</p> <p>Will</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>You might find this command helpful as well. This will export the database without the messages:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> su - postgres -c <span class="code-quote">"pg_dump -U chroma -F p -w -T <span class="code-quote">'chroma_core_series*'</span> -T <span class="code-quote">'chroma_core_sample*'</span> -T <span class="code-quote">'chroma_core_logmessage*'</span> -f /tmp/chromadb_backup_xxx.sql"</span> </pre> </div></div> <p>What I would recommend is exporting your data with this command, and then importing this database on a separate test system to make sure you are able to load it without any problems with the same version of IML.</p> <p>Here is a script you can use to load the database onto the test system:</p> <p><a href="https://whamcloud.github.io/Support/docs/support/scripts/import-customer-database.html" class="external-link" target="_blank" rel="nofollow noopener">https://whamcloud.github.io/Support/docs/support/scripts/import-customer-database.html</a></p> <p>Regards,</p> <p>Will</p> <p>Hello, I have backed-up with the procedure that you send.</p> <p> </p> <p>After that, I have installed a new "Intel®️ Manager for Lustre IML* software 4.0.3.0" to a fresh Centos 7.4. (<a href="https://github.com/intel-hpdd/intel-manager-for-lustre/releases/download/v4.0.3.0/iml-4.0.3.0.tar.gz)" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/intel-hpdd/intel-manager-for-lustre/releases/download/v4.0.3.0/iml-4.0.3.0.tar.gz)</a></p> <p> </p> <p>After install, it does not work in any way. I'm sending the diagdostic file. Could you asisst me; where did a do wrong?</p> <p> </p> <p>Regards.</p> <p> </p> <p>You can see the screenshot after install:</p> <p><span class="image-wrap" style=""><a id="37972_thumb" href="https://jira.whamcloud.com/secure/attachment/37972/37972_screenshot-1.png" title="screenshot-1.png" file-preview-type="image" file-preview-id="37972" file-preview-title="screenshot-1.png"><img src="https://jira.whamcloud.com/secure/thumbnail/37972/_thumb_37972.png" style="border: 0px solid black" role="presentation"/></a></span></p> <p>Any updates?</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a></p> <p>The installation was not successful. According to the install logs, it looks like it installed a later version of python-django from epel:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [15/Mar/2021:12:40:40] DEBUG 0.017776: Error: Package: chroma-manager-4.0.3.0-5002.el7.x86_64 (chroma-manager) [15/Mar/2021:12:40:40] DEBUG 0.000163: Requires: Django &lt; 1.5 [15/Mar/2021:12:40:40] DEBUG 0.000070: Available: python-django-1.4.5-3.wc2.el7.centos.noarch (managerforlustre-manager-<span class="code-keyword">for</span>-lustre) [15/Mar/2021:12:40:40] DEBUG 0.000051: Django = 1.4.5-3.wc2.el7.centos [15/Mar/2021:12:40:40] DEBUG 0.000060: Available: python2-django-1.4.5-4.wc1.el7.centos.noarch (managerforlustre-manager-<span class="code-keyword">for</span>-lustre) [15/Mar/2021:12:40:40] DEBUG 0.000045: Django = 1.4.5-4.wc1.el7.centos [15/Mar/2021:12:40:40] DEBUG 0.000048: Available: python2-django-1.11.27-1.el7.noarch (epel) [15/Mar/2021:12:40:40] DEBUG 0.000042: Django = 1.11.27-1.el7 [15/Mar/2021:12:40:40] DEBUG 0.000047: Available: python2-django16-1.6.11.7-5.el7.noarch (epel) [15/Mar/2021:12:40:40] DEBUG 0.000041: Django = 1.6.11.7-5.el7 [15/Mar/2021:12:40:40] DEBUG 0.000049: Error: Package: chroma-manager-4.0.3.0-5002.el7.x86_64 (chroma-manager) [15/Mar/2021:12:40:40] DEBUG 0.000040: Requires: Django &lt; 1.5 [15/Mar/2021:12:40:40] DEBUG 0.000056: Available: python-django-1.4.5-3.wc2.el7.centos.noarch (managerforlustre-manager-<span class="code-keyword">for</span>-lustre) [15/Mar/2021:12:40:40] DEBUG 0.000052: Django = 1.4.5-3.wc2.el7.centos [15/Mar/2021:12:40:40] DEBUG 0.000053: Available: python2-django-1.4.5-4.wc1.el7.centos.noarch (managerforlustre-manager-<span class="code-keyword">for</span>-lustre) [15/Mar/2021:12:40:40] DEBUG 0.000048: Django = 1.4.5-4.wc1.el7.centos [15/Mar/2021:12:40:40] DEBUG 0.000046: Installing: python2-django-1.11.27-1.el7.noarch (epel) [15/Mar/2021:12:40:40] DEBUG 0.000037: Django = 1.11.27-1.el7 [15/Mar/2021:12:40:40] DEBUG 0.000075: Available: python2-django16-1.6.11.7-5.el7.noarch (epel) [15/Mar/2021:12:40:40] DEBUG 0.000042: Django = 1.6.11.7-5.el7 [15/Mar/2021:12:40:40] DEBUG 0.000046: You could <span class="code-keyword">try</span> using --skip-broken to work around the problem [15/Mar/2021:12:40:40] DEBUG 0.114907: You could <span class="code-keyword">try</span> running: rpm -Va --nofiles --nodigest [15/Mar/2021:12:40:40] ERROR The <span class="code-keyword">package</span> installation failed. Please contact support with details from /<span class="code-keyword">var</span>/log/chroma/install.log. [15/Mar/2021:12:40:40] DEBUG SystemExit: -1 </pre> </div></div> <p>The errors, such as the following, are a result of the above installation issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [2021-03-15 09:51:24,739: ERROR/service_thread] Exception in main loop. backtrace: Traceback (most recent call last): File <span class="code-quote">"/usr/share/chroma-manager/chroma_core/services/__init__.py"</span>, line 75, in run self.service.run() File <span class="code-quote">"/usr/share/chroma-manager/chroma_core/services/http_agent/queues.py"</span>, line 96, in run self._queue.serve(self.on_message) File <span class="code-quote">"/usr/share/chroma-manager/chroma_core/services/queue.py"</span>, line 67, in serve message = q.get(timeout = 1) File <span class="code-quote">"/usr/lib/python2.7/site-packages/kombu/simple.py"</span>, line 61, in get self.channel.connection.client.drain_events(timeout=remaining) File <span class="code-quote">"/usr/lib/python2.7/site-packages/kombu/connection.py"</span>, line 301, in drain_events <span class="code-keyword">return</span> self.transport.drain_events(self.connection, **kwargs) File <span class="code-quote">"/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py"</span>, line 103, in drain_events <span class="code-keyword">return</span> connection.drain_events(**kwargs) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/connection.py"</span>, line 500, in drain_events <span class="code-keyword">while</span> not self.blocking_read(timeout): File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/connection.py"</span>, line 506, in blocking_read <span class="code-keyword">return</span> self.on_inbound_frame(frame) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/method_framing.py"</span>, line 55, in on_frame callback(channel, method_sig, buf, None) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/connection.py"</span>, line 510, in on_inbound_method method_sig, payload, content, File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py"</span>, line 126, in dispatch_method listener(*args) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/connection.py"</span>, line 637, in _on_close self._x_close_ok() File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/connection.py"</span>, line 652, in _x_close_ok self.send_method(spec.Connection.CloseOk, callback=self._on_close_ok) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py"</span>, line 51, in send_method conn.frame_writer(1, self.channel_id, sig, args, content) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/method_framing.py"</span>, line 172, in write_frame write(view[:offset]) File <span class="code-quote">"/usr/lib/python2.7/site-packages/amqp/transport.py"</span>, line 282, in write self._write(s) File <span class="code-quote">"/usr/lib64/python2.7/site-packages/gevent/socket.py"</span>, line 458, in sendall data_sent += self.send(_get_memory(data, data_sent), flags) File <span class="code-quote">"/usr/lib64/python2.7/site-packages/gevent/socket.py"</span>, line 435, in send <span class="code-keyword">return</span> sock.send(data, flags) </pre> </div></div> <p>This is an older version of IML and since epel doesn't keep older versions of its packages around, you will probably need to install the required version of python-django first and disable it from epel during the installation. You should be able to do this by configuring yum. You will need to remove the installation (and the python-django package) and make sure that the correct version of python-django is installed before installing again or make sure that when the installation begins that it does not pull this package from epel.</p> <p>Regards,</p> <p>Will</p> <p>I setup a new server then I have installed without problems or errors. But it fails again with same error when i try to access web interface:</p> <p> </p> <p><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/37989/37989_image-2021-03-17-14-57-12-044.png" height="134" width="550" style="border: 0px solid black" /></span></p> <p> </p> <p>I'm sending current diagnostic file: "sosreport-iml-2021-03-17-vosbpsm.tar.xz"</p> <p> </p> <p>Can you check the issuse again?</p> <p> </p> <p>Regards </p> <p>Ozgur</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>Looking at the logs it looks like the installation is successful this time and the service logs are not showing anything relevant. Can you go into the "/var/log/chroma" directory and tail the logs for any exceptions that occur when you load the page? The logs in the package you sent aren't showing any exceptions this time. It's also worth checking the IML services to see if any of them are starting and stopping continuously. </p> <p>One other thing I can think of is to take all of the mentioned rpms installed in the install log and compare their versions to what you have on your production machine. With this being an older installation, it's possible epel installed a newer version of an rpm that could be causing an issue. </p> <p>Regards,</p> <p>Will</p> <p>I have regerated issue and gives following logs and errors on gunicorn-access.log: (I have traced whole directory with "tail -f", only  gunicorn-access.log is updating.)</p> <p> </p> <p>192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:29&#93;</span> "GET /api/user/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:29&#93;</span> "GET /api/storage_resource_class/?plugin_internal=false&amp;limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:30&#93;</span> "GET /api/filesystem/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:30&#93;</span> "GET /api/target/?limit=0&amp;dehydrate__volume=false HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:30&#93;</span> "GET /api/session/ HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:30&#93;</span> "GET /api/user/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:30&#93;</span> "GET /api/storage_resource_class/?plugin_internal=false&amp;limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:31&#93;</span> "GET /api/filesystem/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:31&#93;</span> "GET /api/target/?limit=0&amp;dehydrate__volume=false HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:31&#93;</span> "GET /api/session/ HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:31&#93;</span> "GET /api/user/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:31&#93;</span> "GET /api/storage_resource_class/?plugin_internal=false&amp;limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/session/ HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/filesystem/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/target/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/power_control_type/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/host/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/lnet_configuration/?limit=0&amp;dehydrate__host=false HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/alert/?limit=0&amp;active=true HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/server_profile/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/job/?limit=0&amp;state_<em>in=pending&amp;state</em>_in=tasked HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:48&#93;</span> "GET /api/session/ HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/alert/?limit=0&amp;active=true HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/server_profile/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/power_control_type/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/job/?limit=0&amp;state_<em>in=pending&amp;state</em>_in=tasked HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/host/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/session/ HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/storage_resource_class/?plugin_internal=false&amp;limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/user/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/session/ HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/storage_resource_class/?plugin_internal=false&amp;limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/user/?limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/filesystem/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/alert/?limit=0&amp;active=true HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/lnet_configuration/?limit=0&amp;dehydrate__host=false HTTP/1.0" 200 - "<del>" "</del>"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/alert/?active=true&amp;severity_<em>in=WARNING&amp;severity</em>_in=ERROR&amp;limit=0 HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/target/?limit=0&amp;dehydrate__volume=false HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/alert/?severity_<em>in=WARNING&amp;severity</em>_in=ERROR&amp;limit=0&amp;active=true HTTP/1.0" 200 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"<br/> 192.168.3.104 - - <span class="error">&#91;19/Mar/2021:11:38:49&#93;</span> "GET /api/target/?limit=0 HTTP/1.0" 200 - "<del>" "</del>"</p> <p> </p> <p>Also I have checked the services:</p> <p> </p> <p>iml-srcmap-reverse.socket loaded active running <br/> iml-supervisor-status.socket loaded active running </p> <p>chroma-supervisor.service loaded active running </p> <p> </p> <p>They are working without errors.</p> <p> </p> <p>Actualy We just need a IML server for our existing Lustre  cluster. do we need to use axactly the version of IML? Can we install newer version of IML with older version of Lustre?</p> <p> </p> <p>Any updates?</p> <p>I have get the list of installed rpms from production machine and trying to install with that packages. But it generates the same issue:</p> <p><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/38060/38060_image-2021-03-24-17-02-34-894.png" height="170" width="722" style="border: 0px solid black" /></span></p> <p>I'm working on it (may be some packegas differ). </p> <p>Is that possible to install newer version for that cluster? Are the newer versions backward compatible? Or i must to install exact version for that cluster?</p> <p> </p> <p>Regards.</p> <p> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>The lustre version is tied to IML if you are using a managed install. Unfortunately, trying to make an older version of IML like this work is going to be difficult due to workarounds, such as the django package being a later version on epel that we saw earlier. I have a couple of questions:</p> <ol> <li>Is this a monitored or managed install?</li> <li>Can you upgrade your Lustre filesystem and centos OS? </li> </ol> <p>Regards,</p> <p>Will</p> <p>Hello again;</p> <p> </p> <ol> <li>Is this a monitored or managed install? <ol> <li>Its a monitored install</li> </ol> </li> <li>Can you upgrade your Lustre filesystem and centos OS?  <ol> <li>Its not possible to upgrade FS due to heavy load and backup consideration.</li> </ol> </li> </ol> <p>There in only way; install IML 4.0.3 <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>...</p> <p> </p> <p>Thanks for your support. Please leave that case open.</p> <p> </p> <p>Best Regards</p> <p> </p> <p>Ozgur</p> <p>Good morning <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>In order to troubleshoot this further, I will need to investigate on a VM. I'll keep you posted.</p> <p>Regards,</p> <p>Will </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>I was able to narrow down the issue to the installed version of the gui. This version of IML is quite old and the installer manages to pull down the oldest version of the gui that is currently published, which is greater than the version that IML@4.0.3 needs. To get around this, we will need to bypass the current installed gui and instead replace some files in the "/usr/lib/iml-manager/iml-gui" folder. This shouldn't be hard to do, however. Here is a screenshot of 4.0.3 working on my VM after making the change:</p> <p><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/38120/38120_image-2021-03-29-15-52-45-496.png" style="border: 0px solid black" /></span><br/> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/38122/38122_gui.zip" title="gui.zip attached to LU-14354">gui.zip^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>To fix the install:<br/> 1. cd /usr/lib/iml-manager<br/> 2. mv iml-gui iml-gui.bk<br/> 3. mkdir iml-gui<br/> 4. cp gui.zip into /usr/lib/iml-manager/iml-gui<br/> 5. cd iml-gui<br/> 6. unzip gui.zip<br/> 7. chroma-config restart</p> <p>You should now be able to reload the page and it should load correctly.</p> <p>Regards,</p> <p>Will</p> <p>Hello again;</p> <p>Thanks for your reply. Installation is succesfull. But when i try to import database which exported with you commands:<br/> su - postgres -c "pg_dump -U chroma -F p -w -T 'chroma_core_series*' -T 'chroma_core_sample*' -T 'chroma_core_logmessage*' -f /tmp/chromadb_backup_xxx.sql"<br/> after export i created the .py script with following link:</p> <p><a href="https://whamcloud.github.io/Support/docs/support/scripts/import-customer-database.html" class="external-link" target="_blank" rel="nofollow noopener">https://whamcloud.github.io/Support/docs/support/scripts/import-customer-database.html</a></p> <p>then:</p> <p><span class="error">&#91;root@iml ~&#93;</span># ./import.py chromadb_backup_xxx.sql<br/> Importing chromadb_backup_xxx.sql<br/> Stopping chroma manager.<br/> Writing other db bits.<br/> Dropping chroma db.<br/> Creating chroma db.<br/> Importing Database.<br/> Error importing database.<br/> Importing other db bits.<br/> starting chroma manager<br/> Result:<br/> Stdout: Cannot start, database not configured</p> <p>Stderr:<br/> <span class="error">&#91;root@iml ~&#93;</span># chroma-config restart<br/> Stopping daemons<br/> Cannot start, database not configured</p> <p> </p> <p>i can not import customer data to new installation. I'm sending iml-diagnostic also. Do you have any suggestions?</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>Can you attach the exported database so I can try to import it locally?</p> <p>Regards,</p> <p>Will</p> <p>I have e-mailed backup file to you. </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>I was able to load your database without any issues. Can you please attach the exported database to this ticket for reference? Here are the manual commands I ran to import your database on a clean system:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 1. pg_dump -U chroma -F p -w -f /tmp/other-db-bits.sql -t <span class="code-quote">'chroma_core_series*'</span> -t <span class="code-quote">'chroma_core_sample_*'</span> -t <span class="code-quote">'chroma_core_logmessage*'</span> 2. chroma-config stop 3. su - postgres -c <span class="code-quote">"dropdb chroma"</span> 4. su - postgres -c <span class="code-quote">"createdb chroma"</span> 5. su - postgres -c <span class="code-quote">"psql chroma -f /tmp/chromadb_backup_xxx.sql"</span> 6. su - postgres -c <span class="code-quote">"psql chroma -f /tmp/other-db-bits.sql"</span> 7. chroma-config start </pre> </div></div> <p>Here is a screenshot after importing your database and starting IML:<br/> <span class="image-wrap" style=""><a id="38175_thumb" href="https://jira.whamcloud.com/secure/attachment/38175/38175_after-db-import.png" title="after-db-import.png" file-preview-type="image" file-preview-id="38175" file-preview-title="after-db-import.png"><img src="https://jira.whamcloud.com/secure/thumbnail/38175/_thumb_38175.png" style="border: 0px solid black" role="presentation"/></a></span> </p> <p>Try running those steps manually and let me know what happens.</p> <p>Regards,</p> <p>Will</p> <p>For this customer; distrubition of data files are restricted. I can not upload the backup file to here. Did you received the sql backup via e-mail? Did you import that file? I will try to import as you suggested.</p> <p>Thanks for your support.</p> <p>Regards.</p> <p> </p> <p>Ozgur</p> <p> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <blockquote> <p>For this customer; distrubition of data files are restricted. I can not upload the backup file to here. Did you received the sql backup via e-mail?</p></blockquote> <p>Yes, I did. No problem.</p> <blockquote> <p>Did you import that file?</p></blockquote> <p>Yes. The screenshot above shows that I loaded IML after importing the database and starting IML.</p> <p>Regards,</p> <p>Will</p> <p>Hello again;</p> <p> </p> <p>After i try to setup database on product machine (IML) with your recent method; service is not starting:</p> <p> </p> <p><span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/38233/38233_image-2021-04-07-10-29-19-585.png" height="175" width="706" style="border: 0px solid black" /></span></p> <p> </p> <p>Production machine is not working now. I'm sending sosreport and all logs on iml. Can you help to re-run the service again?</p> <p> </p> <p>attached files: </p> <p>sosreport.xz</p> <p>chroma.log.tar.gz</p> <p> </p> <p>Regards.</p> <p> </p> <p> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>Can we set up a screen share session?</p> <p>Regards,</p> <p>Will</p> <p>I have asked for remote session to customer. I will inform you about that.</p> <p> </p> <p>What is your available time for remote?</p> <p>Today 13:30 -  16:30 (GMT+3) </p> <p>or</p> <p>Tomorrow  10:00-12:00 / 14:00-16:30 (GMT+3) </p> <p> </p> <p>are availble times for remote connection. Customer network is protected, we can connect only with their Webex system. They can send a invitation for those gaps(above).</p> <p>How do we proceed? </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>Before we meet with the client I think we should have a screen session together to go through things in more detail. I would like to see your test system and how it is working after importing the database. Would you be available today? I am in the Eastern Standard Time timezone.</p> <p>Regards,</p> <p>Will</p> <p>Also, were you able to get this working on your test machine before trying it on the production machine?</p> <p>Yes it were working on test machine but; I have done many changes on that test machine after the failure on customer site, there is no backup,. So I have to install a new server for testing again. </p> <p> </p> <p>Is that posssible to connect directy to customer system? If you give me  exact avaible time-shift for you, I can arrange with customer. </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>Let's take a step back for a moment. Since this is a monitored filesystem, we can scrap the whole IML installation on both the manager and storage server nodes and then re-install IML without trying to load the database. Once you've installed IML on the manager node, you can then add the servers and it should pick everything up just fine. There are a couple of uninstall notes you should be aware of:</p> <p>1. Uninstall on the manager</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> chroma-config stop chkconfig --del chroma-supervisor yum clean all --enablerepo=* yum remove chroma* fence-agents* rm -rf /usr/lib/iml-* rm -rf /usr/share/chroma-manager/ rm -rf /usr/lib/python2.7/site-packages/chroma* rm -rf /<span class="code-keyword">var</span>/cache/yum/x86_64/7/chroma-manager </pre> </div></div> <p>I would also check for a "/var/lib/chroma" directory and delete any contents inside of it if it exists. </p> <p>2. Clean up the agents on each storage node:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # Stop and deregister the agent service chroma-agent stop /sbin/chkconfig --del chroma-agent # Cleanup yum remove -y chroma-agent chroma-agent-management iml_sos_plugin iml-device-scanner python2-iml-common* lustre-iokit lustre-osd-ldiskfs-mount lustre-osd-zfs-mount rm -rf /etc/yum.repos.d/Intel-Lustre-Agent.repo rm -rf /<span class="code-keyword">var</span>/lib/chroma/ rm -rf /<span class="code-keyword">var</span>/lib/iml/ rm -rf /etc/iml/ rm -rf /etc/yum.repos.d/Intel-Lustre-Agent.repo rm -rf /usr/lib/python2.7/site-packages/chroma_agent* </pre> </div></div> <p>After uninstalling on the manager and storage servers you should be able to re-install IML and the manager and then add the servers in monitored mode. </p> <p>Regards,</p> <p>Will</p> <p>Hello again,</p> <p> </p> <p>I can install 4.0.3 IML without any issues. After the install i try to add hosts with without doing "# Cleanup" part of host. I have added one OSS node with "force and override option". OSS host seemd added on IML. Can i continue without to do "# Cleanup"? Does that will cause any problems?</p> <p> </p> <p>Regards</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dagli" class="user-hover" rel="dagli">dagli</a>,</p> <p>Since it's a monitored install I think that should be fine. Try it and see if it starts working and if not, remove the nodes and try again after cleaning them up.</p> <p>Regards,</p> <p>Will</p> <p>Thank you very much. I have finished setup. We can close the case.</p> <p> </p> <p> </p> Development Rank 1|i01jvj: Rank (Obsolete) 9223372036854775807 Severity