https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15823 Lustre <p>Tried to upload test results to Maloo DB with my own account several times following the guide <a href="https://wiki.whamcloud.com/display/TEI/Test+results+format" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.whamcloud.com/display/TEI/Test+results+format</a>.<br/> Either with maloo_upload.sh or through the website <a href="https://testing.whamcloud.com/import_tasks/new" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks/new</a>. But can't see any upload in the upload queue <br/> <a href="https://testing.whamcloud.com/import_tasks?user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks?user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7</a>.</p> LU-15823

Maloo DB test result upload from external doesn't work

Bug Minor Resolved Fixed Lee Ochoa Xinliang Liu malooDB Thu, 5 May 2022 01:30:21 +0000 Mon, 6 Jun 2022 01:29:18 +0000 Tue, 10 May 2022 14:15:53 +0000 1 4 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=leonel8a" class="user-hover" rel="leonel8a">leonel8a</a>, could you take a look?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a>, could you please attach here the results tarball so I can investigate? Thanks.</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/43562/43562_upload-liuxl.tar.gz" title="upload-liuxl.tar.gz attached to LU-15823">upload-liuxl.tar.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>  Here is the tar file downloaded from <a href="https://testing.whamcloud.com/import_tasks/f1c8dce1-7f23-4d59-be53-1d3f23cae755/download" class="external-link" target="_blank" rel="nofollow noopener">here</a> and change </p> <p>'cumulative_result_id' and 'session_group_id' to different UUIDs.</p> <p>Hmm, not sure what issue you were having, I uploaded the tarball using <a href="https://testing.whamcloud.com/import_tasks/new" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks/new</a>, you can find it here <a href="https://testing.whamcloud.com/test_sessions/69b2f7d4-cf36-11ec-881a-07ebe308fd35" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/69b2f7d4-cf36-11ec-881a-07ebe308fd35</a> or by going to the import queue, of course this time under my name <a href="https://testing.whamcloud.com/import_tasks?user_id=5b41a8e0-84b5-11e4-b2c6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks?user_id=5b41a8e0-84b5-11e4-b2c6-5254006e85c2</a>.</p> <p>Hi Lee, thank you for your verifying.</p> <p>The new attached tar file <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/43576/43576_upload-for-test.tar.gz" title="upload-for-test.tar.gz attached to LU-15823">upload-for-test.tar.gz^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>which delete 'cumulative_result_id' and 'session_group_id'  should be uploadable every time.</p> <p>This time I can upload it by using <a href="https://testing.whamcloud.com/import_tasks/new" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks/new.</a>. I can see it in upload queue now: <a href="https://testing.whamcloud.com/import_tasks?user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks?user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7</a></p> <p>And I can upload it with below cmd too. The maloo_upload.sh script is downloaded from <a href="https://testing.whamcloud.com/import_tasks/new" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/import_tasks/new.</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">  ./maloo_upload.sh upload-<span class="code-keyword">for</span>-test.tar.gz</pre> </div></div> <p>But when I extract it to test-result-dir, I can not upload it successfully with below cmd.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ./maloo_upload.sh test-result-dir</pre> </div></div> <p>I found a bug on maloo_upload.sh, I can fix it with bellow change.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> -  tar cfz upload.tar.gz * | curl -F <span class="code-quote">"user_id=${MALOO_USER_ID}"</span> -F <span class="code-quote">"upload=@upload.tar.gz"</span> \ +  tar cfz upload.tar.gz * &amp;&amp; curl -F <span class="code-quote">"user_id=${MALOO_USER_ID}"</span> -F <span class="code-quote">"upload=@upload.tar.gz"</span> \ </pre> </div></div> <p>Ah yes, thank you for noticing, I'll fix that.</p> <p>Just FYI,</p> <p>I found that if the submission date of results.yml is too old then the result tar ball can't be uploaded.</p> <p>This can explain why I can't upload previous old test results.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=leonel8a" class="user-hover" rel="leonel8a">leonel8a</a>, another issue I found is that only Maloo defined job name can be display on <a href="https://testing.whamcloud.com/test_sessions." class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions.</a>  For example, for the upload-for-test.tar.gz test result, if I change the triggering_job_name: lustre-reviews to "lustre-reviews-1" in file results.yml, although the test result tar ball can be uploaded successfully but it can't  be display on <a href="https://testing.whamcloud.com/test_sessions." class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions. </a></p> <p>This makes me can't set my own job name.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a>, I'm not sure the above is correct. I found all of the tarballs you uploaded by simply filtering on your name, see the screenshot in the attachments above.</p> <p>Note that to prevent an explosion of job names and test groups we probably want to limit the amount of names used, I can help consolidate these if you want all the results under a certain job name and/or test group</p> <p>EDIT: Actually, for job you should probably use "custom", and for test group we can probably create one that is specific to you, let me know what you'd like to call it and I can change all the existing results, if you need more than one that's ok too, that way you can split your results into different test groups, just let me know what you'd like them to be</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=leonel8a" class="user-hover" rel="leonel8a">leonel8a</a>, thank you for your clarifying.</p> <p>I've tried the job name "custom", it doesn't work. If possible, please help to create an job name "linaro-lustre-daily-master".</p> <p>And regarding to the test group names, as you can see in the screen shot, it doesn't has any limitation, I can use my own defined names, right?</p> <p>I've used "custom" in the past without issues, just tried to import a copied tarball in my dev environment, changed the sessions group id and trigger job name to "custom" and it worked fine. Having said that, yes you can use "linaro-lustre-daily-master" if you'd like, the first imported set of results will create the job if it not there so feel free to go ahead, and regarding test group names also feel free to create your own they'll also be added if missing, but please keep the list short, we definitely don't want to create dozens of these.</p> <p>Just occurred to me, are you by any chance looking for your results in the "EXASCALER" project? That might explain why you are not seeing them, they're saved in the "LUSTRE" project (see screenshot above)</p> <p>I'd like to remove all of the test_xxx groups that were created as this is a production system, what can I rename them to? So far you've added</p> <p>test-xx-01</p> <p>test-xx-02</p> <p>test-xx-03</p> <p>xx-01</p> <p>xx-02</p> <p>xx-03</p> <p>xxx</p> <p>Thank you for your help Lee, please removed those test group names. I will try my best to make the group name list short. Thanks again.</p> <p>I can't see the "EXASCALER"  project, it says</p> <p>"Access to project EXASCALER is restricted"</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=leonel8a" class="user-hover" rel="leonel8a">leonel8a</a>,  can we just reuse job name "lustre-master" and group names "full-dne-part-1/2/3" ? Like: <a href="https://testing.whamcloud.com/test_sessions?jobs=lustre-master&amp;user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7&amp;builds=104#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions?jobs=lustre-master&amp;user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7&amp;builds=104#redirect</a></p> <p>Or use distinguishable names？</p> <p>You really don't want to use an existing name for a job and test group that doesn't run those exact same tests because at that point you're mixing results that are not related. And since you're running tests unrelated from the normal autotest pipeline I would definitely suggest you use your own custom distinguishable test group names and job name "custom"</p> <p>OK, understand I will change group names to "linaro-full-part-1/2/3".</p> <p>But job name "custom" doen't works for me, I can't see my test session with job name "custom".</p> <p>They're right here <a href="https://testing.whamcloud.com/test_sessions?horizon=518400&amp;jobs%5B%5D=custom&amp;test_groups%5B%5D=linaro-full-part-1&amp;test_groups%5B%5D=linaro-full-part-2&amp;test_groups%5B%5D=linaro-full-part-3&amp;source=test_sessions#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions?horizon=518400&amp;jobs%5B%5D=custom&amp;test_groups%5B%5D=linaro-full-part-1&amp;test_groups%5B%5D=linaro-full-part-2&amp;test_groups%5B%5D=linaro-full-part-3&amp;source=test_sessions#redirect</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> please remember to set the trigger_job attribute in the results file as "custom", looks like you've uploaded results with "lustre-master" as the job in the last few days, this generated some errors on the maloo side because other properties are missing which are required for non-custom jobs</p> <p> </p> <p><a href="https://testing.whamcloud.com/test_sessions?jobs%5B%5D=lustre-master&amp;user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7&amp;start_date=2022-05-29&amp;end_date=2022-06-01&amp;source=test_sessions#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions?jobs%5B%5D=lustre-master&amp;user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7&amp;start_date=2022-05-29&amp;end_date=2022-06-01&amp;source=test_sessions#redirect</a></p> <p>OK, I stop uploading results with job name "lustre-master" now.</p> <p>Again, I can't see any record in the links: <a href="https://testing.whamcloud.com/test_sessions?horizon=518400&amp;jobs%5B%5D=custom&amp;test_groups%5B%5D=linaro-full-part-1&amp;test_groups%5B%5D=linaro-full-part-2&amp;test_groups%5B%5D=linaro-full-part-3&amp;source=test_sessions#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions?horizon=518400&amp;jobs%5B%5D=custom&amp;test_groups%5B%5D=linaro-full-part-1&amp;test_groups%5B%5D=linaro-full-part-2&amp;test_groups%5B%5D=linaro-full-part-3&amp;source=test_sessions#redirect</a> or <a href="https://testing.whamcloud.com/test_sessions?jobs=custom&amp;user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions?jobs=custom&amp;user_id=b8340029-197d-4ce0-a8f1-40f76d3bb8c7#redirect</a></p> <p><span class="image-wrap" style=""><a id="43923_thumb" href="https://jira.whamcloud.com/secure/attachment/43923/43923_custom-snapshot-20220602182123.png" title="custom-snapshot-20220602182123.png" file-preview-type="image" file-preview-id="43923" file-preview-title="custom-snapshot-20220602182123.png"><img src="https://jira.whamcloud.com/secure/thumbnail/43923/_thumb_43923.png" style="border: 0px solid black" role="presentation"/></a></span></p> <p>I have uploaded two tarball with the job name "custom" today.</p> <p>This job name doesn't work for me. Please help to check it.</p> <p> </p> <p>I see what is happening, you should be able to see all your uploads now, any new ones will not show until I get a fix in place, maybe tomorrow or early next week at the latest. Sorry for the inconvenience.</p> <p>Hi Lee, I can see the new upload test sessions with group name "custom" now. Thanks a lot.</p> Development Rank 1|i02p1z: Rank (Obsolete) 9223372036854775807 Severity