[LU-17000] Coverity static analysis issues Created: 28/Jul/23 Updated: 08/Feb/24 |
|
| Status: | Open |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor |
| Reporter: | Tim Day | Assignee: | Tim Day |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Severity: | 3 | ||||||||||||||||||||||||||||||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||||||||||||||||||||||||||||||
| Description |
|
I've been experimenting with more static analyzers (Coverity in this case). While it does have some false positives, it looks like it found a number of legitimate bugs. See https://scan.coverity.com/projects/lustre for details. It's seems like Coverity has been used with Lustre in the past (see: https://wiki.lustre.org/images/8/8a/LUG2013-Lustre_Static_Code_Analysis-Bull.pdf). Although, it doesn't seem like it's been run in a while. Also, there's a number of Coverity comments that seem out-dated and should probably just be cleaned up. |
| Comments |
| Comment by Gerrit Updater [ 28/Jul/23 ] |
|
"Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51793 |
| Comment by Gerrit Updater [ 28/Jul/23 ] |
|
"Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51795 |
| Comment by Gerrit Updater [ 29/Jul/23 ] |
|
"Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51806 |
| Comment by Gerrit Updater [ 29/Jul/23 ] |
|
"Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51807 |
| Comment by Gerrit Updater [ 02/Aug/23 ] |
|
"Jake McManus <jacobpmcmanus@gmail.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51846 |
| Comment by Gerrit Updater [ 05/Aug/23 ] |
|
"Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51876 |
| Comment by Gerrit Updater [ 07/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51795/ |
| Comment by Gerrit Updater [ 19/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51806/ |
| Comment by Andreas Dilger [ 30/Aug/23 ] |
|
Tim, thanks for looking into this. I registered with the project mostly to look around at what is available there, and definitely there are a number of issues that should be fixed. I also added Oleg, Patrick, Serguei, Arshad, and James to this ticket since I think this may be of interest to them. I thought about inviting them directly to the project, but I wasn't sure which email they would want to use. I've excluded lustre/tests, lustre/kernel_patches, ldiskfs/kernel_patches, contrib, and build from the scan, since these are either not user-visible defects, or I don't think a scan on a patch will produce useful results. There still appear to be 250k LOC that are not covered by the component subdivisions, so I'm not sure where they are coming from. There are too many issues to handle all at once, but hopefully with some chipping away at the more serious issues and/or eliminating the false positives will make it a manageable number. It is totally fine to fix multiple issues of the same type in one patch, or multiple issues in the same file, as long as the patch doesn't get too large. If there are a lot of issues being fixed, it might make sense to make new tracking tickets for each release (e.g. "coverity issues fixed in 2.16.0" or similar) so they can be summarized more easily. |
| Comment by Tim Day [ 31/Aug/23 ] |
|
Thanks for organizing the tickets and CC'ing people. I'd been meaning to gauge interest in Coverity, since it seems to have a pretty good hit-rate. Always great to have more interest in this stuff. Someone familiar with ldiskfs should look at the ldiskfs warnings, since I rarely work with it and probably won't submit any patches for it. As for false positives, I need to make a model file. It explains to Coverity what certain functions do i.e. panic, free memory, etc. OpenZFS has an example I could look at for inspiration. That should work better than littering 100s of annotations in the code. For the uncovered LoC, some of the LNDs don't get compiled on my setup. I need to sort that out. snmp support isn't compiled either. But snmp for Lustre should probably be formally deprecated anyway, so people don't use it thinking it's actually supported. I could probably track down the rest of the LoC. Ideally, if we run enough static analysis tools over a consistent period of time, the amount of effort to keep things in good shape will decrease a lot over time. |
| Comment by Patrick Farrell [ 31/Aug/23 ] |
|
In theory, you should be able to compile everything except gnilnd, which is for Cray's previous generation Aries/Gemini networks. (I mean, you might be able to get the necessary headers, but it's likely not worth the trouble for an LND that's just going to get maintenance fixes). o2iblnd and socklnd can compile against the kernel, and while I've never tried, kfilnd should be compilable with freely available code. hornc would be able to help there I'm sure. RE: snmp... I have literally never noticed that directory in 10 years of working on Lustre. Wow. That can, I suspect, be removed. The last change to that that wasn't a general tree cleanup or build fix was in 2012 |
| Comment by Gerrit Updater [ 31/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51793/ |
| Comment by Gerrit Updater [ 31/Aug/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51846/ |
| Comment by Arshad Hussain [ 31/Aug/23 ] |
|
Andreas, Thanks for this ticket. I really wanted to dabble with Coverity (Seeing Tim's) patches. I heard of Coverity, however never used it. It definitely looks like a good time for me to get started with this. Thanks |
| Comment by Gerrit Updater [ 31/Aug/23 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52210 |
| Comment by Gerrit Updater [ 01/Sep/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52216 |
| Comment by Gerrit Updater [ 01/Sep/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52218 |
| Comment by Gerrit Updater [ 05/Sep/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52272 |
| Comment by Gerrit Updater [ 06/Sep/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51876/ |
| Comment by Gerrit Updater [ 13/Sep/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52210/ |
| Comment by Gerrit Updater [ 13/Sep/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52218/ |
| Comment by Gerrit Updater [ 15/Sep/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52384 |
| Comment by Arshad Hussain [ 18/Sep/23 ] |
|
I have a question. For Coverity False Positive, is it expected to add coverity annotation in code and push the patch. Or just marking that Coverity ID as false positive under "scan coverity" webpage/dashboard be sufficient? |
| Comment by Tim Day [ 19/Sep/23 ] |
|
I need to make a Coverity model file. A model file explains special macros and functions to Coverity. That should reduce the number of false positives we see and make the analysis more robust. I'm hoping that eliminates the need to have annotations in the code. |
| Comment by Arshad Hussain [ 19/Sep/23 ] |
|
Yes. Model file is another way to supress false positives. Thanks |
| Comment by Gerrit Updater [ 28/Sep/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52272/ |
| Comment by Gerrit Updater [ 28/Sep/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52384/ |
| Comment by Gerrit Updater [ 06/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52583 |
| Comment by Gerrit Updater [ 12/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52656 |
| Comment by Gerrit Updater [ 12/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52658 |
| Comment by Gerrit Updater [ 12/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52660 |
| Comment by Gerrit Updater [ 13/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52686 |
| Comment by Gerrit Updater [ 25/Oct/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52216/ |
| Comment by Gerrit Updater [ 25/Oct/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52583/ |
| Comment by Gerrit Updater [ 25/Oct/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52658/ |
| Comment by Gerrit Updater [ 25/Oct/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52660/ |
| Comment by Andreas Dilger [ 30/Oct/23 ] |
|
Tim, can you please edit the ticket Description (or provide a few words here and I can edit it) on how the patches that address Coverity bugs should be tagged, so that this is done consistently. It would also be good to edit the Commit Message wiki page to add the same info. Your patches used: Addresses-Coverity: NNNNNN ("issue type")
though I think "Addresses" is implicit anyway for a bug fix (certainly we wouldn't ave an ID for creating an issue in Coverity), so IMHO it could be: CoverityID: NNNNNN ("issue type"): filename
as Arshad has been using? Would it be better to have the actual URL of the Coverity issue (which will also include the ID number)? That would make it easier to jump to the page that describes the issue to seethe background of why Coverity thinks this is a bug. The pages themselves are currently not public, but I think it is reasonably to give access to any Lustre developer who asks. I think it is better to avoid making the pages totally public to make it harder to expose zero-day vulnerabilities in the code, since we do not have a dedicated team that can address all of the Coverity issues quickly (thanks to you and Arshad that these are being worked on at all). It might make sense to include a sentence on how to request Coverity access, if that isn't already obvious once you get to the page itself. |
| Comment by Andreas Dilger [ 30/Oct/23 ] |
|
timday, can you also provide a brief overview on what needs to be done to update the build stats for the Lustre Coverity build. I was looking at the site and see that it hasn't been updated since Sept 10, but patches for at least 20 Coverity issues have landed since then (just doing a grep for "Coverity|CID". |
| Comment by Andreas Dilger [ 30/Oct/23 ] |
|
Looking at the upstream kernel commits, the majority of patches that reference Coverity are using "Addresses-Coverity-ID:" (62 in past 2 years vs. 25 using "Addresses-Coverity:", and 19 using "CID" in some form though often in conjunction with one of the other forms as well). |
| Comment by Tim Day [ 30/Oct/23 ] |
|
The commit message needs at least "Coverity" and the ID number. Beyond that, no strong preference. Links would be nice, but I think they get very long and ugly - so not practical. The style I used (as you already saw) I copied from kernel commits. I can update the ticket and commit message wiki.
I think works well enough. The filename can be seen already by looking at the commit itself. I updated the Coverity build Oct. 6th. I try (and fail) to update it once at every landing. You need to download the Coverity build tool and run `cov-build --dir cov-int make` instead of a regular `make` then upload a tarball containing the newly created `cov-int` directory. Detailed instructions are on the Coverity site. It ought to be automated. But until then, I'll probably manually run it myself. |
| Comment by Gerrit Updater [ 31/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52901 |
| Comment by Gerrit Updater [ 31/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52902 |
| Comment by Gerrit Updater [ 31/Oct/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52903 |
| Comment by Gerrit Updater [ 01/Nov/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52920 |
| Comment by Gerrit Updater [ 01/Nov/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52921 |
| Comment by Gerrit Updater [ 02/Nov/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52950 |
| Comment by Gerrit Updater [ 03/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52656/ |
| Comment by Gerrit Updater [ 03/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52686/ |
| Comment by Gerrit Updater [ 08/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52901/ |
| Comment by Gerrit Updater [ 08/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52950/ |
| Comment by Gerrit Updater [ 18/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52902/ |
| Comment by Gerrit Updater [ 18/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52903/ |
| Comment by Gerrit Updater [ 18/Nov/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52921/ |
| Comment by Gerrit Updater [ 20/Nov/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53182 |
| Comment by Gerrit Updater [ 01/Dec/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53305 |
| Comment by Gerrit Updater [ 05/Dec/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53327 |
| Comment by Gerrit Updater [ 05/Dec/23 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53331 |
| Comment by Gerrit Updater [ 10/Dec/23 ] |
|
"Timothy Day <timday@amazon.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53400 |
| Comment by Gerrit Updater [ 13/Dec/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52920/ |
| Comment by Arshad Hussain [ 17/Dec/23 ] |
|
Andreas, Tim - I see this happening from sometime. That is, I cannot assign any CID to myself. I login into web dashboard and If I select a CID and on the left panel try to populate the "owner" as myself I cannot seem to do it. The cursor keeps spinning and nothing happens. I am certain that I am logged in and session has not timed out. I am not sure what is happening. Are you also facing the same thing as I am? I am wondering how this could be solved. |
| Comment by Andreas Dilger [ 17/Dec/23 ] |
|
I've only assigned a few issues to myself and never had a problem, so I can't say if this is happening often. |
| Comment by Arshad Hussain [ 18/Dec/23 ] |
|
This is something which is recently started to happen. It was working fine in the past. Could you please Assign CID: 397828 to me. I first wanted to rule out if this is something with I am only affected. Maybe you could drop me from members and re-add back, this could work. I am not sure if anyting could be done from my end. |
| Comment by Gerrit Updater [ 20/Dec/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53182/ |
| Comment by Gerrit Updater [ 20/Dec/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53305/ |
| Comment by Gerrit Updater [ 20/Dec/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53331/ |
| Comment by Gerrit Updater [ 08/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53608 |
| Comment by Gerrit Updater [ 08/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53609 |
| Comment by Arshad Hussain [ 08/Jan/24 ] |
|
Andreas/Tim - Please check CID 412758 and 412759 and let me know your thoughts (https://review.whamcloud.com/c/fs/lustre-release/+/53608) I have added a coverity annotation. The advantage of this would be that false failures would be taged with comments for all to read. On the other hand it might get little noisy. My own perferance would be wihin code. so all reviewers could see why this is being tagged as false failure. |
| Comment by Tim Day [ 10/Jan/24 ] |
|
I wouldn't be surprised if the Coverity site had a bug. But I've never personally experienced problems with assigning CIDs. You could try clearing browser cache. That might help.
As for CID 412758 and 412759, I left some comments in the patch. I'm not a fan of the Coverity annotation comments. Lustre used to have quite a few of them. But they didn't seem to actually impact the false positive rate of the scans. I removed them all and nothing changed. I think we should try using a model file first if we can (it's much easier to test and doesn't produce as much clutter in the code). I've been meaning to write one, but haven't found the time. |
| Comment by Arshad Hussain [ 11/Jan/24 ] |
I did try that. Did not work. It looks like I can atleast mark bug/CID as "fix submitted"/ False failure consistetly. So, this should be enough for me.
Thanks for the feedback. Understood on this. (Apologies for not getting around quicker, Was in other work) |
| Comment by Arshad Hussain [ 12/Jan/24 ] |
|
Tim, The Fixed number do not look correct (From the coverity Dashboard).
|
| Comment by Gerrit Updater [ 16/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53686 |
| Comment by Gerrit Updater [ 17/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53693 |
| Comment by Gerrit Updater [ 19/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53742 |
| Comment by Tim Day [ 20/Jan/24 ] |
|
The large number of 'Fixed' defects came from a bug in a newer version of the Coverity build tool (a compiler wrapper needed to submit builds). I've reverted to using an older version for now. Unfortunately, we're stuck with all of the false positives showing up as 'Fixed' on the site. |
| Comment by Arshad Hussain [ 22/Jan/24 ] |
Got it and Thanks for looking into this. |
| Comment by Gerrit Updater [ 22/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53758 |
| Comment by Gerrit Updater [ 24/Jan/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53796 |
| Comment by Gerrit Updater [ 04/Feb/24 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53400/ |
| Comment by Gerrit Updater [ 04/Feb/24 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53608/ |
| Comment by Gerrit Updater [ 04/Feb/24 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53686/ |
| Comment by Gerrit Updater [ 04/Feb/24 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53693/ |
| Comment by Gerrit Updater [ 04/Feb/24 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53742/ |
| Comment by Gerrit Updater [ 04/Feb/24 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53758/ |
| Comment by Gerrit Updater [ 06/Feb/24 ] |
|
"Arshad Hussain <arshad.hussain@aeoncomputing.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53936 |