How to preprocess data

[bstee615]

Hi, I am trying to run your project because I would love to use it in my research work. I'm working from branch new_implementation (e93a3b6) since that seems to be the most stable and I wasn't able to run the master branch. I was able to run through the training code with the provided sample data. Thank you for making your project available and for providing preprocessed data.

I want to train the model on a new dataset. Unfortunately, I can't figure out how to preprocess raw code to produce a form like IVDetect/data/graph_data/devign_sample_output.json. Can you give me detailed instructions for how to preprocess data from raw source code, running through Joern, and into JSON so I can faithfully reproduce your work? This is how far I got:

• I assume that the keys CPG, CDG, and DDG are the direct output of Joern, from running joern-export --repr XXX --out outdir with --repr as cpg14, cdg, and ddg respectively. Is this correct?
• How is the key No2St generated? As far as I can tell, this maps nodes to statements, but I don't know what the indexes mean.

[liwenke1]

For your second question, i think the indeses in the key No2St represent the row number where these nodes are located

[bstee615]

@liwenke1 Thanks for the info!

[liwenke1]

@bstee615 I aslo get the key CPG,CDG,andDDG througn Joern,but i cannot get the key No2St .I thougnt the tool codepropertygraph can extract the line number in CPG nodes, but i am not sure,How did you get the key No2St? Thanks

[bstee615]

@liwenke1 Sorry, I haven't progressed in reproducing this implementation so I haven't gotten the key No2St. I guess No2St is probably generated outside of Joern. Best of luck.

[1112z-c]

@bstee615 Hi,brother,Can you share the IVDetect/data/graph_data/devign_sample_output.json for me ? This version do not have !Thanks!

[bstee615]

@bstee615 Hi,brother,Can you share the IVDetect/data/graph_data/devign_sample_output.json for me ? This version do not have !Thanks!

Sure, go to the version of the repository at commit e93a3b6. Here's the link: https://github.com/vulnerabilitydetection/VulnerabilityDetectionResearch/blob/e93a3b664babe5551361d176df6e07ee8c81e1ee/IVDetect/data/graph_data/devign_sample_output.json

[1112z-c]

Thanks！

…

------------------ 原始邮件 ------------------ 发件人: "vulnerabilitydetection/VulnerabilityDetectionResearch" ***@***.***>; 发送时间: 2022年3月22日(星期二) 凌晨0:01 ***@***.***>; ***@***.******@***.***>; 主题: Re: [vulnerabilitydetection/VulnerabilityDetectionResearch] How to preprocess data (Issue #4) @bstee615 Hi,brother,Can you share the IVDetect/data/graph_data/devign_sample_output.json for me ? This version do not have !Thanks! Sure, go to the version of the repository at commit e93a3b6. Here's the link: https://github.com/vulnerabilitydetection/VulnerabilityDetectionResearch/blob/e93a3b664babe5551361d176df6e07ee8c81e1ee/IVDetect/data/graph_data/devign_sample_output.json — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[GuoXinrong]

@bstee615 hello，do you figure out how to use joern to parse a new dataset(csv) into the form as the author mentioned in readme.txt ? I don't know how to use joern(https://github.com/vulnerabilitydetection/VulnerabilityDetectionResearch/tree/new_implementation/IVDetect/scripts/joern_graphs.sc) the author mentioned in the project to preprocess the dataset.Or maybe I should learn how to use the newer versions of Joern by myself?.here is the the generated dataset :

We put an example CSV dataset to show how the generated dataset looks: https://drive.google.com/file/d/1LHOC4JDpnQ7gWnEHGfc4soQYHAPomlNp/view?usp=sharing

[bstee615]

@GuoXinrong I haven't reproduced IVDetect. For my work I used LineVD project, which is a more recent SOTA published in MSR22. https://github.com/davidhin/linevd
Here is the relevant script for processing with Joern: https://github.com/davidhin/linevd/blob/main/sastvd/scripts/getgraphs.py
Hope it helps!

[GuoXinrong]

@bstee615 ,thanks very much ,I have run the LineVD project but have encounter some problems, do you use the singularity tool the project mentioned in the readme.txt? or just follow the command in the Singularity doc?
I find that the output of the gitdiff() in /linevd/sastvd/helpers/git.py is (b'', b'/bin/sh: 1: singularity: not found\n') ,I guess it falis to extract patch,so the following method can not continue to extract the diff,added,removed infomation.I wanted to ask if you have ever had this problem？

[bstee615]

@bstee615 ,thanks very much ,I have run the LineVD project but have encounter some problems, do you use the singularity tool the project mentioned in the readme.txt? or just follow the command in the Singularity doc? I find that the output of the gitdiff() in /linevd/sastvd/helpers/git.py is (b'', b'/bin/sh: 1: singularity: not found\n') ,I guess it falis to extract patch,so the following method can not continue to extract the diff,added,removed infomation.I wanted to ask if you have ever had this problem？

I ran with the singularity image. Given your error message, you probably need to install singularity or remove the invocation of the singularity command and just run on native.

[Funkotronic]

Hi all, has there been any progress with running joern_graph.sc? I'm currently trying to replicate the authors' results though have little experience with Scala and can't seem to figure this out. I've tried running this through scala and sbt in bash with no luck - not sure if it's meant to be implemented alongside Joern.

[CFCl-BrI]

Hi all, has there been any progress with running joern_graph.sc? I'm currently trying to replicate the authors' results though have little experience with Scala and can't seem to figure this out. I've tried running this through scala and sbt in bash with no luck - not sure if it's meant to be implemented alongside Joern.

Hello! Have you solved this problem? I have the same problem on running joern_graph.sc, for I can't run it on the latest version of joern because of the missing germlin package, meanwhile I don't know how to run the joern after git checkout operation.

[wanglianmei]

你好，请问能分享使用joern处理devign数据集的过程吗？还有处理后的数据集

[mcf20]

我没有复制 IVDetect。在我的工作中，我使用了 LineVD 项目，这是最近在 MSR22 中发布的 SOTA。https://github.com/davidhin/linevd 以下是使用 Joern 进行处理的相关脚本：https://github.com/davidhin/linevd/blob/main/sastvd/scripts/getgraphs.py 希望有所帮助
请问该项目能浮复现么