diff --git a/Code-Code/CodeCompletion-line/README.md b/Code-Code/CodeCompletion-line/README.md index 6515362aa6..98aefaf950 100644 --- a/Code-Code/CodeCompletion-line/README.md +++ b/Code-Code/CodeCompletion-line/README.md @@ -84,7 +84,7 @@ def __init__ ( self ) : ## Pipeline -We provide a pipeline that evaluate line completion on [CodeGPT](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-token#codegpt) model. You could directly use the model trained on token level code completion to test on line-level completion. +We provide a pipeline that evaluate line completion on [CodeGPT](/Code-Code/CodeCompletion-token#codegpt) model. You could directly use the model trained on token level code completion to test on line-level completion. ### Dependency diff --git a/Code-Code/code-refinement/README.md b/Code-Code/code-refinement/README.md index f5334491ab..e3633c45df 100644 --- a/Code-Code/code-refinement/README.md +++ b/Code-Code/code-refinement/README.md @@ -4,7 +4,7 @@ Code refinement aims to automatically fix bugs in the code, which can contribute to reducing the cost of bug-fixes for developers. In CodeXGLUE, given a piece of Java code with bugs, the task is to remove the bugs to output the refined code. -Models are evaluated by BLEU scores, accuracy (exactly match) and [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/CodeBLEU.MD). +Models are evaluated by BLEU scores, accuracy (exactly match) and [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD). ## Dataset @@ -119,16 +119,16 @@ The results on the test set are shown as below: Small: -| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) | +| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) | | ---------- | :-------: | :-------: | :-------: | | Naive copy | 78.06 | 0.0 | - | | LSTM | 76.76 | 10.0 | - | -| Transformer| 77.21 | 14.7 | 73.31 | +| Transformer| 77.21 | 14.7 | 73.31 | | CodeBERT | **77.42** | **16.4** | **75.58** | Medium: -| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) | +| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) | | ---------- | :-------: | :-------: | :-------: | | Naive copy | 90.91 | 0.0 | - | | LSTM | 72.08 | 2.5 | - | diff --git a/Code-Code/code-to-code-trans/CodeBLEU.MD b/Code-Code/code-to-code-trans/CodeBLEU.MD index 4caaa20c75..50ef13fafe 100644 --- a/Code-Code/code-to-code-trans/CodeBLEU.MD +++ b/Code-Code/code-to-code-trans/CodeBLEU.MD @@ -28,13 +28,13 @@ BLEU, but can also consider the grammatical correctness and the logic correctnes An ideal evaluation metric should consider the grammatical correctness and the logic correctness. We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness. -![CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/CodeBLEU.jpg) +![CodeBLEU](CodeBLEU.jpg) ### Examples Here we will give two toy examples and show the qualitative advantages of CodeBLEU compared with the traditional BLEU score. -![Example](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/example.png) +![Example](example.png) diff --git a/Code-Code/code-to-code-trans/README.md b/Code-Code/code-to-code-trans/README.md index 9fba4377a3..a195746092 100644 --- a/Code-Code/code-to-code-trans/README.md +++ b/Code-Code/code-to-code-trans/README.md @@ -4,7 +4,7 @@ Code translation aims to migrate legacy software from one programming language in a platform toanother. In CodeXGLUE, given a piece of Java (C#) code, the task is to translate the code into C# (Java) version. -Models are evaluated by BLEU scores, accuracy (exactly match), and [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/CodeBLEU.MD) scores. +Models are evaluated by BLEU scores, accuracy (exactly match), and [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) scores. ## Dataset @@ -119,7 +119,7 @@ The results on the test set are shown as below: Java to C#: -| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) | +| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) | | ---------- | :--------: | :-------: | :-------: | | Naive copy | 18.54 | 0.0 | - | | PBSMT | 43.53 | 12.5 | 42.71 | @@ -129,7 +129,7 @@ Java to C#: C# to Java: -| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) | +| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) | | ---------- | :--------: | :-------: | :-------: | | Naive copy | 18.69 | 0.0 | - | | PBSMT | 40.06 | 16.1 | 43.48 | diff --git a/README.md b/README.md index 0a50f46fb8..862cdf1c58 100644 --- a/README.md +++ b/README.md @@ -10,10 +10,10 @@ Recent years have seen a surge of applying of statistical models, including neur To address this, researchers from Microsoft Research Asia, Developer Division, and Bing introduce CodeXGLUE, a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios: -* **[code-code](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code)** (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation) -* **[text-code](https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code)** (natural language code search, text-to-code generation) -* **[code-text](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/)** (code summarization) -* **[text-text](https://github.com/microsoft/CodeXGLUE/tree/main/Text-Text)** (documentation translation) +* **[code-code](/Code-Code)** (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation) +* **[text-code](/Text-Code)** (natural language code search, text-to-code generation) +* **[code-text](/Code-Text/)** (code summarization) +* **[text-text](/Text-Text)** (documentation translation) A brief summary of CodeXGLUE is given below, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced. ![A brief summary of CodeXGLUE, including tasks, datasets, baseline systems, etc.](tasks.jpg) diff --git a/Text-Code/text-to-code/README.md b/Text-Code/text-to-code/README.md index f8a0184b23..ec3419a45f 100644 --- a/Text-Code/text-to-code/README.md +++ b/Text-Code/text-to-code/README.md @@ -49,7 +49,7 @@ The outputs are: BLEU: 16.68, EM: 17.0 ``` -The CodeBLEU score can be calculated by this [script](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/code-to-code-trans/evaluator/CodeBLEU) +The CodeBLEU score can be calculated by this [script](/Code-Code/code-to-code-trans/evaluator/CodeBLEU) ### Input Format @@ -69,7 +69,7 @@ public void inc ( ) { this . add ( 1 ) ; } ## Pipeline -We provide a pipeline for this task with [CodeGPT and CodeGPT-adapted](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-token#codegpt) model. +We provide a pipeline for this task with [CodeGPT and CodeGPT-adapted](/Code-Code/CodeCompletion-token#codegpt) model. ## Dependency