DSN26 Tutorial Activities

sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/README.md

@@ -1,4 +1,13 @@
-## Steps to execute Roberta-Seq-Clasification:
+## NLP Fault Injection Example - Roberta-Seq-Clasification Model
+
+Pre-set Input YAML Configurations
+---
+
+Replace `input.yaml` with `input1.yaml` (for reduced number of runs) and `input2.yaml` for greater fault intensity.
+
+
+Fault Injection Runtime Steps
+---
 
 1. After converting the ONNX model to LLVM IR i.e. after executig the `compile.sh` script, run fault injection experiment on the required input. Below is the command to execute the model on 'input 0'. A total of 10 different inputs (numbered 0-9) are available and can be specified in the argument.
 ```

sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/input1.yaml

@@ -0,0 +1,32 @@
+compileOption:
+    instSelMethod:
+      - customInstselector:
+          include:
+            - CustomTensorOperator
+          options:
+            - -layerNo=0
+            - -layerName=all
+
+
+    regSelMethod: regloc
+    regloc: dstreg
+
+    includeInjectionTrace:
+        - forward
+
+    tracingPropagation: False # trace dynamic instruction values.
+
+    tracingPropagationOption:
+        maxTrace: 250 # max number of instructions to trace during fault injection run
+        debugTrace: False
+        mlTrace: False # enable for tracing ML programs
+        generateCDFG: True
+
+runOption:
+    - run:
+        numOfRuns: 10
+        fi_type: bitflip
+        window_len_multiple_startindex: 1
+        window_len_multiple_endindex: 500
+        fi_max_multiple: 5
+        fi_num_bits: 8

sample_programs/ml_sample_programs/vision_models/mnist/README.md

@@ -5,19 +5,35 @@ We link the LLVM IR of the model to an image processing program `image.c`, which
 This generated LLVM IR `model.ll` is instrumented, profiled, and fault injected by LLTFI.\
 All of the following steps can be replicated for `mnist-nn.py`.
 
-Running PyTorch example
+
+Pre-set Input YAML Configurations
+---
+
+Replace `input.yaml` with `input1.yaml` (for reduced number of runs) and `input2.yaml` for greater fault intensity.
+
+
+
+Running a  Pre-trained Model on MNIST example
 ---
 
-1. First ensure that PyTorch framework (v1.9.0 or greater) is installed.
+1. Compile a pre-trained CNN model on MNIST to LLVM IR. The final output file is `model.ll`.
+```
+./compile.sh
+```
+
+2. Run LLTFI on one of the images, `eight.png` by default.
+```
+./runllfi.sh
+```
 
-2. Train CNN on MNIST and compile it to LLVM IR. The final output file is `model.ll`.
+3. Compute the number of SDCs where the model output deviates from the golden output.
 ```
-./compile-pytorch.sh
+./check_sdc.sh
 ```
 
-3. Select one of the test image files, e.g. `eight.png` and run LLFI on it.
+4. Compute the number of critical SDCs where the model output causes a different prediction from the golden prediction.
 ```
-./runllfi.sh eight.png
+python check_critical_sdc.py
 ```
 
 

sample_programs/ml_sample_programs/vision_models/mnist/check_critical_sdc.py

@@ -0,0 +1,38 @@
+import os
+
+GOLDEN_INDEX = 8
+DIRECTORY = "./llfi/std_output"
+
+mismatches = []
+total_files = 0
+
+for filename in os.listdir(DIRECTORY):
+    path = os.path.join(DIRECTORY, filename)
+
+    if not os.path.isfile(path):
+        continue
+
+    total_files += 1
+
+    try:
+        with open(path, "r") as f:
+            last_line = f.readlines()[-1].strip()
+
+        # Extract everything after "is:"
+        values_str = last_line.split("is:", 1)[1].strip()
+        values = [float(x) for x in values_str.split()]
+
+        max_index = values.index(max(values))
+
+        if max_index != GOLDEN_INDEX:
+            mismatches.append((path, max_index))
+
+    except Exceptpath:
+        print(f"Skipping {path}: {e}")
+
+for path, idx in mismatches:
+    print(f"Different: {path}: predicted class = {idx}")
+
+print(f"Total Number of Runs: {total_files}")
+print(f"NUmber of Critical SDCs: {len(mismatches)}")
+

sample_programs/ml_sample_programs/vision_models/mnist/check_sdc.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+golden_file="./llfi/baseline/golden_std_output"
+directory="./llfi/std_output"
+
+if [[ ! -f "$golden_file" ]]; then
+    echo "Error: Golden file '$golden_file' not found."
+    exit 1
+fi
+
+golden_last=$(tail -n 1 "$golden_file")
+
+checked=0
+different=0
+
+for file in "$directory"/*; do
+    [[ -f "$file" ]] || continue
+    [[ "$file" -ef "$golden_file" ]] && continue
+
+    ((checked++))
+
+    last_line=$(tail -n 1 "$file")
+
+    if [[ "$last_line" != "$golden_last" ]]; then
+        echo "Different: $file"
+        ((different++))
+    fi
+done
+
+echo
+echo "Total Number of Runs: $checked"
+echo "Number of SDCs: $different"
+

sample_programs/ml_sample_programs/vision_models/mnist/input1.yaml

@@ -0,0 +1,30 @@
+compileOption:
+    instSelMethod:
+      - customInstselector:
+          include:
+            - CustomTensorOperator
+          options:
+            - -layerNo=0;0;0;0;0;0;0
+            - -layerName=conv;relu;matmul;maxpool;add;avgpool;softmax
+
+    regSelMethod: regloc
+    regloc: dstreg
+
+    includeInjectionTrace:
+        - forward
+
+    tracingPropagation: False # trace dynamic instruction values.
+
+    tracingPropagationOption:
+        maxTrace: 250 # max number of instructions to trace during fault injection run
+        debugTrace: False
+        mlTrace: False # enable for tracing ML programs
+        generateCDFG: True
+
+runOption:
+    - run:
+        numOfRuns: 20
+        fi_type: bitflip
+        window_len_multiple_startindex: 1
+        window_len_multiple_endindex: 500
+        fi_max_multiple: 2

sample_programs/ml_sample_programs/vision_models/mnist/input2.yaml

@@ -0,0 +1,31 @@
+compileOption:
+    instSelMethod:
+      - customInstselector:
+          include:
+            - CustomTensorOperator
+          options:
+            - -layerNo=2
+            - -layerName=conv
+
+    regSelMethod: regloc
+    regloc: dstreg
+
+    includeInjectionTrace:
+        - forward
+
+    tracingPropagation: False # trace dynamic instruction values.
+
+    tracingPropagationOption:
+        maxTrace: 250 # max number of instructions to trace during fault injection run
+        debugTrace: False
+        mlTrace: False # enable for tracing ML programs
+        generateCDFG: True
+
+runOption:
+    - run:
+        numOfRuns: 20
+        fi_type: bitflip
+        window_len_multiple_startindex: 1
+        window_len_multiple_endindex: 500
+        fi_max_multiple: 5
+        fi_num_bits: 8

tutorials/DSN26/Extra_NLP_Activity.md

@@ -0,0 +1,66 @@
+# DSN 2026 Tutorial Activity - Instructions (Additional NLP Activity)
+
+This README describes an additional tutorial activity - performing fault injection into an NLP model.
+
+## Benchmarks
+* **Roberta-Seq-Clasification** - This NLP model performs sentiment analysis to determine whether text inputs are positive or negative.
+
+
+## Part 3: Fault Injection into an NLP Model (15 min)
+1. Navigate to the Roberta-Seq-Clasification sample program.
+   ```
+   cd /home/LLTFI/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/
+   ```
+2. Change `input.yaml` so that fault injection is only ran 100 times. This line `numOfRuns: 1000` should be changed to `numOfRuns: 10`.
+   For your convenience, all of these changes have been made in `input1.yaml`. We can simply copy the YAML settings by running these following commands.
+
+   ```
+   cp input1.yaml input.yaml
+   ```
+
+
+   ```
+   runOption:
+       - run:
+          numOfRuns: 10 (Change this line)
+          fi_type: bitflip
+          ...
+          fi_max_multiple: 5 (Change this line)
+          fi_num_bits: 8 (Add this line)
+   ```
+
+3. Compile the pre-trained NLP model from ONNX into LLVM IR. If running this step for the first time, it will automatically download the pre-trained model.
+   You should see the `model.ll` file generated - this is the LLVM IR representation of the trained neural network.
+   ```
+   ./compile.sh
+   ```
+
+4. After converting the ONNX model to LLVM IR i.e. after executig the compile.sh script, run fault injection experiment on the required input. Below is the command to execute the model on 'input 0'. A total of 10 different inputs (numbered 0-9) are available and can be specified in the argument.
+   ```
+   ./runllfiSingleInp.sh 0
+   ```
+
+  The task here is sentiment analysis and the prediction for any input is either 'Positive' or 'Negative'.
+  In this example, input 0 is chosen, which corresponds to the input: "It is a bright, sunny day".
+  The expected sentiment output for this input is **Positive**.
+
+
+5. Execute the 'createPredFile' python file to generate prediction. The output file generated is 'prediction/PredResult.txt'. Print out the predicted output file to examine the result. You should find at least one erroneous prediction that is negative, in lieu of positive (expected output).
+   ```
+   python createPredFile.py && cat prediction/PredResult.txt
+   ```
+
+   Expected Example Output:
+   ```
+   Run #0 Prediction: Positive
+   Run #1 Prediction: Positive
+   Run #2 Prediction: Positive
+   Run #3 Prediction: Positive
+   Run #4 Prediction: Negative
+   Run #5 Prediction: Positive
+   Run #6 Prediction: Positive
+   Run #7 Prediction: Positive
+   Run #8 Prediction: Positive
+   Run #9 Prediction: Positive
+   ```
+