diff --git a/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/README.md b/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/README.md index ff97ee3b..c85998da 100644 --- a/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/README.md +++ b/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/README.md @@ -1,4 +1,13 @@ -## Steps to execute Roberta-Seq-Clasification: +## NLP Fault Injection Example - Roberta-Seq-Clasification Model + +Pre-set Input YAML Configurations +--- + +Replace `input.yaml` with `input1.yaml` (for reduced number of runs) and `input2.yaml` for greater fault intensity. + + +Fault Injection Runtime Steps +--- 1. After converting the ONNX model to LLVM IR i.e. after executig the `compile.sh` script, run fault injection experiment on the required input. Below is the command to execute the model on 'input 0'. A total of 10 different inputs (numbered 0-9) are available and can be specified in the argument. ``` diff --git a/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/input1.yaml b/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/input1.yaml new file mode 100644 index 00000000..059d1bcb --- /dev/null +++ b/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/input1.yaml @@ -0,0 +1,32 @@ +compileOption: + instSelMethod: + - customInstselector: + include: + - CustomTensorOperator + options: + - -layerNo=0 + - -layerName=all + + + regSelMethod: regloc + regloc: dstreg + + includeInjectionTrace: + - forward + + tracingPropagation: False # trace dynamic instruction values. + + tracingPropagationOption: + maxTrace: 250 # max number of instructions to trace during fault injection run + debugTrace: False + mlTrace: False # enable for tracing ML programs + generateCDFG: True + +runOption: + - run: + numOfRuns: 10 + fi_type: bitflip + window_len_multiple_startindex: 1 + window_len_multiple_endindex: 500 + fi_max_multiple: 5 + fi_num_bits: 8 diff --git a/sample_programs/ml_sample_programs/vision_models/mnist/README.md b/sample_programs/ml_sample_programs/vision_models/mnist/README.md index b6e5c93a..2bfe412e 100644 --- a/sample_programs/ml_sample_programs/vision_models/mnist/README.md +++ b/sample_programs/ml_sample_programs/vision_models/mnist/README.md @@ -5,19 +5,35 @@ We link the LLVM IR of the model to an image processing program `image.c`, which This generated LLVM IR `model.ll` is instrumented, profiled, and fault injected by LLTFI.\ All of the following steps can be replicated for `mnist-nn.py`. -Running PyTorch example + +Pre-set Input YAML Configurations +--- + +Replace `input.yaml` with `input1.yaml` (for reduced number of runs) and `input2.yaml` for greater fault intensity. + + + +Running a Pre-trained Model on MNIST example --- -1. First ensure that PyTorch framework (v1.9.0 or greater) is installed. +1. Compile a pre-trained CNN model on MNIST to LLVM IR. The final output file is `model.ll`. +``` +./compile.sh +``` + +2. Run LLTFI on one of the images, `eight.png` by default. +``` +./runllfi.sh +``` -2. Train CNN on MNIST and compile it to LLVM IR. The final output file is `model.ll`. +3. Compute the number of SDCs where the model output deviates from the golden output. ``` -./compile-pytorch.sh +./check_sdc.sh ``` -3. Select one of the test image files, e.g. `eight.png` and run LLFI on it. +4. Compute the number of critical SDCs where the model output causes a different prediction from the golden prediction. ``` -./runllfi.sh eight.png +python check_critical_sdc.py ``` diff --git a/sample_programs/ml_sample_programs/vision_models/mnist/check_critical_sdc.py b/sample_programs/ml_sample_programs/vision_models/mnist/check_critical_sdc.py new file mode 100644 index 00000000..734800c8 --- /dev/null +++ b/sample_programs/ml_sample_programs/vision_models/mnist/check_critical_sdc.py @@ -0,0 +1,38 @@ +import os + +GOLDEN_INDEX = 8 +DIRECTORY = "./llfi/std_output" + +mismatches = [] +total_files = 0 + +for filename in os.listdir(DIRECTORY): + path = os.path.join(DIRECTORY, filename) + + if not os.path.isfile(path): + continue + + total_files += 1 + + try: + with open(path, "r") as f: + last_line = f.readlines()[-1].strip() + + # Extract everything after "is:" + values_str = last_line.split("is:", 1)[1].strip() + values = [float(x) for x in values_str.split()] + + max_index = values.index(max(values)) + + if max_index != GOLDEN_INDEX: + mismatches.append((path, max_index)) + + except Exceptpath: + print(f"Skipping {path}: {e}") + +for path, idx in mismatches: + print(f"Different: {path}: predicted class = {idx}") + +print(f"Total Number of Runs: {total_files}") +print(f"NUmber of Critical SDCs: {len(mismatches)}") + diff --git a/sample_programs/ml_sample_programs/vision_models/mnist/check_sdc.sh b/sample_programs/ml_sample_programs/vision_models/mnist/check_sdc.sh new file mode 100755 index 00000000..87a1947f --- /dev/null +++ b/sample_programs/ml_sample_programs/vision_models/mnist/check_sdc.sh @@ -0,0 +1,33 @@ +#!/bin/bash + +golden_file="./llfi/baseline/golden_std_output" +directory="./llfi/std_output" + +if [[ ! -f "$golden_file" ]]; then + echo "Error: Golden file '$golden_file' not found." + exit 1 +fi + +golden_last=$(tail -n 1 "$golden_file") + +checked=0 +different=0 + +for file in "$directory"/*; do + [[ -f "$file" ]] || continue + [[ "$file" -ef "$golden_file" ]] && continue + + ((checked++)) + + last_line=$(tail -n 1 "$file") + + if [[ "$last_line" != "$golden_last" ]]; then + echo "Different: $file" + ((different++)) + fi +done + +echo +echo "Total Number of Runs: $checked" +echo "Number of SDCs: $different" + diff --git a/sample_programs/ml_sample_programs/vision_models/mnist/input1.yaml b/sample_programs/ml_sample_programs/vision_models/mnist/input1.yaml new file mode 100644 index 00000000..2d229bff --- /dev/null +++ b/sample_programs/ml_sample_programs/vision_models/mnist/input1.yaml @@ -0,0 +1,30 @@ +compileOption: + instSelMethod: + - customInstselector: + include: + - CustomTensorOperator + options: + - -layerNo=0;0;0;0;0;0;0 + - -layerName=conv;relu;matmul;maxpool;add;avgpool;softmax + + regSelMethod: regloc + regloc: dstreg + + includeInjectionTrace: + - forward + + tracingPropagation: False # trace dynamic instruction values. + + tracingPropagationOption: + maxTrace: 250 # max number of instructions to trace during fault injection run + debugTrace: False + mlTrace: False # enable for tracing ML programs + generateCDFG: True + +runOption: + - run: + numOfRuns: 20 + fi_type: bitflip + window_len_multiple_startindex: 1 + window_len_multiple_endindex: 500 + fi_max_multiple: 2 diff --git a/sample_programs/ml_sample_programs/vision_models/mnist/input2.yaml b/sample_programs/ml_sample_programs/vision_models/mnist/input2.yaml new file mode 100644 index 00000000..6c9f41ed --- /dev/null +++ b/sample_programs/ml_sample_programs/vision_models/mnist/input2.yaml @@ -0,0 +1,31 @@ +compileOption: + instSelMethod: + - customInstselector: + include: + - CustomTensorOperator + options: + - -layerNo=2 + - -layerName=conv + + regSelMethod: regloc + regloc: dstreg + + includeInjectionTrace: + - forward + + tracingPropagation: False # trace dynamic instruction values. + + tracingPropagationOption: + maxTrace: 250 # max number of instructions to trace during fault injection run + debugTrace: False + mlTrace: False # enable for tracing ML programs + generateCDFG: True + +runOption: + - run: + numOfRuns: 20 + fi_type: bitflip + window_len_multiple_startindex: 1 + window_len_multiple_endindex: 500 + fi_max_multiple: 5 + fi_num_bits: 8 diff --git a/tutorials/DSN26/Extra_NLP_Activity.md b/tutorials/DSN26/Extra_NLP_Activity.md new file mode 100644 index 00000000..71199e12 --- /dev/null +++ b/tutorials/DSN26/Extra_NLP_Activity.md @@ -0,0 +1,66 @@ +# DSN 2026 Tutorial Activity - Instructions (Additional NLP Activity) + +This README describes an additional tutorial activity - performing fault injection into an NLP model. + +## Benchmarks +* **Roberta-Seq-Clasification** - This NLP model performs sentiment analysis to determine whether text inputs are positive or negative. + + +## Part 3: Fault Injection into an NLP Model (15 min) +1. Navigate to the Roberta-Seq-Clasification sample program. + ``` + cd /home/LLTFI/sample_programs/ml_sample_programs/nlp_models/roberta-seq-classification-9/ + ``` +2. Change `input.yaml` so that fault injection is only ran 100 times. This line `numOfRuns: 1000` should be changed to `numOfRuns: 10`. + For your convenience, all of these changes have been made in `input1.yaml`. We can simply copy the YAML settings by running these following commands. + + ``` + cp input1.yaml input.yaml + ``` + + + ``` + runOption: + - run: + numOfRuns: 10 (Change this line) + fi_type: bitflip + ... + fi_max_multiple: 5 (Change this line) + fi_num_bits: 8 (Add this line) + ``` + +3. Compile the pre-trained NLP model from ONNX into LLVM IR. If running this step for the first time, it will automatically download the pre-trained model. + You should see the `model.ll` file generated - this is the LLVM IR representation of the trained neural network. + ``` + ./compile.sh + ``` + +4. After converting the ONNX model to LLVM IR i.e. after executig the compile.sh script, run fault injection experiment on the required input. Below is the command to execute the model on 'input 0'. A total of 10 different inputs (numbered 0-9) are available and can be specified in the argument. + ``` + ./runllfiSingleInp.sh 0 + ``` + + The task here is sentiment analysis and the prediction for any input is either 'Positive' or 'Negative'. + In this example, input 0 is chosen, which corresponds to the input: "It is a bright, sunny day". + The expected sentiment output for this input is **Positive**. + + +5. Execute the 'createPredFile' python file to generate prediction. The output file generated is 'prediction/PredResult.txt'. Print out the predicted output file to examine the result. You should find at least one erroneous prediction that is negative, in lieu of positive (expected output). + ``` + python createPredFile.py && cat prediction/PredResult.txt + ``` + + Expected Example Output: + ``` + Run #0 Prediction: Positive + Run #1 Prediction: Positive + Run #2 Prediction: Positive + Run #3 Prediction: Positive + Run #4 Prediction: Negative + Run #5 Prediction: Positive + Run #6 Prediction: Positive + Run #7 Prediction: Positive + Run #8 Prediction: Positive + Run #9 Prediction: Positive + ``` + diff --git a/tutorials/DSN26/README.md b/tutorials/DSN26/README.md new file mode 100644 index 00000000..abe88265 --- /dev/null +++ b/tutorials/DSN26/README.md @@ -0,0 +1,209 @@ +# DSN 2026 Tutorial Activity - Instructions + +## Setup (10 min) + +You may choose from one of three choices to use LLTFI for this tutorial. +You are also welcome to use your own installation of LLTFI, but these instructions assume you are using the Docker or Docker on VM image with specific experiment folder paths. + +*Note: Please have at least **18 GB** of free disk space available for this tutorial.* + + +1. If you are using either Linux or Mac OS, please use [Docker](https://docs.docker.com/engine/install/). + If you are using Ubuntu, please follow [these specific instructions](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository). + You will need sudo power to run Docker, unless you have been added to a [Docker group](https://docs.docker.com/engine/install/linux-postinstall/). + + ``` + sudo docker image pull abrahamchan/lltfi-dsn26 + sudo docker tag abrahamchan/lltfi-dsn26 lltfi-dsn26 + ``` + + +## Running LLTFI (1 min) + +1. Once you have installed Docker image, please check that the Docker image is successfully loaded by typing `sudo docker images` in the terminal. + + ``` + >>> sudo docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + lltfi-dsn26 latest b0c7d6e417c2 7 days ago 16.6GB + ``` + +2. Run the Docker image as a container. + + ``` + sudo docker run -it lltfi-dsn26 /bin/bash + ``` + + +## Benchmarks +We provide a benchmark to run fault injections for this tutorial. +* **MNIST** - This vision ML model classifies handwritten images of numeric digits (0-9) and returns a list of probability scores for each digit. + + +## Part 1: Fault Injection into a Vision ML Model (10 min) +1. Navigate to the MNIST sample program. + ``` + cd /home/LLTFI/sample_programs/ml_sample_programs/vision_models/mnist/ + ``` + +2. Print out the `input.yaml` file to see the default fault injection options. Originally, the number of runs is set to 1000 times. + ``` + cat input.yaml + ``` + + Snippet from `input.yaml` file: + + ```yaml + runOption: + - run: + numOfRuns: 1000 + fi_type: bitflip + ``` + +3. We modify the `input.yaml` file so that the number of runs is reduced to 20 times. The new version of this file can be copied in at `input1.yaml`. + ``` + cp input1.yaml input.yaml + ``` + + Snippet from `input.yaml` file: + ``` + runOption: + - run: + numOfRuns: 20 (This line has changed from 1000 to 20.) + fi_type: bitflip + ``` + +4. Compile the pre-trained Convolutional Neural Network into ONNX. + You should see the `model.ll` file generated - this is the LLVM IR representation of the trained neural network, originally written in TensorFlow. + ``` + ./compile.sh + ``` + +5. Run LLTFI on the LLVM IR file. + By default, an image of a handwritten eight, `eight.png`, is used as input. + ``` + ./runllfi.sh + ``` + +6. The baseline (golden) run should show the probabilty score for eight with the highest value as shown. + In this example, `eight.png` is predicted as eight with a probability score of 0.998805. + ``` + >>> cat llfi/baseline/golden_std_output + Time taken to execute the model: 0.029147 + Final prediction for eight.png is: 0.000001 0.000000 0.000417 0.000076 0.000000 0.000024 0.000000 0.000001 0.998805 0.000676 + ``` + +7. Compute the number of SDCs and critical SDCs encountered. SDCs are those that lead to deviations in the output, while critical SDCs are a subset of SDCs that lead to a different ML prediction. + Note that with the default fault injection settings, it is possible to see no erroneous outputs - single bit flips may be masked by ML models. + ``` + >>> ./check_sdc.sh + Total Number of Runs: 20 + Number of SDCs: 4 + ``` + + ``` + >>> python check_critical_sdc.py + Total Number of Runs: 20 + Number of Critical SDCs: 1 + ``` + + You can also cycle through the run#, and examine the different fault injected runs. Modify the 3 to the desired run# number. + In this example, `eight.png` is mispredicted as 2, with a probabilility score of 1.000000. + ``` + >>> cat llfi/std_output/std_outputfile-run-0-3 + Time taken to execute the model: 0.026183 + Final prediction for eight.png is: 0.000000 0.000000 1.00000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 + ``` + + +## Part 2: Specifying Specific Neural Network Layers and Increasing Fault Intensity (5 min) + +View the original model, `mnist-cnn.py`, written in TensorFlow for reference. +In this exercise, we wish to select specific layers in the model for fault injection. +Previously, `input.yaml` was configured so that many different layers were chosen at random for fault injection. + +Reference code for CNN model: +```py +def get_model(): + model = models.Sequential() + model.add( + layers.Conv2D( + 32, (5, 5), activation="relu", input_shape=( + 28, 28, 1))) + model.add(layers.MaxPooling2D((2, 2))) + model.add(layers.Conv2D(64, (5, 5), activation="relu")) + model.add(layers.MaxPooling2D((2, 2))) + + model.add(layers.Flatten()) + model.add(layers.Dense(10, activation="softmax")) + + model.compile( + optimizer="adam", + loss=losses.SparseCategoricalCrossentropy(), + metrics=["accuracy"], + ) + return model +``` + +1. Modify the `input.yaml` file so that we perform fault injection into the second convolution layer only. + The number of entries, separated with semicolons, in `layerNo` must match that of the `layerName`. + To see some observable differences in fault injection, increase the number of `fi_num_bits` to 8 to increase the fault injection intensity. + Normally, we would run this 1000 times and automatically examine the traces. + Rerun LLTFI with `./runllfi.sh` - there is no need to recompile again, as the model has not changed. + + For your convenience, all of these changes have been made in `input2.yaml`. We can simply copy the YAML settings by running these following commands. + ``` + cp input2.yaml input.yaml + ./runllfi.sh + ``` + + +``` +compileOption: +    instSelMethod: +      - customInstselector: +          include: +            - CustomTensorOperator +          options: +            - -layerNo=2 (Change this line) +            - -layerName=conv (Change this line) + +    regSelMethod: regloc +    regloc: dstreg + +    includeInjectionTrace: +        - forward + +    tracingPropagation: False # trace dynamic instruction values. + +    tracingPropagationOption: +        maxTrace: 250 # max number of instructions to trace during fault injection run +        debugTrace: False +        mlTrace: False # enable for tracing ML programs +        generateCDFG: True + +runOption: +    - run: +        numOfRuns: 20 +        fi_type: bitflip +        window_len_multiple_startindex: 1 +        window_len_multiple_endindex: 500 +        fi_max_multiple: 5 (Change this line) +        fi_num_bits: 8 (Add this line) +``` + +2. Analyze the number of SDCs and critical SDCs under increased fault intensity. The values for SDCs and critical SDCs should increase compared to Part 1. + ``` + >>> ./check_sdc.sh + ... + Total Number of Runs: 20 + Number of SDCs: 7 + ``` + + ``` + >>> python check_critical_sdc.py + ... + Total Number of Runs: 20 + Number of Critical SDCs: 3 + ``` +